You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apex.apache.org by Pramod Immaneni <pr...@datatorrent.com> on 2017/03/01 22:24:09 UTC
Re: Java packages: legacy -> org.apache.apex
Inline
On Mon, Feb 27, 2017 at 11:03 PM, Thomas Weise <th...@apache.org> wrote:
> -->
>
> On Mon, Feb 27, 2017 at 1:27 PM, Pramod Immaneni <pr...@datatorrent.com>
> wrote:
>
> > For malhar, for existing operators, I prefer we do this as part of the
> > planned refactoring for breaking the monolith modules into baby packages
> > and would also prefer deprecating the existing operators in place.
>
>
> Refactor into smaller modules was discussed for malhar-contrib and given
> the overall state of that module I think it is OK to defer package renaming
> there. I do however prefer to see the package rename addressed for other
> modules, especially for the main library module.
>
Should we consider breaking the library into smaller modules as well, the
file/block operators for example probably can be in their own module from
just an organizational perspective.
>
>
> > This
> > will help us achieve two things. First, the user will see all the new
> > changes at once as opposed to dealing with it twice (with package rename
> > and dependency changes) and second it will allow for a smoother
> transition
> > as the existing code will still work in a deprecated state. It will also
> > give a more consistent structure to malhar. For new operators, we can go
> > with the new package path but we need to ensure they will get moved into
> > the baby packages as well.
> >
>
> I think existing operators should be renamed so that git history remains. A
> possible solution for backward compatibility could be to subsequently add
> empty subclasses in the previous location (for existing concrete operators
> that we know are actually in use) to simplify migration for users.
>
>
Yes we can do that.
>
> > For demos, we can modify the paths as the apps are typically used
> wholesale
> > and the interface is typically manual interaction.
> >
> > For core, if we are adding new api subsystems, like the launcher api we
> > added recently for example, we can go with new package path but if we are
> > making incremental additions to existing functionality, I feel it is
> better
> > to keep it in the same package. I also prefer we keep the package of the
> > implementation classes consistent with api, for understandability and
> > readability of the code. So, for example, we don't change package path of
> > LogicalPlan as it is an implementation of DAG. It is subjective, but it
> > will be good if we can also do the same with classes closely related to
> the
> > implementation classes as well. Maybe we can moving these on a package by
> > package basis, like everything in com.datatorrent.stram.engine could be
> > moved. For completely internal components like buffer server, we can move
> > them wholesale. We can consider moving all api and classes, when we go to
> > next major release but would like to see if we can find a way to support
> > existing api for one more major release in deprecated mode.
> >
> >
> The point of the major release is to enable backward incompatible changes
> and I don't think it is realistic to support the existing API for another
> major release. IMO it is also not necessary as most existing application
> code refers to operators, attributes and the application interface. Perhaps
> it is possible to keep those around as interface extensions to help
> migration. Custom operators may need to be migrated to reflect API changes,
> and I would consider that a reasonable task for operator developers as part
> of a major upgrade.
>
It would be good if we can keep them as deprecated interface extensions for
one release to provide a smoother transition.
>
> API and implementation in engine are kept separate intentionally. They
> reside in different packages today, so I don't see a problem renaming
> com.datatorrent.stram.engine as you say, even when the API cannot be
> touched right away.
>
They are different packages but sharing a common prefix with api will be
helpful to someone new to codebase in terms of readability. Not a big deal
and can be changed.
>
> Thanks
> >
> > On Mon, Feb 27, 2017 at 7:39 AM, Thomas Weise <th...@apache.org> wrote:
> >
> > > Hi,
> > >
> > > This topic has come up on several PRs and I think it warrants a broader
> > > discussion.
> > >
> > > At the time of incubation, the decision was to defer change of Java
> > > packages from com.datatorrent to org.apache.apex till next major
> release
> > to
> > > ensure backward compatibility for users.
> > >
> > > Unfortunately that has lead to some confusion, as contributors continue
> > to
> > > add new code under legacy packages.
> > >
> > > It is also a wider issue that examples for using Apex continue to refer
> > to
> > > com.datatorrent packages, nearly one year after graduation. More and
> more
> > > user code is being built on top of something that needs to change, the
> > can
> > > is being kicked down the road and users will face more changes later.
> > >
> > > I would like to propose the following:
> > >
> > > 1. All new code has to be submitted under org.apache.apex packages
> > >
> > > 2. Not all code is under backward compatibility restriction and in
> those
> > > cases we can rename the packages right away. Examples: buffer server,
> > > engine, demos/examples, benchmarks
> > >
> > > 3. Discuss when the core API and operators can be changed. For
> operators
> > we
> > > have a bit more freedom to do changes before a major release as most of
> > > them are marked @Evolving and users have the ability to continue using
> > > prior version of Malhar with newer engine due to engine backward
> > > compatibility guarantee.
> > >
> > > Thanks,
> > > Thomas
> > >
> >
>
Re: Java packages: legacy -> org.apache.apex
Posted by Sandesh Hegde <sa...@datatorrent.com>.
Why is there a urgency, why cant this go into 4.0 Malhar with possibly
other breaking changes?
On Sat, Jul 15, 2017 at 7:57 AM Thomas Weise <th...@apache.org> wrote:
> Discussing what in the future might become stable needs to be a separate
> thread, it will be a much bigger discussion.
>
> The topic here is to relocate the packages. With a few exceptions
> relocation won't affect the semantic versioning. Semantic versioning is
> essentially not effective for Malhar because almost everything is @Evolving
> (and there are reasons for that.. -> separate topic)
>
> I don't really like the idea of creating bw compatibility stubs for the
> follow up PR. It creates even more clutter in the source tree than there
> already is and so here is an alternative suggestion:
>
>
> https://github.com/tweise/apex-malhar/blob/malhar37-compat/shaded-malhar37/pom.xml
>
> Create a shaded artifact that provides the old com.datatorrent.* classes as
> of release 3.7. Users can include that artifact if they don't want to
> change import statements. At the same time they have an incentive to switch
> to the relocated classes to take advantage of bug fixes and new
> functionality.
>
> I will work on the first PR that does the relocate. In the meantime, we can
> finalize what backward compatibility support we want to provide and how.
>
> Thanks,
> Thomas
>
>
>
>
> On Fri, Jul 14, 2017 at 11:33 AM, Pramod Immaneni <pr...@datatorrent.com>
> wrote:
>
> > How about coming up with a list of @Evolving operators that we would like
> > to see in the eventual stable list and move those along with the not
> > @Evolving ones in org.apache.apex with b/w stubs and leave the rest as
> they
> > are. Then have a follow up JIRA for the rest to be moved over to contrib
> > and be deprecated.
> >
> > Thanks
> >
> > On Fri, Jul 14, 2017 at 10:37 AM, Thomas Weise <th...@gmail.com>
> > wrote:
> >
> > > We need to keep the discussion here on topic, if other things are piled
> > on
> > > top then nothing gets done.
> > >
> > > Most operators today are not stable, they are @Evolving. So perhaps it
> is
> > > necessary to have a separate discussion about that, outside of this
> > thread.
> > > My guess is that there are only a few operators that we could declare
> > > stable. Specifically, under contrib the closest one might have been
> > Kafka,
> > > but that is already superseded by the newer versions.
> > >
> > > Thomas
> > >
> > >
> > > On Fri, Jul 14, 2017 at 10:21 AM, Pramod Immaneni <
> > pramod@datatorrent.com>
> > > wrote:
> > >
> > > > We had a discussion a while back, agreed to relegate non-stable and
> > > > experimental operators to contrib and also added this to contribution
> > > > guidelines. We aexecuted on this and cleaned up the repo by moving
> > > > operators deemed non-suitable for production use at that time to
> > contrib.
> > > > So I wouldn't say the operators that are at the top level today or
> the
> > > ones
> > > > in library are 0.x.x quality. Granted, we may need to do one more
> pass
> > as
> > > > some of the operators may no longer be considered the right
> > > implementations
> > > > with the advent of the windowed operator.
> > > >
> > > > Since contrib used to be the place that housed operators that
> required
> > > > third party libraries, there are some production quality operators in
> > > there
> > > > that need to be pulled out into top level modules.
> > > >
> > > > I think we are in agreement that for operators that we consider
> stable,
> > > we
> > > > should provide a b/w stub. I would add that we strongly consider
> > creating
> > > > the org.apache.apex counterparts of any stable operators that are in
> > > > contrib out in top level modules and have the com.datatorrent stubs
> in
> > > > contrib.
> > > >
> > > > For the operators not considered stable, I would prefer we either
> leave
> > > the
> > > > current package path as is or if we change the package path then
> create
> > > the
> > > > b/w stub as I am not sure how widely they are in use (so, in essence,
> > > > preserve semantic versioning). It would be good if there is a
> followup
> > > JIRA
> > > > that takes a look at what other operators can be moved to contrib
> with
> > > the
> > > > advent of the newer frameworks and understanding.
> > > >
> > > > Thanks
> > > >
> > > > On Fri, Jul 14, 2017 at 9:44 AM, Thomas Weise <th...@apache.org>
> wrote:
> > > >
> > > > > Most of the operators evolve, as is quite clearly indicated by
> > > @Evolving
> > > > > annotations. So while perhaps 0.x.x would be a more appropriate
> > version
> > > > > number, I don't think you can change that.
> > > > >
> > > > > Thomas
> > > > >
> > > > > On Fri, Jul 14, 2017 at 9:35 AM, Vlad Rozov <
> v.rozov@datatorrent.com
> > >
> > > > > wrote:
> > > > >
> > > > > > If entire library is not stable, its version should be 0.x.x and
> > > there
> > > > > > should not be any semantic versioning enabled or implied. It
> > evolves.
> > > > If
> > > > > > some operators are stable as 3.8.x version suggests, the library
> > > should
> > > > > > follow semantic versioning and it is not OK to make a major
> change
> > > that
> > > > > > breaks semantic versioning across the entire library. It is not a
> > > > finding
> > > > > > an excuse not to make a change. For me semantic versioning and
> > > backward
> > > > > > compatibility is more important than a proper package name.
> > > > > >
> > > > > > Thank you,
> > > > > >
> > > > > > Vlad
> > > > > >
> > > > > >
> > > > > > On 7/14/17 09:11, Thomas Weise wrote:
> > > > > >
> > > > > >> Semantic versioning makes sense when you have a stable baseline.
> > As
> > > > long
> > > > > >> as
> > > > > >> frequent fixes need to be made to address basic issues, it makes
> > no
> > > > > sense
> > > > > >> to declare operators stable, which is why they are marked
> > evolving.
> > > > > >>
> > > > > >> Instead of finding reasons to avoid changes that should have
> been
> > > > made a
> > > > > >> long time ago, why not discuss what a "stable operator" is and
> > which
> > > > > ones
> > > > > >> deserve to be in that category. Those are the ones that will get
> > the
> > > > > >> backward compatibility stub.
> > > > > >>
> > > > > >> Thanks,
> > > > > >> Thomas
> > > > > >>
> > > > > >>
> > > > > >> On Fri, Jul 14, 2017 at 8:34 AM, Vlad Rozov <
> > > v.rozov@datatorrent.com>
> > > > > >> wrote:
> > > > > >>
> > > > > >> My preference is to agree that the next Malhar release is 4.0.0
> > and
> > > > make
> > > > > >>> the proposed changes when 3.8.0 goes out. Otherwise why to keep
> > > > > semantic
> > > > > >>> versioning checks on Malhar in case there is no version
> > > > compatibility.
> > > > > >>>
> > > > > >>> Thank you,
> > > > > >>>
> > > > > >>> Vlad
> > > > > >>>
> > > > > >>>
> > > > > >>> On 7/14/17 08:11, Thomas Weise wrote:
> > > > > >>>
> > > > > >>> next release is 3.8.0
> > > > > >>>>
> > > > > >>>> On Fri, Jul 14, 2017 at 7:55 AM, Vlad Rozov <
> > > > v.rozov@datatorrent.com>
> > > > > >>>> wrote:
> > > > > >>>>
> > > > > >>>> next release - 3.9.0 or 4.0.0?
> > > > > >>>>
> > > > > >>>>> Thank you,
> > > > > >>>>>
> > > > > >>>>> Vlad
> > > > > >>>>>
> > > > > >>>>> On 7/13/17 22:25, Thomas Weise wrote:
> > > > > >>>>>
> > > > > >>>>> It is time to resurrect this thread and get going with the
> > work.
> > > > > >>>>>
> > > > > >>>>>> For the next release, I will sign up to do the package move
> in
> > > > > Malhar:
> > > > > >>>>>>
> > > > > >>>>>> https://issues.apache.org/jira/browse/APEXMALHAR-2517
> > > > > >>>>>>
> > > > > >>>>>> In general this will be straightforward; most classes in
> > Malhar
> > > > are
> > > > > >>>>>> marked
> > > > > >>>>>> evolving and it is trivial for users to change import
> > > statements.
> > > > > >>>>>> However,
> > > > > >>>>>> I would suggest to discuss if there are selected operators
> > that
> > > > are
> > > > > >>>>>> worth
> > > > > >>>>>> keeping a backward compatibility stub in the old location.
> > > > > >>>>>>
> > > > > >>>>>> Here is my plan:
> > > > > >>>>>>
> > > > > >>>>>> 1. Relocate all classes in *lib* and *contrib* within one
> PR -
> > > > this
> > > > > is
> > > > > >>>>>> all
> > > > > >>>>>>
> > > > > >>>>>> IDE automated work
> > > > > >>>>>> 2. Add backward compatibility classes, if, any in separate
> PR
> > > > > >>>>>> 3. Create PR for Megh library to reflect moved classes
> > > > > >>>>>>
> > > > > >>>>>> Thanks,
> > > > > >>>>>> Thomas
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>>> On Wed, Mar 1, 2017 at 2:24 PM, Pramod Immaneni <
> > > > > >>>>>> pramod@datatorrent.com
> > > > > >>>>>> wrote:
> > > > > >>>>>>
> > > > > >>>>>> Inline
> > > > > >>>>>>
> > > > > >>>>>> On Mon, Feb 27, 2017 at 11:03 PM, Thomas Weise <
> > thw@apache.org>
> > > > > >>>>>>> wrote:
> > > > > >>>>>>>
> > > > > >>>>>>> -->
> > > > > >>>>>>>
> > > > > >>>>>>> On Mon, Feb 27, 2017 at 1:27 PM, Pramod Immaneni <
> > > > > >>>>>>>> pramod@datatorrent.com
> > > > > >>>>>>>>
> > > > > >>>>>>>> wrote:
> > > > > >>>>>>>>
> > > > > >>>>>>>> For malhar, for existing operators, I prefer we do this as
> > > part
> > > > of
> > > > > >>>>>>>> the
> > > > > >>>>>>>>
> > > > > >>>>>>>> planned refactoring for breaking the monolith modules into
> > > baby
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> packages
> > > > > >>>>>>>>>
> > > > > >>>>>>>> and would also prefer deprecating the existing operators
> in
> > > > place.
> > > > > >>>>>>>> Refactor into smaller modules was discussed for
> > malhar-contrib
> > > > and
> > > > > >>>>>>>> given
> > > > > >>>>>>>> the overall state of that module I think it is OK to defer
> > > > package
> > > > > >>>>>>>>
> > > > > >>>>>>>> renaming
> > > > > >>>>>>>>
> > > > > >>>>>>> there. I do however prefer to see the package rename
> > addressed
> > > > for
> > > > > >>>>>>>
> > > > > >>>>>>>> other
> > > > > >>>>>>>> modules, especially for the main library module.
> > > > > >>>>>>>>
> > > > > >>>>>>>> Should we consider breaking the library into smaller
> modules
> > > as
> > > > > >>>>>>>> well,
> > > > > >>>>>>>>
> > > > > >>>>>>>> the
> > > > > >>>>>>> file/block operators for example probably can be in their
> own
> > > > > module
> > > > > >>>>>>> from
> > > > > >>>>>>> just an organizational perspective.
> > > > > >>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>> This
> > > > > >>>>>>>
> > > > > >>>>>>>> will help us achieve two things. First, the user will see
> > all
> > > > the
> > > > > >>>>>>>>> new
> > > > > >>>>>>>>> changes at once as opposed to dealing with it twice (with
> > > > package
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> rename
> > > > > >>>>>>>>>
> > > > > >>>>>>>> and dependency changes) and second it will allow for a
> > > smoother
> > > > > >>>>>>>> transition
> > > > > >>>>>>>>
> > > > > >>>>>>>> as the existing code will still work in a deprecated
> state.
> > It
> > > > > will
> > > > > >>>>>>>>
> > > > > >>>>>>>>> also
> > > > > >>>>>>>>>
> > > > > >>>>>>>> give a more consistent structure to malhar. For new
> > operators,
> > > > we
> > > > > >>>>>>>> can
> > > > > >>>>>>>> go
> > > > > >>>>>>>> with the new package path but we need to ensure they will
> > get
> > > > > moved
> > > > > >>>>>>>> into
> > > > > >>>>>>>> the baby packages as well.
> > > > > >>>>>>>>
> > > > > >>>>>>>> I think existing operators should be renamed so that git
> > > history
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> remains. A
> > > > > >>>>>>>>
> > > > > >>>>>>> possible solution for backward compatibility could be to
> > > > > subsequently
> > > > > >>>>>>>
> > > > > >>>>>>>> add
> > > > > >>>>>>>> empty subclasses in the previous location (for existing
> > > concrete
> > > > > >>>>>>>>
> > > > > >>>>>>>> operators
> > > > > >>>>>>>>
> > > > > >>>>>>> that we know are actually in use) to simplify migration for
> > > > users.
> > > > > >>>>>>>
> > > > > >>>>>>>>
> > > > > >>>>>>>> Yes we can do that.
> > > > > >>>>>>>>
> > > > > >>>>>>>> For demos, we can modify the paths as the apps are
> typically
> > > > used
> > > > > >>>>>>>
> > > > > >>>>>>> wholesale
> > > > > >>>>>>>>
> > > > > >>>>>>>> and the interface is typically manual interaction.
> > > > > >>>>>>>>
> > > > > >>>>>>>>> For core, if we are adding new api subsystems, like the
> > > > launcher
> > > > > >>>>>>>>> api
> > > > > >>>>>>>>> we
> > > > > >>>>>>>>> added recently for example, we can go with new package
> path
> > > but
> > > > > if
> > > > > >>>>>>>>> we
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> are
> > > > > >>>>>>>>>
> > > > > >>>>>>>> making incremental additions to existing functionality, I
> > feel
> > > > it
> > > > > is
> > > > > >>>>>>>> better
> > > > > >>>>>>>>
> > > > > >>>>>>>> to keep it in the same package. I also prefer we keep the
> > > > package
> > > > > of
> > > > > >>>>>>>>
> > > > > >>>>>>>>> the
> > > > > >>>>>>>>>
> > > > > >>>>>>>> implementation classes consistent with api, for
> > > > understandability
> > > > > >>>>>>>> and
> > > > > >>>>>>>>
> > > > > >>>>>>>> readability of the code. So, for example, we don't change
> > > > package
> > > > > >>>>>>>>> path
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> of
> > > > > >>>>>>>>>
> > > > > >>>>>>>> LogicalPlan as it is an implementation of DAG. It is
> > > subjective,
> > > > > but
> > > > > >>>>>>>> it
> > > > > >>>>>>>>
> > > > > >>>>>>>> will be good if we can also do the same with classes
> closely
> > > > > related
> > > > > >>>>>>>>> to
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> the
> > > > > >>>>>>>>>
> > > > > >>>>>>>> implementation classes as well. Maybe we can moving these
> > on a
> > > > > >>>>>>>> package
> > > > > >>>>>>>>
> > > > > >>>>>>>>> by
> > > > > >>>>>>>>>
> > > > > >>>>>>>> package basis, like everything in
> > com.datatorrent.stram.engine
> > > > > could
> > > > > >>>>>>>> be
> > > > > >>>>>>>>
> > > > > >>>>>>>> moved. For completely internal components like buffer
> > server,
> > > we
> > > > > can
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> move
> > > > > >>>>>>>>>
> > > > > >>>>>>>> them wholesale. We can consider moving all api and
> classes,
> > > when
> > > > > we
> > > > > >>>>>>>> go
> > > > > >>>>>>>> to
> > > > > >>>>>>>> next major release but would like to see if we can find a
> > way
> > > to
> > > > > >>>>>>>> support
> > > > > >>>>>>>> existing api for one more major release in deprecated
> mode.
> > > > > >>>>>>>>
> > > > > >>>>>>>> The point of the major release is to enable backward
> > > > incompatible
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> changes
> > > > > >>>>>>>> and I don't think it is realistic to support the existing
> > API
> > > > for
> > > > > >>>>>>>> another
> > > > > >>>>>>>> major release. IMO it is also not necessary as most
> existing
> > > > > >>>>>>>> application
> > > > > >>>>>>>> code refers to operators, attributes and the application
> > > > > interface.
> > > > > >>>>>>>>
> > > > > >>>>>>>> Perhaps
> > > > > >>>>>>>>
> > > > > >>>>>>> it is possible to keep those around as interface extensions
> > to
> > > > help
> > > > > >>>>>>>
> > > > > >>>>>>>> migration. Custom operators may need to be migrated to
> > reflect
> > > > API
> > > > > >>>>>>>>
> > > > > >>>>>>>> changes,
> > > > > >>>>>>>>
> > > > > >>>>>>> and I would consider that a reasonable task for operator
> > > > developers
> > > > > >>>>>>> as
> > > > > >>>>>>>
> > > > > >>>>>>>> part
> > > > > >>>>>>>>
> > > > > >>>>>>> of a major upgrade.
> > > > > >>>>>>>
> > > > > >>>>>>>> It would be good if we can keep them as deprecated
> interface
> > > > > >>>>>>>> extensions
> > > > > >>>>>>>>
> > > > > >>>>>>>> for
> > > > > >>>>>>> one release to provide a smoother transition.
> > > > > >>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>> API and implementation in engine are kept separate
> > > intentionally.
> > > > > >>>>>>> They
> > > > > >>>>>>>
> > > > > >>>>>>> reside in different packages today, so I don't see a
> problem
> > > > > renaming
> > > > > >>>>>>>> com.datatorrent.stram.engine as you say, even when the API
> > > > cannot
> > > > > be
> > > > > >>>>>>>> touched right away.
> > > > > >>>>>>>>
> > > > > >>>>>>>> They are different packages but sharing a common prefix
> with
> > > api
> > > > > >>>>>>>> will
> > > > > >>>>>>>> be
> > > > > >>>>>>>>
> > > > > >>>>>>>> helpful to someone new to codebase in terms of
> readability.
> > > Not
> > > > a
> > > > > >>>>>>> big
> > > > > >>>>>>> deal
> > > > > >>>>>>> and can be changed.
> > > > > >>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>> Thanks
> > > > > >>>>>>>
> > > > > >>>>>>> On Mon, Feb 27, 2017 at 7:39 AM, Thomas Weise <
> > thw@apache.org>
> > > > > >>>>>>>> wrote:
> > > > > >>>>>>>>
> > > > > >>>>>>>>> Hi,
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> This topic has come up on several PRs and I think it
> > > warrants a
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> broader
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>> discussion.
> > > > > >>>>>>>>
> > > > > >>>>>>>> At the time of incubation, the decision was to defer
> change
> > of
> > > > > Java
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>> packages from com.datatorrent to org.apache.apex till
> next
> > > > major
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> release
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>> to
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> ensure backward compatibility for users.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>> Unfortunately that has lead to some confusion, as
> > > contributors
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> continue
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>> to
> > > > > >>>>>>>>
> > > > > >>>>>>>> add new code under legacy packages.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>> It is also a wider issue that examples for using Apex
> > > continue
> > > > > to
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> refer
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>> to
> > > > > >>>>>>>>
> > > > > >>>>>>>> com.datatorrent packages, nearly one year after
> graduation.
> > > More
> > > > > and
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>> more
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>> user code is being built on top of something that needs
> to
> > > > > change,
> > > > > >>>>>>>>> the
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> can
> > > > > >>>>>>>>
> > > > > >>>>>>>> is being kicked down the road and users will face more
> > changes
> > > > > >>>>>>>>> later.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>> I would like to propose the following:
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> 1. All new code has to be submitted under
> org.apache.apex
> > > > > packages
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> 2. Not all code is under backward compatibility
> > restriction
> > > > and
> > > > > in
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> those
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>> cases we can rename the packages right away. Examples:
> > buffer
> > > > > >>>>>>>>> server,
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> engine, demos/examples, benchmarks
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> 3. Discuss when the core API and operators can be
> changed.
> > > For
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> operators
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>> we
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> have a bit more freedom to do changes before a major
> > release
> > > as
> > > > > >>>>>>>>> most
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>> of
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>> them are marked @Evolving and users have the ability to
> > > > continue
> > > > > >>>>>>>>
> > > > > >>>>>>>> using
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> prior version of Malhar with newer engine due to engine
> > > > backward
> > > > > >>>>>>>>
> > > > > >>>>>>>> compatibility guarantee.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>> Thanks,
> > > > > >>>>>>>>>> Thomas
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Java packages: legacy -> org.apache.apex
Posted by Thomas Weise <th...@apache.org>.
I agree that 3.x and 4.x for the current maturity level and state of the
library is inappropriate and that we should consider a version rebase. I
don't think changing artifact IDs is a big deal, especially when it is an
opportunity to bring consistency and clarity. Also I think that the
perspective of folks that are new to the project should be given greater
weight than legacy constraints when we discuss a major version change.
I also think that repository and JIRA could be renamed, as long as both
support redirect from old name for existing links?
What about git tags though, repository has past releases and existing tags.
New tags would overlap.
Thanks,
Thomas
On Mon, Aug 21, 2017 at 7:20 AM, Vlad Rozov <vr...@apache.org> wrote:
> Problem is the stability of the library - number of @Evolving operators
> suggests that it is not 4.0 version, but rather 0.x and without artifactId
> change version can't go back.
>
> In some cases preserving history does not provide any benefit and only
> causes problems by itself.
>
> Thank you,
>
> Vlad
>
>
> On 8/20/17 19:05, Pramod Immaneni wrote:
>
>> Not git history.. just from a historical perspective keeping something
>> around because it is not causing problems.
>>
>> On Sat, Aug 19, 2017 at 6:52 PM, Vlad Rozov <vr...@apache.org> wrote:
>>
>> Do you refer to git history or something else? The git history must be
>>> preserved.
>>>
>>> Thank you,
>>>
>>> Vlad
>>>
>>>
>>> On 8/19/17 13:42, Pramod Immaneni wrote:
>>>
>>> It will be consistent but there is something to be said about preserving
>>>> some history in the present where it allows itself.
>>>> On Sat, Aug 19, 2017 at 11:05 AM Vlad Rozov <vr...@apache.org> wrote:
>>>>
>>>> Yes, consistency.
>>>>
>>>>> Thank you,
>>>>>
>>>>> Vlad
>>>>>
>>>>> On 8/19/17 10:57, Pramod Immaneni wrote:
>>>>>
>>>>> Is there any significant gain by doing this?
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> On Sat, Aug 19, 2017 at 10:36 AM, Vlad Rozov <vr...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>> The parent level pom artifact Id is not referenced by applications, it
>>>>>>
>>>>>>> exists simply to group modules together under current malhar project.
>>>>>>>
>>>>>>> The
>>>>>> name can be changed to library-parent(root) or
>>>>>> apex-library-parent(root) or
>>>>>> something similar. Note that the proposal includes rename of the
>>>>>> project
>>>>>>
>>>>>>> from apex-malhar to apex-library along with the version change.
>>>>>>>
>>>>>>> Thank you,
>>>>>>>
>>>>>>> Vlad
>>>>>>>
>>>>>>>
>>>>>>> On 8/19/17 10:10, Pramod Immaneni wrote:
>>>>>>>
>>>>>>> Would library module then become apex-library-library? Why change the
>>>>>>>
>>>>>>>> malhar artifact id as the project is called apex-malhar.
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>>
>>>>>>>> On Sat, Aug 19, 2017 at 10:03 AM, Vlad Rozov <vr...@apache.org>
>>>>>>>>
>>>>>>>> wrote:
>>>>>>>
>>>>>> Overall I support the proposal and this is what I was in favor of
>>>>>>
>>>>>>> during
>>>>>>>
>>>>>> the last discussion. I'd also propose changing the name of the
>>>>>>
>>>>>>> artifact
>>>>>>>>
>>>>>>> id
>>>>>>
>>>>>>> and git repo from malhar to apex-library as part of the major version
>>>>>>>>> change. It will make it more consistent with the entire project and
>>>>>>>>>
>>>>>>>>> will
>>>>>>>>
>>>>>>> also allow using 1.0 (1.0-SNAPSHOT) version that more closely
>>>>>> reflects
>>>>>>
>>>>>>> maturity of the library.
>>>>>>>>>
>>>>>>>>> Thank you,
>>>>>>>>>
>>>>>>>>> Vlad
>>>>>>>>>
>>>>>>>>> On 8/18/17 17:04, Thomas Weise wrote:
>>>>>>>>>
>>>>>>>>> The proposed change does NOT require changes to existing apps. It
>>>>>>>>>
>>>>>>>>> follows
>>>>>>>>
>>>>>>> what you see for example in the JDK where older classes are retired
>>>>>>
>>>>>>> without
>>>>>>>>>> making breaking major release changes.
>>>>>>>>>>
>>>>>>>>>> Even though not required technically, it looks like the preference
>>>>>>>>>>
>>>>>>>>>> is to
>>>>>>>>>
>>>>>>>> make this change with a 4.0 release. I will therefore propose to
>>>>>>
>>>>>>> change
>>>>>>>>>
>>>>>>>> the
>>>>>>
>>>>>>> master branch to 4.0-SNAPSHOT and make this and other changes deemed
>>>>>>>>>> worthwhile.
>>>>>>>>>>
>>>>>>>>>> To me it is more important that to newcomers the codebase does not
>>>>>>>>>>
>>>>>>>>>> look
>>>>>>>>>
>>>>>>>> like an outdated legacy situation and therefore if I have to make a
>>>>>>
>>>>>>> choice
>>>>>>>>>> than pretty much any version number will do for that. So far I
>>>>>>>>>> have
>>>>>>>>>>
>>>>>>>>>> seen
>>>>>>>>>
>>>>>>>> very little interest and initiative in addressing maintenance work
>>>>>>
>>>>>>> like
>>>>>>>>>
>>>>>>>> the
>>>>>>
>>>>>>> packages, checkstyle, CI stability, documentation etc. So it would
>>>>>>>>>> be
>>>>>>>>>> great
>>>>>>>>>> if people interested would step up and contribute.
>>>>>>>>>>
>>>>>>>>>> Here is the next proposal how to proceed:
>>>>>>>>>>
>>>>>>>>>> - create release-3.8 branch for last 3.8 release right
>>>>>>>>>> away
>>>>>>>>>> - update master to 4.0-SNAPSHOT as part of the open PR
>>>>>>>>>> #664
>>>>>>>>>> - identify further cleanup work for master
>>>>>>>>>> - release 4.0
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Thomas
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, Aug 17, 2017 at 4:48 PM, Amol Kekre <amol@datatorrent.com
>>>>>>>>>> >
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> This following pull request should be taken up in 4.0.0. See my
>>>>>>>>>>
>>>>>>>>>> comments
>>>>>>>>>
>>>>>>>> in
>>>>>>
>>>>>>> https://github.com/apache/apex-malhar/pull/664
>>>>>>>>>>> <https://www.google.com/url?q=https%3A%2F%2Fgithub.com%
>>>>>>>>>>> 2Fapache%2Fapex-malhar%2Fpull%2F664&sa=D&ust=150302026944400
>>>>>>>>>>> 0&usg=
>>>>>>>>>>> AFQjCNHDPZIg2e6I33Jb1XjB5Ir1FkdURQ>
>>>>>>>>>>> https://github.com/apache/apex-malhar/pull/662
>>>>>>>>>>> <https://www.google.com/url?q=https%3A%2F%2Fgithub.com%
>>>>>>>>>>> 2Fapache%2Fapex-malhar%2Fpull%2F662&sa=D&ust=150302026944400
>>>>>>>>>>> 0&usg=
>>>>>>>>>>> AFQjCNF7Xde7X55M3qmc8z-D5y3ZwNg7Fg>
>>>>>>>>>>>
>>>>>>>>>>> This merge should not be done without a consensus. This will
>>>>>>>>>>> require
>>>>>>>>>>> code
>>>>>>>>>>> changes to existing apps.
>>>>>>>>>>>
>>>>>>>>>>> Thks,
>>>>>>>>>>> Amol
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> E:amol@datatorrent.com | M: 510-449-2606 | Twitter:
>>>>>>>>>>> @*amolhkekre*
>>>>>>>>>>>
>>>>>>>>>>> www.datatorrent.com
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Aug 14, 2017 at 7:48 PM, Thomas Weise <th...@apache.org>
>>>>>>>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>
>>>>>>> I opened following PRs for the package change:
>>>>>>>>>>>
>>>>>>>>>>>> https://github.com/apache/apex-malhar/pull/662
>>>>>>>>>>>>
>>>>>>>>>>>> Moves all classes with history retained (hence 2 commits). Also
>>>>>>>>>>>> contains
>>>>>>>>>>>> checkstyle and other mechanical changes.
>>>>>>>>>>>>
>>>>>>>>>>>> https://github.com/apache/apex-malhar/pull/664
>>>>>>>>>>>>
>>>>>>>>>>>> Adds backward compatibility jar.
>>>>>>>>>>>>
>>>>>>>>>>>> Once above PRs are merged the new artifact can be deployed and
>>>>>>>>>>>> introduced
>>>>>>>>>>>> as dependency in malhar-library.
>>>>>>>>>>>>
>>>>>>>>>>>> Please review.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Thomas
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Sun, Jul 16, 2017 at 7:04 AM, Thomas Weise <th...@apache.org>
>>>>>>>>>>>>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>> My original list of work items contained the b/w compatibility
>>>>>>
>>>>>>> aspect,
>>>>>>>>>>>
>>>>>>>>>> I
>>>>>>
>>>>>>> don't think there should be any confusion of whether it will be
>>>>>>>>>>>> covered
>>>>>>>>>>>>
>>>>>>>>>>>> here or not.
>>>>>>>>>>>>
>>>>>>>>>>>>> The proposed shading will provide the old classes and they will
>>>>>>>>>>>>> be
>>>>>>>>>>>>>
>>>>>>>>>>>>> frozen
>>>>>>>>>>>>>
>>>>>>>>>>>>> as of release 3.7. That's the same as making a copy of the code
>>>>>>>>>>>> and
>>>>>>>>>>>> never
>>>>>>>>>>>> again making changes to the original classes. This cannot be
>>>>>>>>>>>> accomplished
>>>>>>>>>>>> by using the older 3.7 release in your project because you
>>>>>>>>>>>> cannot
>>>>>>>>>>>>
>>>>>>>>>>>> use
>>>>>>>>>>>
>>>>>>>>>> 2
>>>>>>
>>>>>>> different versions of Malhar in parallel unless you apply shading.
>>>>>>>>>>>>
>>>>>>>>>>>>> The shaded artifact will only expose the com.datatorrent
>>>>>>>>>>>>> classes,
>>>>>>>>>>>>>
>>>>>>>>>>>>> and
>>>>>>>>>>>>
>>>>>>>>>>> they
>>>>>>
>>>>>>> will be self-contained as the rest of the classes that they may
>>>>>>>>>>>>
>>>>>>>>>>>> depend
>>>>>>>>>>>
>>>>>>>>>> on
>>>>>>
>>>>>>> are shaded. The shaded artifact does not evolve, there are not
>>>>>>>>>>>> more
>>>>>>>>>>>> changes
>>>>>>>>>>>>
>>>>>>>>>>>> to com.datatorrent classes after they are relocated in master.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Thomas
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Sun, Jul 16, 2017 at 2:00 AM, Pramod Immaneni <
>>>>>>>>>>>>>
>>>>>>>>>>>>> pramod@datatorrent.com
>>>>>>>>>>>>>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> I don't think we can limit the topic strictly to relocation
>>>>>>>>>>>> without
>>>>>>>>>>>>
>>>>>>>>>>> having
>>>>>>
>>>>>>> a good b/w compatibility story or at least one that goes far
>>>>>>>>>>>>>
>>>>>>>>>>>>> enough.
>>>>>>>>>>>>
>>>>>>>>>>> The shading idea sounds interesting. Why not let the shaded
>>>>>>
>>>>>>> version
>>>>>>>>>>>>>
>>>>>>>>>>>> move
>>>>>>
>>>>>>> forward with each release till we hit a major release. If it is
>>>>>>>>>>>>>
>>>>>>>>>>>> going
>>>>>>>>>>>
>>>>>>>>>> to
>>>>>>
>>>>>>> remain pegged at 3.7.0, why shade in the first place as the
>>>>>>>>>>>>>
>>>>>>>>>>>> regular
>>>>>>>>>>>>
>>>>>>>>>>>> 3.7.0
>>>>>>>>>>>>
>>>>>>>>>>>>> release would do the same job and it would be same as the loss
>>>>>>>>>>>>> of
>>>>>>>>>>>>>
>>>>>>>>>>>>> b/w
>>>>>>>>>>>>
>>>>>>>>>>> compatibility with newer releases.
>>>>>>
>>>>>>> Thanks
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Sat, Jul 15, 2017 at 7:57 AM, Thomas Weise <thw@apache.org
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Discussing what in the future might become stable needs to be
>>>>>>>>>>>>>> a
>>>>>>>>>>>>>> separate
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> thread, it will be a much bigger discussion.
>>>>>>>>>>>>>>
>>>>>>>>>>>>> The topic here is to relocate the packages. With a few
>>>>>>>>>>>>> exceptions
>>>>>>>>>>>>>
>>>>>>>>>>>>>> relocation won't affect the semantic versioning. Semantic
>>>>>>>>>>>>>>> versioning
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> essentially not effective for Malhar because almost
>>>>>>>>>>>>>> everything
>>>>>>>>>>>>>>
>>>>>>>>>>>>> is
>>>>>>>>>>>>>
>>>>>>>>>>>>> @Evolving
>>>>>>>>>>>>>
>>>>>>>>>>>>>> (and there are reasons for that.. -> separate topic)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I don't really like the idea of creating bw compatibility
>>>>>>>>>>>>>>> stubs
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>
>>>>>>>>>>>>> the
>>>>>>
>>>>>>> follow up PR. It creates even more clutter in the source tree
>>>>>>>>>>>>>>
>>>>>>>>>>>>> than
>>>>>>>>>>>>>
>>>>>>>>>>>>> there
>>>>>>>>>>>>>
>>>>>>>>>>>>>> already is and so here is an alternative suggestion:
>>>>>>>>>>>>>>
>>>>>>>>>>>>> https://github.com/tweise/apex-malhar/blob/malhar37-
>>>>>>>>>>>>>
>>>>>>>>>>>>>> compat/shaded-malhar37/pom.xml
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Create a shaded artifact that provides the old
>>>>>>>>>>>>>>> com.datatorrent.*
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> classes as
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> of release 3.7. Users can include that artifact if they don't
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> want
>>>>>>>>>>>>>
>>>>>>>>>>>> to
>>>>>>
>>>>>>> change import statements. At the same time they have an
>>>>>>>>>>>>>>
>>>>>>>>>>>>> incentive
>>>>>>>>>>>>>
>>>>>>>>>>>>> to
>>>>>>>>>>>>
>>>>>>>>>>> switch
>>>>>>
>>>>>>> to the relocated classes to take advantage of bug fixes and new
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> functionality.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I will work on the first PR that does the relocate. In the
>>>>>>>>>>>>>>> meantime,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>
>>>>>>>>>>>>> finalize what backward compatibility support we want to provide
>>>>>>>>>>>>> and
>>>>>>>>>>>>>
>>>>>>>>>>>> how.
>>>>>>
>>>>>>> Thanks,
>>>>>>>>>>>>>>
>>>>>>>>>>>>> Thomas
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, Jul 14, 2017 at 11:33 AM, Pramod Immaneni <
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> pramod@datatorrent.com>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> How about coming up with a list of @Evolving operators that we
>>>>>>>>>>>>>>> would
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> like
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> to see in the eventual stable list and move those along with
>>>>>>>>>>>>> the
>>>>>>>>>>>>>
>>>>>>>>>>>>> not
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> @Evolving ones in org.apache.apex with b/w stubs and leave
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> rest
>>>>>>>>>>>>>
>>>>>>>>>>>>> as
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> they
>>>>>>>>>>>>> are. Then have a follow up JIRA for the rest to be moved over
>>>>>>>>>>>>> to
>>>>>>>>>>>>>
>>>>>>>>>>>>>> contrib
>>>>>>>>>>>>>>> and be deprecated.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Fri, Jul 14, 2017 at 10:37 AM, Thomas Weise <
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> thomas.weise@gmail.com>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> We need to keep the discussion here on topic, if other things
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> piled
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>> top then nothing gets done.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Most operators today are not stable, they are @Evolving. So
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> perhaps
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> necessary to have a separate discussion about that, outside of
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> thread.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> My guess is that there are only a few operators that we could
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> declare
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> stable. Specifically, under contrib the closest one might
>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> been
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Kafka,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> but that is already superseded by the newer versions.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thomas
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Fri, Jul 14, 2017 at 10:21 AM, Pramod Immaneni <
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> pramod@datatorrent.com>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> We had a discussion a while back, agreed to relegate
>>>>>>>>>>>>>>>>> non-stable
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> experimental operators to contrib and also added this to
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> contribution
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> guidelines. We aexecuted on this and cleaned up the repo by
>>>>>>>>>>>>>>>> moving
>>>>>>>>>>>>>>>> operators deemed non-suitable for production use at that
>>>>>>>>>>>>>>>> time
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> contrib.
>>>>>>>>>>>>>>> So I wouldn't say the operators that are at the top level
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> today
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ones
>>>>>>>>>>>>>
>>>>>>>>>>>>>> in library are 0.x.x quality. Granted, we may need to do one
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> more
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> pass
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>> some of the operators may no longer be considered the right
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> implementations
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> with the advent of the windowed operator.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Since contrib used to be the place that housed operators
>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> required
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> third party libraries, there are some production quality
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> operators in
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> there
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> that need to be pulled out into top level modules.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I think we are in agreement that for operators that we
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> consider
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> stable,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> should provide a b/w stub. I would add that we strongly
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> consider
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> creating
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> the org.apache.apex counterparts of any stable operators
>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> contrib out in top level modules and have the
>>>>>>>>>>>>>>> com.datatorrent
>>>>>>>>>>>>>>> stubs
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> contrib.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> For the operators not considered stable, I would prefer we
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> either
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> leave
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> current package path as is or if we change the package path
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> then
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> create
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> b/w stub as I am not sure how widely they are in use (so, in
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> essence,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> preserve semantic versioning). It would be good if there
>>>>>>>>>>>>>>>>> is a
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> followup
>>>>>>>>>>>>>>>> JIRA
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> that takes a look at what other operators can be moved to
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> contrib
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> advent of the newer frameworks and understanding.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Fri, Jul 14, 2017 at 9:44 AM, Thomas Weise <
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> thw@apache.org
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> wrote:
>>>>>>
>>>>>>> Most of the operators evolve, as is quite clearly indicated
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> @Evolving
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> annotations. So while perhaps 0.x.x would be a more
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> appropriate
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> version
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> number, I don't think you can change that.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thomas
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Fri, Jul 14, 2017 at 9:35 AM, Vlad Rozov <
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> v.rozov@datatorrent.com
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> If entire library is not stable, its version should be
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 0.x.x
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> there
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> should not be any semantic versioning enabled or implied.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> It
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> evolves.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> If
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> some operators are stable as 3.8.x version suggests, the
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> library
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> follow semantic versioning and it is not OK to make a
>>>>>>>>>>>>>>>>> major
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> change
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> breaks semantic versioning across the entire library. It
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> not a
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> finding
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> an excuse not to make a change. For me semantic versioning
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> backward
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> compatibility is more important than a proper package
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> name.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thank you,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Vlad
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 7/14/17 09:11, Thomas Weise wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Semantic versioning makes sense when you have a stable
>>>>>>>>>>>>>>>>>>>> baseline.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> As
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> long
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> frequent fixes need to be made to address basic issues,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> makes
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> no
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> sense
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> to declare operators stable, which is why they are marked
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> evolving.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Instead of finding reasons to avoid changes that should
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> been
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> made a
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> long time ago, why not discuss what a "stable operator"
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> which
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> ones
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> deserve to be in that category. Those are the ones that
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> get
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> backward compatibility stub.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thomas
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Fri, Jul 14, 2017 at 8:34 AM, Vlad Rozov <
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> v.rozov@datatorrent.com>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> My preference is to agree that the next Malhar release is
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 4.0.0
>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> make
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> the proposed changes when 3.8.0 goes out. Otherwise why
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> keep
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> semantic
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> versioning checks on Malhar in case there is no version
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> compatibility.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thank you,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Vlad
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On 7/14/17 08:11, Thomas Weise wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> next release is 3.8.0
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Fri, Jul 14, 2017 at 7:55 AM, Vlad Rozov <
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> v.rozov@datatorrent.com>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> next release - 3.9.0 or 4.0.0?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thank you,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Vlad
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On 7/13/17 22:25, Thomas Weise wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> It is time to resurrect this thread and get going
>>>>>>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> work.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> For the next release, I will sign up to do the
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> package
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> move
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Malhar:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> https://issues.apache.org/
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> jira/browse/APEXMALHAR-2517
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> In general this will be straightforward; most classes
>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Malhar
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> marked
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> evolving and it is trivial for users to change import
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> statements.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> However,
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I would suggest to discuss if there are selected
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> operators
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> worth
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> keeping a backward compatibility stub in the old
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> location.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Here is my plan:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> 1. Relocate all classes in *lib* and *contrib* within
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> one
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> PR -
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> IDE automated work
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> 2. Add backward compatibility classes, if, any in
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> separate
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> PR
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> 3. Create PR for Megh library to reflect moved
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> classes
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thomas
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Wed, Mar 1, 2017 at 2:24 PM, Pramod Immaneni <
>>>>>>>>>>>>>>>>>>>>>>>>> pramod@datatorrent.com
>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Inline
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Feb 27, 2017 at 11:03 PM, Thomas Weise <
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> thw@apache.org>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> -->
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Mon, Feb 27, 2017 at 1:27 PM, Pramod Immaneni <
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> pramod@datatorrent.com
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> For malhar, for existing operators, I prefer we
>>>>>>>>>>>>>>>>>>>>>>>>>>> do
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> this as
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> part
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> planned refactoring for breaking the monolith
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> modules
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> into
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> baby
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> packages
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> and would also prefer deprecating the existing
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> operators
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> place.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Refactor into smaller modules was discussed for
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> malhar-contrib
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> given
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> the overall state of that module I think it is OK
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> defer
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> package
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> renaming
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> there. I do however prefer to see the package rename
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> addressed
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> other
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> modules, especially for the main library module.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Should we consider breaking the library into
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> smaller
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> modules
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> well,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> file/block operators for example probably can be in
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> their
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> own
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> module
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> just an organizational perspective.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> This
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> will help us achieve two things. First, the user
>>>>>>>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> see
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> changes at once as opposed to dealing with it
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> twice
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> (with
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> package
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> rename
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> and dependency changes) and second it will allow
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> smoother
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> transition
>>>>>>>>>>>>>
>>>>>>>>>>>>>> as the existing code will still work in a
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> deprecated
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> state.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> It
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> give a more consistent structure to malhar. For new
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> operators,
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> go
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> with the new package path but we need to ensure
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> they
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> get
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> moved
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> into
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> the baby packages as well.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I think existing operators should be renamed so
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> git
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> history
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> remains. A
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> possible solution for backward compatibility could
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> subsequently
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> add
>>>>>>>>>>>>>
>>>>>>>>>>>>>> empty subclasses in the previous location (for
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> existing
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Thank you,
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Vlad
>>>>>>>
>>>>>>>
>>>>>>> Thank you,
>>>>>
>>>>> Vlad
>>>>>
>>>>>
>>>>> Thank you,
>>>
>>> Vlad
>>>
>>>
>
> Thank you,
>
> Vlad
>
Re: Java packages: legacy -> org.apache.apex
Posted by Vlad Rozov <vr...@apache.org>.
Problem is the stability of the library - number of @Evolving operators
suggests that it is not 4.0 version, but rather 0.x and without
artifactId change version can't go back.
In some cases preserving history does not provide any benefit and only
causes problems by itself.
Thank you,
Vlad
On 8/20/17 19:05, Pramod Immaneni wrote:
> Not git history.. just from a historical perspective keeping something
> around because it is not causing problems.
>
> On Sat, Aug 19, 2017 at 6:52 PM, Vlad Rozov <vr...@apache.org> wrote:
>
>> Do you refer to git history or something else? The git history must be
>> preserved.
>>
>> Thank you,
>>
>> Vlad
>>
>>
>> On 8/19/17 13:42, Pramod Immaneni wrote:
>>
>>> It will be consistent but there is something to be said about preserving
>>> some history in the present where it allows itself.
>>> On Sat, Aug 19, 2017 at 11:05 AM Vlad Rozov <vr...@apache.org> wrote:
>>>
>>> Yes, consistency.
>>>> Thank you,
>>>>
>>>> Vlad
>>>>
>>>> On 8/19/17 10:57, Pramod Immaneni wrote:
>>>>
>>>>> Is there any significant gain by doing this?
>>>>>
>>>>> Thanks
>>>>>
>>>>> On Sat, Aug 19, 2017 at 10:36 AM, Vlad Rozov <vr...@apache.org> wrote:
>>>>>
>>>>> The parent level pom artifact Id is not referenced by applications, it
>>>>>> exists simply to group modules together under current malhar project.
>>>>>>
>>>>> The
>>>>> name can be changed to library-parent(root) or
>>>>> apex-library-parent(root) or
>>>>> something similar. Note that the proposal includes rename of the project
>>>>>> from apex-malhar to apex-library along with the version change.
>>>>>>
>>>>>> Thank you,
>>>>>>
>>>>>> Vlad
>>>>>>
>>>>>>
>>>>>> On 8/19/17 10:10, Pramod Immaneni wrote:
>>>>>>
>>>>>> Would library module then become apex-library-library? Why change the
>>>>>>> malhar artifact id as the project is called apex-malhar.
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>> On Sat, Aug 19, 2017 at 10:03 AM, Vlad Rozov <vr...@apache.org>
>>>>>>>
>>>>>> wrote:
>>>>> Overall I support the proposal and this is what I was in favor of
>>>>>> during
>>>>> the last discussion. I'd also propose changing the name of the
>>>>>>> artifact
>>>>> id
>>>>>>>> and git repo from malhar to apex-library as part of the major version
>>>>>>>> change. It will make it more consistent with the entire project and
>>>>>>>>
>>>>>>> will
>>>>> also allow using 1.0 (1.0-SNAPSHOT) version that more closely reflects
>>>>>>>> maturity of the library.
>>>>>>>>
>>>>>>>> Thank you,
>>>>>>>>
>>>>>>>> Vlad
>>>>>>>>
>>>>>>>> On 8/18/17 17:04, Thomas Weise wrote:
>>>>>>>>
>>>>>>>> The proposed change does NOT require changes to existing apps. It
>>>>>>>>
>>>>>>> follows
>>>>> what you see for example in the JDK where older classes are retired
>>>>>>>>> without
>>>>>>>>> making breaking major release changes.
>>>>>>>>>
>>>>>>>>> Even though not required technically, it looks like the preference
>>>>>>>>>
>>>>>>>> is to
>>>>> make this change with a 4.0 release. I will therefore propose to
>>>>>>>> change
>>>>> the
>>>>>>>>> master branch to 4.0-SNAPSHOT and make this and other changes deemed
>>>>>>>>> worthwhile.
>>>>>>>>>
>>>>>>>>> To me it is more important that to newcomers the codebase does not
>>>>>>>>>
>>>>>>>> look
>>>>> like an outdated legacy situation and therefore if I have to make a
>>>>>>>>> choice
>>>>>>>>> than pretty much any version number will do for that. So far I have
>>>>>>>>>
>>>>>>>> seen
>>>>> very little interest and initiative in addressing maintenance work
>>>>>>>> like
>>>>> the
>>>>>>>>> packages, checkstyle, CI stability, documentation etc. So it would
>>>>>>>>> be
>>>>>>>>> great
>>>>>>>>> if people interested would step up and contribute.
>>>>>>>>>
>>>>>>>>> Here is the next proposal how to proceed:
>>>>>>>>>
>>>>>>>>> - create release-3.8 branch for last 3.8 release right away
>>>>>>>>> - update master to 4.0-SNAPSHOT as part of the open PR #664
>>>>>>>>> - identify further cleanup work for master
>>>>>>>>> - release 4.0
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Thomas
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Aug 17, 2017 at 4:48 PM, Amol Kekre <am...@datatorrent.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> This following pull request should be taken up in 4.0.0. See my
>>>>>>>>>
>>>>>>>> comments
>>>>> in
>>>>>>>>>> https://github.com/apache/apex-malhar/pull/664
>>>>>>>>>> <https://www.google.com/url?q=https%3A%2F%2Fgithub.com%
>>>>>>>>>> 2Fapache%2Fapex-malhar%2Fpull%2F664&sa=D&ust=1503020269444000&usg=
>>>>>>>>>> AFQjCNHDPZIg2e6I33Jb1XjB5Ir1FkdURQ>
>>>>>>>>>> https://github.com/apache/apex-malhar/pull/662
>>>>>>>>>> <https://www.google.com/url?q=https%3A%2F%2Fgithub.com%
>>>>>>>>>> 2Fapache%2Fapex-malhar%2Fpull%2F662&sa=D&ust=1503020269444000&usg=
>>>>>>>>>> AFQjCNF7Xde7X55M3qmc8z-D5y3ZwNg7Fg>
>>>>>>>>>>
>>>>>>>>>> This merge should not be done without a consensus. This will
>>>>>>>>>> require
>>>>>>>>>> code
>>>>>>>>>> changes to existing apps.
>>>>>>>>>>
>>>>>>>>>> Thks,
>>>>>>>>>> Amol
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> E:amol@datatorrent.com | M: 510-449-2606 | Twitter: @*amolhkekre*
>>>>>>>>>>
>>>>>>>>>> www.datatorrent.com
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, Aug 14, 2017 at 7:48 PM, Thomas Weise <th...@apache.org>
>>>>>>>>>>
>>>>>>>>> wrote:
>>>>> Hi,
>>>>>>>>>> I opened following PRs for the package change:
>>>>>>>>>>> https://github.com/apache/apex-malhar/pull/662
>>>>>>>>>>>
>>>>>>>>>>> Moves all classes with history retained (hence 2 commits). Also
>>>>>>>>>>> contains
>>>>>>>>>>> checkstyle and other mechanical changes.
>>>>>>>>>>>
>>>>>>>>>>> https://github.com/apache/apex-malhar/pull/664
>>>>>>>>>>>
>>>>>>>>>>> Adds backward compatibility jar.
>>>>>>>>>>>
>>>>>>>>>>> Once above PRs are merged the new artifact can be deployed and
>>>>>>>>>>> introduced
>>>>>>>>>>> as dependency in malhar-library.
>>>>>>>>>>>
>>>>>>>>>>> Please review.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Thomas
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Sun, Jul 16, 2017 at 7:04 AM, Thomas Weise <th...@apache.org>
>>>>>>>>>>>
>>>>>>>>>> wrote:
>>>>> My original list of work items contained the b/w compatibility
>>>>>>>>>> aspect,
>>>>> I
>>>>>>>>>>> don't think there should be any confusion of whether it will be
>>>>>>>>>>> covered
>>>>>>>>>>>
>>>>>>>>>>> here or not.
>>>>>>>>>>>> The proposed shading will provide the old classes and they will
>>>>>>>>>>>> be
>>>>>>>>>>>>
>>>>>>>>>>>> frozen
>>>>>>>>>>>>
>>>>>>>>>>> as of release 3.7. That's the same as making a copy of the code
>>>>>>>>>>> and
>>>>>>>>>>> never
>>>>>>>>>>> again making changes to the original classes. This cannot be
>>>>>>>>>>> accomplished
>>>>>>>>>>> by using the older 3.7 release in your project because you cannot
>>>>>>>>>>>
>>>>>>>>>> use
>>>>> 2
>>>>>>>>>>> different versions of Malhar in parallel unless you apply shading.
>>>>>>>>>>>> The shaded artifact will only expose the com.datatorrent classes,
>>>>>>>>>>>>
>>>>>>>>>>> and
>>>>> they
>>>>>>>>>>> will be self-contained as the rest of the classes that they may
>>>>>>>>>>>
>>>>>>>>>> depend
>>>>> on
>>>>>>>>>>> are shaded. The shaded artifact does not evolve, there are not
>>>>>>>>>>> more
>>>>>>>>>>> changes
>>>>>>>>>>>
>>>>>>>>>>> to com.datatorrent classes after they are relocated in master.
>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Thomas
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Sun, Jul 16, 2017 at 2:00 AM, Pramod Immaneni <
>>>>>>>>>>>>
>>>>>>>>>>>> pramod@datatorrent.com
>>>>>>>>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> I don't think we can limit the topic strictly to relocation
>>>>>>>>>>> without
>>>>> having
>>>>>>>>>>>> a good b/w compatibility story or at least one that goes far
>>>>>>>>>>>>
>>>>>>>>>>> enough.
>>>>> The shading idea sounds interesting. Why not let the shaded
>>>>>>>>>>>> version
>>>>> move
>>>>>>>>>>>> forward with each release till we hit a major release. If it is
>>>>>>>>>> going
>>>>> to
>>>>>>>>>>>> remain pegged at 3.7.0, why shade in the first place as the
>>>>>>>>>>> regular
>>>>>>>>>>>
>>>>>>>>>>> 3.7.0
>>>>>>>>>>>> release would do the same job and it would be same as the loss of
>>>>>>>>>>>>
>>>>>>>>>>> b/w
>>>>> compatibility with newer releases.
>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Sat, Jul 15, 2017 at 7:57 AM, Thomas Weise <th...@apache.org>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Discussing what in the future might become stable needs to be a
>>>>>>>>>>>>> separate
>>>>>>>>>>>>>
>>>>>>>>>>>>> thread, it will be a much bigger discussion.
>>>>>>>>>>>> The topic here is to relocate the packages. With a few exceptions
>>>>>>>>>>>>>> relocation won't affect the semantic versioning. Semantic
>>>>>>>>>>>>>> versioning
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>
>>>>>>>>>>>>> essentially not effective for Malhar because almost everything
>>>>>>>>>>>> is
>>>>>>>>>>>>
>>>>>>>>>>>> @Evolving
>>>>>>>>>>>>> (and there are reasons for that.. -> separate topic)
>>>>>>>>>>>>>
>>>>>>>>>>>>>> I don't really like the idea of creating bw compatibility stubs
>>>>>>>>>>>>>>
>>>>>>>>>>>>> for
>>>>> the
>>>>>>>>>>>>> follow up PR. It creates even more clutter in the source tree
>>>>>>>>>>>> than
>>>>>>>>>>>>
>>>>>>>>>>>> there
>>>>>>>>>>>>> already is and so here is an alternative suggestion:
>>>>>>>>>>>> https://github.com/tweise/apex-malhar/blob/malhar37-
>>>>>>>>>>>>>> compat/shaded-malhar37/pom.xml
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Create a shaded artifact that provides the old
>>>>>>>>>>>>>> com.datatorrent.*
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> classes as
>>>>>>>>>>>>>>
>>>>>>>>>>>>> of release 3.7. Users can include that artifact if they don't
>>>>>>>>>>>>>
>>>>>>>>>>>> want
>>>>> to
>>>>>>>>>>>>> change import statements. At the same time they have an
>>>>>>>>>>>> incentive
>>>>>>>>>>>>
>>>>>>>>>>> to
>>>>> switch
>>>>>>>>>>>>> to the relocated classes to take advantage of bug fixes and new
>>>>>>>>>>>>>
>>>>>>>>>>>>>> functionality.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I will work on the first PR that does the relocate. In the
>>>>>>>>>>>>>> meantime,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>
>>>>>>>>>>>>> can
>>>>>>>>>>>> finalize what backward compatibility support we want to provide
>>>>>>>>>>>> and
>>>>> how.
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Thomas
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Jul 14, 2017 at 11:33 AM, Pramod Immaneni <
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> pramod@datatorrent.com>
>>>>>>>>>>>>>>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> How about coming up with a list of @Evolving operators that we
>>>>>>>>>>>>>> would
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> like
>>>>>>>>>>>> to see in the eventual stable list and move those along with the
>>>>>>>>>>>>
>>>>>>>>>>>>> not
>>>>>>>>>>>>>> @Evolving ones in org.apache.apex with b/w stubs and leave the
>>>>>>>>>>>> rest
>>>>>>>>>>>>
>>>>>>>>>>>>> as
>>>>>>>>>>>>>
>>>>>>>>>>>> they
>>>>>>>>>>>> are. Then have a follow up JIRA for the rest to be moved over to
>>>>>>>>>>>>>> contrib
>>>>>>>>>>>>>> and be deprecated.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>> On Fri, Jul 14, 2017 at 10:37 AM, Thomas Weise <
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> thomas.weise@gmail.com>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We need to keep the discussion here on topic, if other things
>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> piled
>>>>>>>>>>>>> on
>>>>>>>>>>>>> top then nothing gets done.
>>>>>>>>>>>>>>>> Most operators today are not stable, they are @Evolving. So
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> perhaps
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> it
>>>>>>>>>>>>> is
>>>>>>>>>>>>>
>>>>>>>>>>>>>> necessary to have a separate discussion about that, outside of
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> thread.
>>>>>>>>>>>>> My guess is that there are only a few operators that we could
>>>>>>>>>>>>>
>>>>>>>>>>>>>> declare
>>>>>>>>>>>>>>> stable. Specifically, under contrib the closest one might have
>>>>>>>>>>>>>> been
>>>>>>>>>>>>>>> Kafka,
>>>>>>>>>>>>> but that is already superseded by the newer versions.
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thomas
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Fri, Jul 14, 2017 at 10:21 AM, Pramod Immaneni <
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> pramod@datatorrent.com>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> We had a discussion a while back, agreed to relegate
>>>>>>>>>>>>>>>> non-stable
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>> experimental operators to contrib and also added this to
>>>>>>>>>>>>> contribution
>>>>>>>>>>>>>>> guidelines. We aexecuted on this and cleaned up the repo by
>>>>>>>>>>>>>>> moving
>>>>>>>>>>>>>>> operators deemed non-suitable for production use at that time
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> contrib.
>>>>>>>>>>>>>> So I wouldn't say the operators that are at the top level
>>>>>>>>>>>>> today
>>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>> the
>>>>>>>>>>>> ones
>>>>>>>>>>>>>> in library are 0.x.x quality. Granted, we may need to do one
>>>>>>>>>>>>>>>>> more
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> pass
>>>>>>>>>>>>>> as
>>>>>>>>>>>>>> some of the operators may no longer be considered the right
>>>>>>>>>>>>>>>> implementations
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> with the advent of the windowed operator.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Since contrib used to be the place that housed operators
>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> required
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> third party libraries, there are some production quality
>>>>>>>>>>>>>>> operators in
>>>>>>>>>>>>>>>> there
>>>>>>>>>>>>>>> that need to be pulled out into top level modules.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I think we are in agreement that for operators that we
>>>>>>>>>>>>>>>>> consider
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> stable,
>>>>>>>>>>>>>> we
>>>>>>>>>>>>> should provide a b/w stub. I would add that we strongly
>>>>>>>>>>>>>>>>> consider
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> creating
>>>>>>>>>>>>>> the org.apache.apex counterparts of any stable operators that
>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>> contrib out in top level modules and have the com.datatorrent
>>>>>>>>>>>>>> stubs
>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> contrib.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> For the operators not considered stable, I would prefer we
>>>>>>>>>>>>>>>>> either
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> leave
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> current package path as is or if we change the package path
>>>>>>>>>>>>>>>>> then
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> create
>>>>>>>>>>>>>> the
>>>>>>>>>>>>> b/w stub as I am not sure how widely they are in use (so, in
>>>>>>>>>>>>>>>>> essence,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> preserve semantic versioning). It would be good if there is a
>>>>>>>>>>>>>>> followup
>>>>>>>>>>>>>>> JIRA
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> that takes a look at what other operators can be moved to
>>>>>>>>>>>>>>>>> contrib
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> advent of the newer frameworks and understanding.
>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Fri, Jul 14, 2017 at 9:44 AM, Thomas Weise <
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> thw@apache.org
>>>>> wrote:
>>>>>>>>>>>>>>>> Most of the operators evolve, as is quite clearly indicated
>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>> @Evolving
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> annotations. So while perhaps 0.x.x would be a more
>>>>>>>>>>>>> appropriate
>>>>>>>>>>>>>>>>> version
>>>>>>>>>>>>>>> number, I don't think you can change that.
>>>>>>>>>>>>>> Thomas
>>>>>>>>>>>>>>>>>> On Fri, Jul 14, 2017 at 9:35 AM, Vlad Rozov <
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> v.rozov@datatorrent.com
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>> If entire library is not stable, its version should be
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 0.x.x
>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>> there
>>>>>>>>>>>>> should not be any semantic versioning enabled or implied.
>>>>>>>>>>>>>>>> It
>>>>>>>>>>>>>>>>>> evolves.
>>>>>>>>>>>>>>>> If
>>>>>>>>>>>>> some operators are stable as 3.8.x version suggests, the
>>>>>>>>>>>>>>>>>> library
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>> follow semantic versioning and it is not OK to make a
>>>>>>>>>>>>>>>> major
>>>>>>>>>>>>>>>>>> change
>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>> breaks semantic versioning across the entire library. It
>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>> not a
>>>>>>>>>>>>>>>> finding
>>>>>>>>>>>>> an excuse not to make a change. For me semantic versioning
>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>> backward
>>>>>>>>>>>>>>>> compatibility is more important than a proper package
>>>>>>>>>>>>>> name.
>>>>>>>>>>>>>>>>>> Thank you,
>>>>>>>>>>>>>>>> Vlad
>>>>>>>>>>>>> On 7/14/17 09:11, Thomas Weise wrote:
>>>>>>>>>>>>>>>>>>> Semantic versioning makes sense when you have a stable
>>>>>>>>>>>>>>>>>>> baseline.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> As
>>>>>>>>>>>>>>>>> long
>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>>> frequent fixes need to be made to address basic issues,
>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> makes
>>>>>>>>>>>>>>>>> no
>>>>>>>>>>>>> sense
>>>>>>>>>>>>>>>> to declare operators stable, which is why they are marked
>>>>>>>>>>>>>>>>>> evolving.
>>>>>>>>>>>>>>>>>>> Instead of finding reasons to avoid changes that should
>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> been
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> made a
>>>>>>>>>>>>>> long time ago, why not discuss what a "stable operator"
>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>> which
>>>>>>>>>>>>> ones
>>>>>>>>>>>>>>>> deserve to be in that category. Those are the ones that
>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>> get
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> backward compatibility stub.
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>> Thomas
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Fri, Jul 14, 2017 at 8:34 AM, Vlad Rozov <
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> v.rozov@datatorrent.com>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>> My preference is to agree that the next Malhar release is
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 4.0.0
>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>> make
>>>>>>>>>>>>>>>> the proposed changes when 3.8.0 goes out. Otherwise why
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> keep
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> semantic
>>>>>>>>>>>>> versioning checks on Malhar in case there is no version
>>>>>>>>>>>>>>>> compatibility.
>>>>>>>>>>>>>>>>>>>> Thank you,
>>>>>>>>>>>>>>>>>>> Vlad
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On 7/14/17 08:11, Thomas Weise wrote:
>>>>>>>>>>>>>>>>>>>>> next release is 3.8.0
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Fri, Jul 14, 2017 at 7:55 AM, Vlad Rozov <
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> v.rozov@datatorrent.com>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>> next release - 3.9.0 or 4.0.0?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thank you,
>>>>>>>>>>>>>>>>>>>>>> Vlad
>>>>>>>>>>>>>>>>>>>>>>> On 7/13/17 22:25, Thomas Weise wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> It is time to resurrect this thread and get going with
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> work.
>>>>>>>>>>>>>>>>>>>> For the next release, I will sign up to do the
>>>>>>>>>>>>>> package
>>>>>>>>>>>>>>>>>> move
>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>> Malhar:
>>>>>>>>>>>>>>> https://issues.apache.org/
>>>>>>>>>>>>>>>>> jira/browse/APEXMALHAR-2517
>>>>>>>>>>>>>>>>>>>>> In general this will be straightforward; most classes
>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>> Malhar
>>>>>>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>> marked
>>>>>>>>>>>>>>>>>> evolving and it is trivial for users to change import
>>>>>>>>>>>>>>>>>>>> statements.
>>>>>>>>>>>>>>>>>>>>>>> However,
>>>>>>>>>>>>>>>>>>>>> I would suggest to discuss if there are selected
>>>>>>>>>>>>>>>>>>> operators
>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>> worth
>>>>>>>>>>>>>>>>>> keeping a backward compatibility stub in the old
>>>>>>>>>>>>>>>>>>>> location.
>>>>>>>>>>>>>>>>>>>>>>> Here is my plan:
>>>>>>>>>>>>>>>>>>>>> 1. Relocate all classes in *lib* and *contrib* within
>>>>>>>>>>>>>>>> one
>>>>>>>>>>>>>>>>>>>>>>> PR -
>>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>>>> IDE automated work
>>>>>>>>>>>>>>>>>>>>> 2. Add backward compatibility classes, if, any in
>>>>>>>>>>>>>>>>>>>>>>>> separate
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> PR
>>>>>>>>>>>>>>>>>>>>> 3. Create PR for Megh library to reflect moved
>>>>>>>>>>>>>>> classes
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>> Thomas
>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Mar 1, 2017 at 2:24 PM, Pramod Immaneni <
>>>>>>>>>>>>>>>>>>>>>>>> pramod@datatorrent.com
>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Inline
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Feb 27, 2017 at 11:03 PM, Thomas Weise <
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> thw@apache.org>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>> -->
>>>>>>>>>>>>>>>>>> On Mon, Feb 27, 2017 at 1:27 PM, Pramod Immaneni <
>>>>>>>>>>>>>>>>>>>>>>>>> pramod@datatorrent.com
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> For malhar, for existing operators, I prefer we do
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> this as
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> part
>>>>>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> planned refactoring for breaking the monolith
>>>>>>>>>>>>>>>>>>>> modules
>>>>>>>>>>>>>>>>>>>>>>>>> into
>>>>>>>>>>>>>>>>>>>>>>> baby
>>>>>>>>>>>>> packages
>>>>>>>>>>>>>>>> and would also prefer deprecating the existing
>>>>>>>>>>>>>>>>>>> operators
>>>>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>> place.
>>>>>>>>>>>>>>> Refactor into smaller modules was discussed for
>>>>>>>>>>>>>>>>> malhar-contrib
>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>> given
>>>>>>>>>>>>>>>>>> the overall state of that module I think it is OK
>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>> defer
>>>>>>>>>>>>>>>>>>>>>>> package
>>>>>>>>>>>>> renaming
>>>>>>>>>>>>>>>> there. I do however prefer to see the package rename
>>>>>>>>>>>>>>>>>>>> addressed
>>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>> other
>>>>>>>>>>>>>>>>>> modules, especially for the main library module.
>>>>>>>>>>>>>>>>>>>> Should we consider breaking the library into
>>>>>>>>>>>>>>>>>>>>>>>>>> smaller
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> modules
>>>>>>>>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>> well,
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> file/block operators for example probably can be in
>>>>>>>>>>>>>>>>>>>>>>>>> their
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> own
>>>>>>>>>>>>>>>>>>>>>> module
>>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>>> just an organizational perspective.
>>>>>>>>>>>>>>>>>>>>>>>>> This
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> will help us achieve two things. First, the user
>>>>>>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> see
>>>>>>>>>>>>>>>>>>>>>>> all
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>> changes at once as opposed to dealing with it
>>>>>>>>>>>>>>>>>>>> twice
>>>>>>>>>>>>>>>>>>>>>>>>>> (with
>>>>>>>>>>>>>>>>>>>>>>>> package
>>>>>>>>>>>>> rename
>>>>>>>>>>>>>>>> and dependency changes) and second it will allow
>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>> smoother
>>>>>>>>>>>> transition
>>>>>>>>>>>>>> as the existing code will still work in a
>>>>>>>>>>>>>>>>>>> deprecated
>>>>>>>>>>>>>>>>>>>>>>>>> state.
>>>>>>>>>>>>>>>>>>>>>>> It
>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>> give a more consistent structure to malhar. For new
>>>>>>>>>>>>>>>>>>>>> operators,
>>>>>>>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>> go
>>>>>>>>>>>>>>>>>>>> with the new package path but we need to ensure
>>>>>>>>>>>>>>>>>>>>>>>>>> they
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>>> get
>>>>>>>>>>>>> moved
>>>>>>>>>>>>>>>> into
>>>>>>>>>>>>>>>>>> the baby packages as well.
>>>>>>>>>>>>>>>>>>>>> I think existing operators should be renamed so
>>>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> git
>>>>>>>>>>>>>>>>>>>>>>> history
>>>>>>>>>>>>> remains. A
>>>>>>>>>>>>>>>> possible solution for backward compatibility could
>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>> subsequently
>>>>>>>>>>>> add
>>>>>>>>>>>>>> empty subclasses in the previous location (for
>>>>>>>>>>>>>>>>>>>>> existing
>>>>>>>>>>>>>>>>>>>>>>>>> Thank you,
>>>>>> Vlad
>>>>>>
>>>>>>
>>>> Thank you,
>>>>
>>>> Vlad
>>>>
>>>>
>> Thank you,
>>
>> Vlad
>>
Thank you,
Vlad
Re: Java packages: legacy -> org.apache.apex
Posted by Pramod Immaneni <pr...@datatorrent.com>.
Not git history.. just from a historical perspective keeping something
around because it is not causing problems.
On Sat, Aug 19, 2017 at 6:52 PM, Vlad Rozov <vr...@apache.org> wrote:
> Do you refer to git history or something else? The git history must be
> preserved.
>
> Thank you,
>
> Vlad
>
>
> On 8/19/17 13:42, Pramod Immaneni wrote:
>
>> It will be consistent but there is something to be said about preserving
>> some history in the present where it allows itself.
>> On Sat, Aug 19, 2017 at 11:05 AM Vlad Rozov <vr...@apache.org> wrote:
>>
>> Yes, consistency.
>>>
>>> Thank you,
>>>
>>> Vlad
>>>
>>> On 8/19/17 10:57, Pramod Immaneni wrote:
>>>
>>>> Is there any significant gain by doing this?
>>>>
>>>> Thanks
>>>>
>>>> On Sat, Aug 19, 2017 at 10:36 AM, Vlad Rozov <vr...@apache.org> wrote:
>>>>
>>>> The parent level pom artifact Id is not referenced by applications, it
>>>>> exists simply to group modules together under current malhar project.
>>>>>
>>>> The
>>>
>>>> name can be changed to library-parent(root) or
>>>>>
>>>> apex-library-parent(root) or
>>>
>>>> something similar. Note that the proposal includes rename of the project
>>>>> from apex-malhar to apex-library along with the version change.
>>>>>
>>>>> Thank you,
>>>>>
>>>>> Vlad
>>>>>
>>>>>
>>>>> On 8/19/17 10:10, Pramod Immaneni wrote:
>>>>>
>>>>> Would library module then become apex-library-library? Why change the
>>>>>> malhar artifact id as the project is called apex-malhar.
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> On Sat, Aug 19, 2017 at 10:03 AM, Vlad Rozov <vr...@apache.org>
>>>>>>
>>>>> wrote:
>>>
>>>> Overall I support the proposal and this is what I was in favor of
>>>>>>
>>>>> during
>>>
>>>> the last discussion. I'd also propose changing the name of the
>>>>>>>
>>>>>> artifact
>>>
>>>> id
>>>>>>> and git repo from malhar to apex-library as part of the major version
>>>>>>> change. It will make it more consistent with the entire project and
>>>>>>>
>>>>>> will
>>>
>>>> also allow using 1.0 (1.0-SNAPSHOT) version that more closely reflects
>>>>>>> maturity of the library.
>>>>>>>
>>>>>>> Thank you,
>>>>>>>
>>>>>>> Vlad
>>>>>>>
>>>>>>> On 8/18/17 17:04, Thomas Weise wrote:
>>>>>>>
>>>>>>> The proposed change does NOT require changes to existing apps. It
>>>>>>>
>>>>>> follows
>>>
>>>> what you see for example in the JDK where older classes are retired
>>>>>>>> without
>>>>>>>> making breaking major release changes.
>>>>>>>>
>>>>>>>> Even though not required technically, it looks like the preference
>>>>>>>>
>>>>>>> is to
>>>
>>>> make this change with a 4.0 release. I will therefore propose to
>>>>>>>>
>>>>>>> change
>>>
>>>> the
>>>>>>>> master branch to 4.0-SNAPSHOT and make this and other changes deemed
>>>>>>>> worthwhile.
>>>>>>>>
>>>>>>>> To me it is more important that to newcomers the codebase does not
>>>>>>>>
>>>>>>> look
>>>
>>>> like an outdated legacy situation and therefore if I have to make a
>>>>>>>> choice
>>>>>>>> than pretty much any version number will do for that. So far I have
>>>>>>>>
>>>>>>> seen
>>>
>>>> very little interest and initiative in addressing maintenance work
>>>>>>>>
>>>>>>> like
>>>
>>>> the
>>>>>>>> packages, checkstyle, CI stability, documentation etc. So it would
>>>>>>>> be
>>>>>>>> great
>>>>>>>> if people interested would step up and contribute.
>>>>>>>>
>>>>>>>> Here is the next proposal how to proceed:
>>>>>>>>
>>>>>>>> - create release-3.8 branch for last 3.8 release right away
>>>>>>>> - update master to 4.0-SNAPSHOT as part of the open PR #664
>>>>>>>> - identify further cleanup work for master
>>>>>>>> - release 4.0
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Thomas
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Aug 17, 2017 at 4:48 PM, Amol Kekre <am...@datatorrent.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> This following pull request should be taken up in 4.0.0. See my
>>>>>>>>
>>>>>>> comments
>>>
>>>> in
>>>>>>>>> https://github.com/apache/apex-malhar/pull/664
>>>>>>>>> <https://www.google.com/url?q=https%3A%2F%2Fgithub.com%
>>>>>>>>> 2Fapache%2Fapex-malhar%2Fpull%2F664&sa=D&ust=1503020269444000&usg=
>>>>>>>>> AFQjCNHDPZIg2e6I33Jb1XjB5Ir1FkdURQ>
>>>>>>>>> https://github.com/apache/apex-malhar/pull/662
>>>>>>>>> <https://www.google.com/url?q=https%3A%2F%2Fgithub.com%
>>>>>>>>> 2Fapache%2Fapex-malhar%2Fpull%2F662&sa=D&ust=1503020269444000&usg=
>>>>>>>>> AFQjCNF7Xde7X55M3qmc8z-D5y3ZwNg7Fg>
>>>>>>>>>
>>>>>>>>> This merge should not be done without a consensus. This will
>>>>>>>>> require
>>>>>>>>> code
>>>>>>>>> changes to existing apps.
>>>>>>>>>
>>>>>>>>> Thks,
>>>>>>>>> Amol
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> E:amol@datatorrent.com | M: 510-449-2606 | Twitter: @*amolhkekre*
>>>>>>>>>
>>>>>>>>> www.datatorrent.com
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Aug 14, 2017 at 7:48 PM, Thomas Weise <th...@apache.org>
>>>>>>>>>
>>>>>>>> wrote:
>>>
>>>> Hi,
>>>>>>>>>
>>>>>>>>> I opened following PRs for the package change:
>>>>>>>>>>
>>>>>>>>>> https://github.com/apache/apex-malhar/pull/662
>>>>>>>>>>
>>>>>>>>>> Moves all classes with history retained (hence 2 commits). Also
>>>>>>>>>> contains
>>>>>>>>>> checkstyle and other mechanical changes.
>>>>>>>>>>
>>>>>>>>>> https://github.com/apache/apex-malhar/pull/664
>>>>>>>>>>
>>>>>>>>>> Adds backward compatibility jar.
>>>>>>>>>>
>>>>>>>>>> Once above PRs are merged the new artifact can be deployed and
>>>>>>>>>> introduced
>>>>>>>>>> as dependency in malhar-library.
>>>>>>>>>>
>>>>>>>>>> Please review.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Thomas
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Sun, Jul 16, 2017 at 7:04 AM, Thomas Weise <th...@apache.org>
>>>>>>>>>>
>>>>>>>>> wrote:
>>>
>>>> My original list of work items contained the b/w compatibility
>>>>>>>>>>
>>>>>>>>> aspect,
>>>
>>>> I
>>>>>>>>>> don't think there should be any confusion of whether it will be
>>>>>>>>>> covered
>>>>>>>>>>
>>>>>>>>>> here or not.
>>>>>>>>>>>
>>>>>>>>>>> The proposed shading will provide the old classes and they will
>>>>>>>>>>> be
>>>>>>>>>>>
>>>>>>>>>>> frozen
>>>>>>>>>>>
>>>>>>>>>> as of release 3.7. That's the same as making a copy of the code
>>>>>>>>>> and
>>>>>>>>>> never
>>>>>>>>>> again making changes to the original classes. This cannot be
>>>>>>>>>> accomplished
>>>>>>>>>> by using the older 3.7 release in your project because you cannot
>>>>>>>>>>
>>>>>>>>> use
>>>
>>>> 2
>>>>>>>>>>
>>>>>>>>>> different versions of Malhar in parallel unless you apply shading.
>>>>>>>>>>>
>>>>>>>>>>> The shaded artifact will only expose the com.datatorrent classes,
>>>>>>>>>>>
>>>>>>>>>> and
>>>
>>>> they
>>>>>>>>>>>
>>>>>>>>>> will be self-contained as the rest of the classes that they may
>>>>>>>>>>
>>>>>>>>> depend
>>>
>>>> on
>>>>>>>>>>>
>>>>>>>>>> are shaded. The shaded artifact does not evolve, there are not
>>>>>>>>>> more
>>>>>>>>>> changes
>>>>>>>>>>
>>>>>>>>>> to com.datatorrent classes after they are relocated in master.
>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Thomas
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Sun, Jul 16, 2017 at 2:00 AM, Pramod Immaneni <
>>>>>>>>>>>
>>>>>>>>>>> pramod@datatorrent.com
>>>>>>>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> I don't think we can limit the topic strictly to relocation
>>>>>>>>>>>
>>>>>>>>>> without
>>>
>>>> having
>>>>>>>>>>> a good b/w compatibility story or at least one that goes far
>>>>>>>>>>>
>>>>>>>>>> enough.
>>>
>>>> The shading idea sounds interesting. Why not let the shaded
>>>>>>>>>>>>
>>>>>>>>>>> version
>>>
>>>> move
>>>>>>>>>>>>
>>>>>>>>>>> forward with each release till we hit a major release. If it is
>>>>>>>>>>
>>>>>>>>> going
>>>
>>>> to
>>>>>>>>>>>
>>>>>>>>>>> remain pegged at 3.7.0, why shade in the first place as the
>>>>>>>>>> regular
>>>>>>>>>>
>>>>>>>>>> 3.7.0
>>>>>>>>>>> release would do the same job and it would be same as the loss of
>>>>>>>>>>>
>>>>>>>>>> b/w
>>>
>>>> compatibility with newer releases.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks
>>>>>>>>>>>>
>>>>>>>>>>>> On Sat, Jul 15, 2017 at 7:57 AM, Thomas Weise <th...@apache.org>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Discussing what in the future might become stable needs to be a
>>>>>>>>>>>> separate
>>>>>>>>>>>>
>>>>>>>>>>>> thread, it will be a much bigger discussion.
>>>>>>>>>>>
>>>>>>>>>>> The topic here is to relocate the packages. With a few exceptions
>>>>>>>>>>>>
>>>>>>>>>>>>> relocation won't affect the semantic versioning. Semantic
>>>>>>>>>>>>> versioning
>>>>>>>>>>>>>
>>>>>>>>>>>>> is
>>>>>>>>>>>>>
>>>>>>>>>>>> essentially not effective for Malhar because almost everything
>>>>>>>>>>> is
>>>>>>>>>>>
>>>>>>>>>>> @Evolving
>>>>>>>>>>>>
>>>>>>>>>>>> (and there are reasons for that.. -> separate topic)
>>>>>>>>>>>>
>>>>>>>>>>>>> I don't really like the idea of creating bw compatibility stubs
>>>>>>>>>>>>>
>>>>>>>>>>>> for
>>>
>>>> the
>>>>>>>>>>>>>
>>>>>>>>>>>> follow up PR. It creates even more clutter in the source tree
>>>>>>>>>>> than
>>>>>>>>>>>
>>>>>>>>>>> there
>>>>>>>>>>>>
>>>>>>>>>>>> already is and so here is an alternative suggestion:
>>>>>>>>>>>
>>>>>>>>>>> https://github.com/tweise/apex-malhar/blob/malhar37-
>>>>>>>>>>>>
>>>>>>>>>>>>> compat/shaded-malhar37/pom.xml
>>>>>>>>>>>>>
>>>>>>>>>>>>> Create a shaded artifact that provides the old
>>>>>>>>>>>>> com.datatorrent.*
>>>>>>>>>>>>>
>>>>>>>>>>>>> classes as
>>>>>>>>>>>>>
>>>>>>>>>>>> of release 3.7. Users can include that artifact if they don't
>>>>>>>>>>>>
>>>>>>>>>>> want
>>>
>>>> to
>>>>>>>>>>>>>
>>>>>>>>>>>> change import statements. At the same time they have an
>>>>>>>>>>> incentive
>>>>>>>>>>>
>>>>>>>>>> to
>>>
>>>> switch
>>>>>>>>>>>
>>>>>>>>>>>> to the relocated classes to take advantage of bug fixes and new
>>>>>>>>>>>>
>>>>>>>>>>>>> functionality.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I will work on the first PR that does the relocate. In the
>>>>>>>>>>>>> meantime,
>>>>>>>>>>>>>
>>>>>>>>>>>>> we
>>>>>>>>>>>>>
>>>>>>>>>>>> can
>>>>>>>>>>>
>>>>>>>>>>> finalize what backward compatibility support we want to provide
>>>>>>>>>>>>
>>>>>>>>>>> and
>>>
>>>> how.
>>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>
>>>>>>>>>>> Thomas
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Jul 14, 2017 at 11:33 AM, Pramod Immaneni <
>>>>>>>>>>>>>
>>>>>>>>>>>>> pramod@datatorrent.com>
>>>>>>>>>>>>>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> How about coming up with a list of @Evolving operators that we
>>>>>>>>>>>>> would
>>>>>>>>>>>>>
>>>>>>>>>>>>> like
>>>>>>>>>>>>
>>>>>>>>>>> to see in the eventual stable list and move those along with the
>>>>>>>>>>>
>>>>>>>>>>>> not
>>>>>>>>>>>>>
>>>>>>>>>>>>> @Evolving ones in org.apache.apex with b/w stubs and leave the
>>>>>>>>>>>>
>>>>>>>>>>> rest
>>>>>>>>>>>
>>>>>>>>>>>> as
>>>>>>>>>>>>
>>>>>>>>>>> they
>>>>>>>>>>
>>>>>>>>>>> are. Then have a follow up JIRA for the rest to be moved over to
>>>>>>>>>>>>
>>>>>>>>>>>>> contrib
>>>>>>>>>>>>>>
>>>>>>>>>>>>> and be deprecated.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Jul 14, 2017 at 10:37 AM, Thomas Weise <
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> thomas.weise@gmail.com>
>>>>>>>>>>>>>>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> We need to keep the discussion here on topic, if other things
>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> piled
>>>>>>>>>>>>>
>>>>>>>>>>>> on
>>>>>>>>>>>
>>>>>>>>>>>> top then nothing gets done.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Most operators today are not stable, they are @Evolving. So
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> perhaps
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> it
>>>>>>>>>>>>>
>>>>>>>>>>>> is
>>>>>>>>>>>>
>>>>>>>>>>>>> necessary to have a separate discussion about that, outside of
>>>>>>>>>>>>>
>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> thread.
>>>>>>>>>>>>>
>>>>>>>>>>>> My guess is that there are only a few operators that we could
>>>>>>>>>>>>
>>>>>>>>>>>>> declare
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> stable. Specifically, under contrib the closest one might have
>>>>>>>>>>>>>
>>>>>>>>>>>>> been
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Kafka,
>>>>>>>>>>>>>
>>>>>>>>>>>> but that is already superseded by the newer versions.
>>>>>>>>>>>>
>>>>>>>>>>>>> Thomas
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, Jul 14, 2017 at 10:21 AM, Pramod Immaneni <
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> pramod@datatorrent.com>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> We had a discussion a while back, agreed to relegate
>>>>>>>>>>>>>>> non-stable
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>
>>>>>>>>>>>>> experimental operators to contrib and also added this to
>>>>>>>>>>>
>>>>>>>>>>>> contribution
>>>>>>>>>>>>>> guidelines. We aexecuted on this and cleaned up the repo by
>>>>>>>>>>>>>> moving
>>>>>>>>>>>>>> operators deemed non-suitable for production use at that time
>>>>>>>>>>>>>>
>>>>>>>>>>>>> to
>>>>>>>>>>>>
>>>>>>>>>>>>> contrib.
>>>>>>>>>>>>>>
>>>>>>>>>>>>> So I wouldn't say the operators that are at the top level
>>>>>>>>>>>
>>>>>>>>>>>> today
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> or
>>>>>>>>>>>>>>
>>>>>>>>>>>>> the
>>>>>>>>>>
>>>>>>>>>>> ones
>>>>>>>>>>>>
>>>>>>>>>>>>> in library are 0.x.x quality. Granted, we may need to do one
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> more
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> pass
>>>>>>>>>>>>>>
>>>>>>>>>>>>> as
>>>>>>>>>>>>
>>>>>>>>>>>>> some of the operators may no longer be considered the right
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> implementations
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> with the advent of the windowed operator.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Since contrib used to be the place that housed operators
>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> required
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> third party libraries, there are some production quality
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> operators in
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> there
>>>>>>>>>>>>>> that need to be pulled out into top level modules.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I think we are in agreement that for operators that we
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> consider
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> stable,
>>>>>>>>>>>>>>
>>>>>>>>>>>>> we
>>>>>>>>>>>
>>>>>>>>>>>> should provide a b/w stub. I would add that we strongly
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> consider
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> creating
>>>>>>>>>>>>>>
>>>>>>>>>>>>> the org.apache.apex counterparts of any stable operators that
>>>>>>>>>>>
>>>>>>>>>>>> are
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>
>>>>>>>>>>>>> contrib out in top level modules and have the com.datatorrent
>>>>>>>>>>>>
>>>>>>>>>>>>> stubs
>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>
>>>>>>>>>>>>> contrib.
>>>>>>>>>>>>>
>>>>>>>>>>>>>> For the operators not considered stable, I would prefer we
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> either
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> leave
>>>>>>>>>>>>>>
>>>>>>>>>>>>> the
>>>>>>>>>>>>
>>>>>>>>>>>>> current package path as is or if we change the package path
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> then
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> create
>>>>>>>>>>>>>>
>>>>>>>>>>>>> the
>>>>>>>>>>>
>>>>>>>>>>>> b/w stub as I am not sure how widely they are in use (so, in
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> essence,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> preserve semantic versioning). It would be good if there is a
>>>>>>>>>>>>>> followup
>>>>>>>>>>>>>> JIRA
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> that takes a look at what other operators can be moved to
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> contrib
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>
>>>>>>>>>>>>> the
>>>>>>>>>>>>
>>>>>>>>>>>>> advent of the newer frameworks and understanding.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Fri, Jul 14, 2017 at 9:44 AM, Thomas Weise <
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> thw@apache.org
>>>
>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Most of the operators evolve, as is quite clearly indicated
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>> @Evolving
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> annotations. So while perhaps 0.x.x would be a more
>>>>>>>>>>>
>>>>>>>>>>>> appropriate
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> version
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> number, I don't think you can change that.
>>>>>>>>>>>>
>>>>>>>>>>>>> Thomas
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Fri, Jul 14, 2017 at 9:35 AM, Vlad Rozov <
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> v.rozov@datatorrent.com
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> If entire library is not stable, its version should be
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 0.x.x
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> there
>>>>>>>>>>>
>>>>>>>>>>>> should not be any semantic versioning enabled or implied.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> It
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> evolves.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> If
>>>>>>>>>>>
>>>>>>>>>>>> some operators are stable as 3.8.x version suggests, the
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> library
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> follow semantic versioning and it is not OK to make a
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> major
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> change
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> that
>>>>>>>>>>>
>>>>>>>>>>>> breaks semantic versioning across the entire library. It
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> not a
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> finding
>>>>>>>>>>>
>>>>>>>>>>>> an excuse not to make a change. For me semantic versioning
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> backward
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> compatibility is more important than a proper package
>>>>>>>>>>>>
>>>>>>>>>>>>> name.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thank you,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Vlad
>>>>>>>>>>>
>>>>>>>>>>>> On 7/14/17 09:11, Thomas Weise wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Semantic versioning makes sense when you have a stable
>>>>>>>>>>>>>>>>>> baseline.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> As
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> long
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> frequent fixes need to be made to address basic issues,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> makes
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> no
>>>>>>>>>>>
>>>>>>>>>>>> sense
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> to declare operators stable, which is why they are marked
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> evolving.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Instead of finding reasons to avoid changes that should
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> been
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> made a
>>>>>>>>>>>>
>>>>>>>>>>>>> long time ago, why not discuss what a "stable operator"
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> which
>>>>>>>>>>>
>>>>>>>>>>>> ones
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> deserve to be in that category. Those are the ones that
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> get
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>
>>>>>>>>>>>>> backward compatibility stub.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thomas
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Fri, Jul 14, 2017 at 8:34 AM, Vlad Rozov <
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> v.rozov@datatorrent.com>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>> My preference is to agree that the next Malhar release is
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 4.0.0
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> make
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> the proposed changes when 3.8.0 goes out. Otherwise why
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> keep
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> semantic
>>>>>>>>>>>
>>>>>>>>>>>> versioning checks on Malhar in case there is no version
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> compatibility.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thank you,
>>>>>>>>>>>>>>>>>> Vlad
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On 7/14/17 08:11, Thomas Weise wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> next release is 3.8.0
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Fri, Jul 14, 2017 at 7:55 AM, Vlad Rozov <
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> v.rozov@datatorrent.com>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> next release - 3.9.0 or 4.0.0?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thank you,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Vlad
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On 7/13/17 22:25, Thomas Weise wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> It is time to resurrect this thread and get going with
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> work.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> For the next release, I will sign up to do the
>>>>>>>>>>>>
>>>>>>>>>>>>> package
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> move
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>
>>>>>>>>>>>> Malhar:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> https://issues.apache.org/
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> jira/browse/APEXMALHAR-2517
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> In general this will be straightforward; most classes
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>
>>>>>>>>>>>> Malhar
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>
>>>>>>>>>>>>> marked
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> evolving and it is trivial for users to change import
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> statements.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> However,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I would suggest to discuss if there are selected
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> operators
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> worth
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> keeping a backward compatibility stub in the old
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> location.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Here is my plan:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> 1. Relocate all classes in *lib* and *contrib* within
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> one
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> PR -
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>
>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> IDE automated work
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> 2. Add backward compatibility classes, if, any in
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> separate
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> PR
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> 3. Create PR for Megh library to reflect moved
>>>>>>>>>>>>>
>>>>>>>>>>>>>> classes
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thomas
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Wed, Mar 1, 2017 at 2:24 PM, Pramod Immaneni <
>>>>>>>>>>>>>>>>>>>>>>> pramod@datatorrent.com
>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Inline
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Mon, Feb 27, 2017 at 11:03 PM, Thomas Weise <
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> thw@apache.org>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> -->
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Mon, Feb 27, 2017 at 1:27 PM, Pramod Immaneni <
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> pramod@datatorrent.com
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> For malhar, for existing operators, I prefer we do
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> this as
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> part
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> planned refactoring for breaking the monolith
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> modules
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> into
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> baby
>>>>>>>>>>>
>>>>>>>>>>>> packages
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> and would also prefer deprecating the existing
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> operators
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> place.
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Refactor into smaller modules was discussed for
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> malhar-contrib
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> given
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> the overall state of that module I think it is OK
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> defer
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> package
>>>>>>>>>>>
>>>>>>>>>>>> renaming
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> there. I do however prefer to see the package rename
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> addressed
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> other
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> modules, especially for the main library module.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Should we consider breaking the library into
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> smaller
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> modules
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> as
>>>>>>>>>>>
>>>>>>>>>>>> well,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> file/block operators for example probably can be in
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> their
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> own
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> module
>>>>>>>>>>>>>
>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> just an organizational perspective.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> This
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> will help us achieve two things. First, the user
>>>>>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> see
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> all
>>>>>>>>>>>
>>>>>>>>>>>> the
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> changes at once as opposed to dealing with it
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> twice
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> (with
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> package
>>>>>>>>>>>
>>>>>>>>>>>> rename
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> and dependency changes) and second it will allow
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> smoother
>>>>>>>>>>
>>>>>>>>>>> transition
>>>>>>>>>>>>
>>>>>>>>>>>>> as the existing code will still work in a
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> deprecated
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> state.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> It
>>>>>>>>>>>
>>>>>>>>>>>> will
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> give a more consistent structure to malhar. For new
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> operators,
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> go
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> with the new package path but we need to ensure
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> they
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> get
>>>>>>>>>>>
>>>>>>>>>>>> moved
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> into
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> the baby packages as well.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I think existing operators should be renamed so
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> git
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> history
>>>>>>>>>>>
>>>>>>>>>>>> remains. A
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> possible solution for backward compatibility could
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> subsequently
>>>>>>>>>>
>>>>>>>>>>> add
>>>>>>>>>>>>
>>>>>>>>>>>>> empty subclasses in the previous location (for
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> existing
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Thank you,
>>>>>
>>>>> Vlad
>>>>>
>>>>>
>>> Thank you,
>>>
>>> Vlad
>>>
>>>
>
> Thank you,
>
> Vlad
>
Re: Java packages: legacy -> org.apache.apex
Posted by Vlad Rozov <vr...@apache.org>.
Do you refer to git history or something else? The git history must be
preserved.
Thank you,
Vlad
On 8/19/17 13:42, Pramod Immaneni wrote:
> It will be consistent but there is something to be said about preserving
> some history in the present where it allows itself.
> On Sat, Aug 19, 2017 at 11:05 AM Vlad Rozov <vr...@apache.org> wrote:
>
>> Yes, consistency.
>>
>> Thank you,
>>
>> Vlad
>>
>> On 8/19/17 10:57, Pramod Immaneni wrote:
>>> Is there any significant gain by doing this?
>>>
>>> Thanks
>>>
>>> On Sat, Aug 19, 2017 at 10:36 AM, Vlad Rozov <vr...@apache.org> wrote:
>>>
>>>> The parent level pom artifact Id is not referenced by applications, it
>>>> exists simply to group modules together under current malhar project.
>> The
>>>> name can be changed to library-parent(root) or
>> apex-library-parent(root) or
>>>> something similar. Note that the proposal includes rename of the project
>>>> from apex-malhar to apex-library along with the version change.
>>>>
>>>> Thank you,
>>>>
>>>> Vlad
>>>>
>>>>
>>>> On 8/19/17 10:10, Pramod Immaneni wrote:
>>>>
>>>>> Would library module then become apex-library-library? Why change the
>>>>> malhar artifact id as the project is called apex-malhar.
>>>>>
>>>>> Thanks
>>>>>
>>>>> On Sat, Aug 19, 2017 at 10:03 AM, Vlad Rozov <vr...@apache.org>
>> wrote:
>>>>> Overall I support the proposal and this is what I was in favor of
>> during
>>>>>> the last discussion. I'd also propose changing the name of the
>> artifact
>>>>>> id
>>>>>> and git repo from malhar to apex-library as part of the major version
>>>>>> change. It will make it more consistent with the entire project and
>> will
>>>>>> also allow using 1.0 (1.0-SNAPSHOT) version that more closely reflects
>>>>>> maturity of the library.
>>>>>>
>>>>>> Thank you,
>>>>>>
>>>>>> Vlad
>>>>>>
>>>>>> On 8/18/17 17:04, Thomas Weise wrote:
>>>>>>
>>>>>> The proposed change does NOT require changes to existing apps. It
>> follows
>>>>>>> what you see for example in the JDK where older classes are retired
>>>>>>> without
>>>>>>> making breaking major release changes.
>>>>>>>
>>>>>>> Even though not required technically, it looks like the preference
>> is to
>>>>>>> make this change with a 4.0 release. I will therefore propose to
>> change
>>>>>>> the
>>>>>>> master branch to 4.0-SNAPSHOT and make this and other changes deemed
>>>>>>> worthwhile.
>>>>>>>
>>>>>>> To me it is more important that to newcomers the codebase does not
>> look
>>>>>>> like an outdated legacy situation and therefore if I have to make a
>>>>>>> choice
>>>>>>> than pretty much any version number will do for that. So far I have
>> seen
>>>>>>> very little interest and initiative in addressing maintenance work
>> like
>>>>>>> the
>>>>>>> packages, checkstyle, CI stability, documentation etc. So it would be
>>>>>>> great
>>>>>>> if people interested would step up and contribute.
>>>>>>>
>>>>>>> Here is the next proposal how to proceed:
>>>>>>>
>>>>>>> - create release-3.8 branch for last 3.8 release right away
>>>>>>> - update master to 4.0-SNAPSHOT as part of the open PR #664
>>>>>>> - identify further cleanup work for master
>>>>>>> - release 4.0
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Thomas
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Aug 17, 2017 at 4:48 PM, Amol Kekre <am...@datatorrent.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> This following pull request should be taken up in 4.0.0. See my
>> comments
>>>>>>>> in
>>>>>>>> https://github.com/apache/apex-malhar/pull/664
>>>>>>>> <https://www.google.com/url?q=https%3A%2F%2Fgithub.com%
>>>>>>>> 2Fapache%2Fapex-malhar%2Fpull%2F664&sa=D&ust=1503020269444000&usg=
>>>>>>>> AFQjCNHDPZIg2e6I33Jb1XjB5Ir1FkdURQ>
>>>>>>>> https://github.com/apache/apex-malhar/pull/662
>>>>>>>> <https://www.google.com/url?q=https%3A%2F%2Fgithub.com%
>>>>>>>> 2Fapache%2Fapex-malhar%2Fpull%2F662&sa=D&ust=1503020269444000&usg=
>>>>>>>> AFQjCNF7Xde7X55M3qmc8z-D5y3ZwNg7Fg>
>>>>>>>>
>>>>>>>> This merge should not be done without a consensus. This will require
>>>>>>>> code
>>>>>>>> changes to existing apps.
>>>>>>>>
>>>>>>>> Thks,
>>>>>>>> Amol
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> E:amol@datatorrent.com | M: 510-449-2606 | Twitter: @*amolhkekre*
>>>>>>>>
>>>>>>>> www.datatorrent.com
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Aug 14, 2017 at 7:48 PM, Thomas Weise <th...@apache.org>
>> wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>>> I opened following PRs for the package change:
>>>>>>>>>
>>>>>>>>> https://github.com/apache/apex-malhar/pull/662
>>>>>>>>>
>>>>>>>>> Moves all classes with history retained (hence 2 commits). Also
>>>>>>>>> contains
>>>>>>>>> checkstyle and other mechanical changes.
>>>>>>>>>
>>>>>>>>> https://github.com/apache/apex-malhar/pull/664
>>>>>>>>>
>>>>>>>>> Adds backward compatibility jar.
>>>>>>>>>
>>>>>>>>> Once above PRs are merged the new artifact can be deployed and
>>>>>>>>> introduced
>>>>>>>>> as dependency in malhar-library.
>>>>>>>>>
>>>>>>>>> Please review.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Thomas
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Sun, Jul 16, 2017 at 7:04 AM, Thomas Weise <th...@apache.org>
>> wrote:
>>>>>>>>> My original list of work items contained the b/w compatibility
>> aspect,
>>>>>>>>> I
>>>>>>>>> don't think there should be any confusion of whether it will be
>>>>>>>>> covered
>>>>>>>>>
>>>>>>>>>> here or not.
>>>>>>>>>>
>>>>>>>>>> The proposed shading will provide the old classes and they will be
>>>>>>>>>>
>>>>>>>>>> frozen
>>>>>>>>> as of release 3.7. That's the same as making a copy of the code and
>>>>>>>>> never
>>>>>>>>> again making changes to the original classes. This cannot be
>>>>>>>>> accomplished
>>>>>>>>> by using the older 3.7 release in your project because you cannot
>> use
>>>>>>>>> 2
>>>>>>>>>
>>>>>>>>>> different versions of Malhar in parallel unless you apply shading.
>>>>>>>>>>
>>>>>>>>>> The shaded artifact will only expose the com.datatorrent classes,
>> and
>>>>>>>>>> they
>>>>>>>>> will be self-contained as the rest of the classes that they may
>> depend
>>>>>>>>>> on
>>>>>>>>> are shaded. The shaded artifact does not evolve, there are not more
>>>>>>>>> changes
>>>>>>>>>
>>>>>>>>> to com.datatorrent classes after they are relocated in master.
>>>>>>>>>> Thanks,
>>>>>>>>>> Thomas
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Sun, Jul 16, 2017 at 2:00 AM, Pramod Immaneni <
>>>>>>>>>>
>>>>>>>>>> pramod@datatorrent.com
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> I don't think we can limit the topic strictly to relocation
>> without
>>>>>>>>>> having
>>>>>>>>>> a good b/w compatibility story or at least one that goes far
>> enough.
>>>>>>>>>>> The shading idea sounds interesting. Why not let the shaded
>> version
>>>>>>>>>>> move
>>>>>>>>> forward with each release till we hit a major release. If it is
>> going
>>>>>>>>>> to
>>>>>>>>>>
>>>>>>>>> remain pegged at 3.7.0, why shade in the first place as the regular
>>>>>>>>>
>>>>>>>>>> 3.7.0
>>>>>>>>>> release would do the same job and it would be same as the loss of
>> b/w
>>>>>>>>>>> compatibility with newer releases.
>>>>>>>>>>>
>>>>>>>>>>> Thanks
>>>>>>>>>>>
>>>>>>>>>>> On Sat, Jul 15, 2017 at 7:57 AM, Thomas Weise <th...@apache.org>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Discussing what in the future might become stable needs to be a
>>>>>>>>>>> separate
>>>>>>>>>>>
>>>>>>>>>> thread, it will be a much bigger discussion.
>>>>>>>>>>
>>>>>>>>>>> The topic here is to relocate the packages. With a few exceptions
>>>>>>>>>>>> relocation won't affect the semantic versioning. Semantic
>>>>>>>>>>>> versioning
>>>>>>>>>>>>
>>>>>>>>>>>> is
>>>>>>>>>> essentially not effective for Malhar because almost everything is
>>>>>>>>>>
>>>>>>>>>>> @Evolving
>>>>>>>>>>>
>>>>>>>>>>> (and there are reasons for that.. -> separate topic)
>>>>>>>>>>>> I don't really like the idea of creating bw compatibility stubs
>> for
>>>>>>>>>>>> the
>>>>>>>>>> follow up PR. It creates even more clutter in the source tree than
>>>>>>>>>>
>>>>>>>>>>> there
>>>>>>>>>>>
>>>>>>>>>> already is and so here is an alternative suggestion:
>>>>>>>>>>
>>>>>>>>>>> https://github.com/tweise/apex-malhar/blob/malhar37-
>>>>>>>>>>>> compat/shaded-malhar37/pom.xml
>>>>>>>>>>>>
>>>>>>>>>>>> Create a shaded artifact that provides the old com.datatorrent.*
>>>>>>>>>>>>
>>>>>>>>>>>> classes as
>>>>>>>>>>> of release 3.7. Users can include that artifact if they don't
>> want
>>>>>>>>>>>> to
>>>>>>>>>> change import statements. At the same time they have an incentive
>> to
>>>>>>>>>> switch
>>>>>>>>>>> to the relocated classes to take advantage of bug fixes and new
>>>>>>>>>>>> functionality.
>>>>>>>>>>>>
>>>>>>>>>>>> I will work on the first PR that does the relocate. In the
>>>>>>>>>>>> meantime,
>>>>>>>>>>>>
>>>>>>>>>>>> we
>>>>>>>>>> can
>>>>>>>>>>
>>>>>>>>>>> finalize what backward compatibility support we want to provide
>> and
>>>>>>>>>>>> how.
>>>>>>>>>> Thanks,
>>>>>>>>>>
>>>>>>>>>>> Thomas
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Jul 14, 2017 at 11:33 AM, Pramod Immaneni <
>>>>>>>>>>>>
>>>>>>>>>>>> pramod@datatorrent.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>> How about coming up with a list of @Evolving operators that we
>>>>>>>>>>>> would
>>>>>>>>>>>>
>>>>>>>>>>> like
>>>>>>>>>> to see in the eventual stable list and move those along with the
>>>>>>>>>>>> not
>>>>>>>>>>>>
>>>>>>>>>>> @Evolving ones in org.apache.apex with b/w stubs and leave the
>>>>>>>>>> rest
>>>>>>>>>>> as
>>>>>>>>> they
>>>>>>>>>>> are. Then have a follow up JIRA for the rest to be moved over to
>>>>>>>>>>>>> contrib
>>>>>>>>>>>> and be deprecated.
>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Jul 14, 2017 at 10:37 AM, Thomas Weise <
>>>>>>>>>>>>>
>>>>>>>>>>>>> thomas.weise@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> We need to keep the discussion here on topic, if other things
>>>>>>>>>>>>> are
>>>>>>>>>>>>>
>>>>>>>>>>>> piled
>>>>>>>>>> on
>>>>>>>>>>>>> top then nothing gets done.
>>>>>>>>>>>>>> Most operators today are not stable, they are @Evolving. So
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> perhaps
>>>>>>>>>>>> it
>>>>>>>>>>> is
>>>>>>>>>>>> necessary to have a separate discussion about that, outside of
>>>>>>>>>>>>> this
>>>>>>>>>>>>>
>>>>>>>>>>>> thread.
>>>>>>>>>>> My guess is that there are only a few operators that we could
>>>>>>>>>>>>>> declare
>>>>>>>>>>>> stable. Specifically, under contrib the closest one might have
>>>>>>>>>>>>
>>>>>>>>>>>>> been
>>>>>>>>>>>>>
>>>>>>>>>>>> Kafka,
>>>>>>>>>>> but that is already superseded by the newer versions.
>>>>>>>>>>>>>> Thomas
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Jul 14, 2017 at 10:21 AM, Pramod Immaneni <
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> pramod@datatorrent.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> We had a discussion a while back, agreed to relegate
>>>>>>>>>>>>>> non-stable
>>>>>>>>>>>>>>
>>>>>>>>>>>>> and
>>>>>>>>>> experimental operators to contrib and also added this to
>>>>>>>>>>>>> contribution
>>>>>>>>>>>>> guidelines. We aexecuted on this and cleaned up the repo by
>>>>>>>>>>>>> moving
>>>>>>>>>>>>> operators deemed non-suitable for production use at that time
>>>>>>>>>>> to
>>>>>>>>>>>>> contrib.
>>>>>>>>>> So I wouldn't say the operators that are at the top level
>>>>>>>>>>>>>> today
>>>>>>>>>>>>>>
>>>>>>>>>>>>> or
>>>>>>>>> the
>>>>>>>>>>> ones
>>>>>>>>>>>>>> in library are 0.x.x quality. Granted, we may need to do one
>>>>>>>>>>>>>>> more
>>>>>>>>>>>>> pass
>>>>>>>>>>> as
>>>>>>>>>>>>> some of the operators may no longer be considered the right
>>>>>>>>>>>>>> implementations
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> with the advent of the windowed operator.
>>>>>>>>>>>>>>> Since contrib used to be the place that housed operators that
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> required
>>>>>>>>>>>>> third party libraries, there are some production quality
>>>>>>>>>>>>>
>>>>>>>>>>>>>> operators in
>>>>>>>>>>>>>>
>>>>>>>>>>>>> there
>>>>>>>>>>>>> that need to be pulled out into top level modules.
>>>>>>>>>>>>>>> I think we are in agreement that for operators that we
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> consider
>>>>>>>>>>>>> stable,
>>>>>>>>>> we
>>>>>>>>>>>>>> should provide a b/w stub. I would add that we strongly
>>>>>>>>>>>>>>> consider
>>>>>>>>>>>>> creating
>>>>>>>>>> the org.apache.apex counterparts of any stable operators that
>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>
>>>>>>>>>>>>> in
>>>>>>>>>>> contrib out in top level modules and have the com.datatorrent
>>>>>>>>>>>>> stubs
>>>>>>>>>>>>> in
>>>>>>>>>>>> contrib.
>>>>>>>>>>>>>> For the operators not considered stable, I would prefer we
>>>>>>>>>>>>>>> either
>>>>>>>>>>>>> leave
>>>>>>>>>>> the
>>>>>>>>>>>>>> current package path as is or if we change the package path
>>>>>>>>>>>>>>> then
>>>>>>>>>>>>> create
>>>>>>>>>> the
>>>>>>>>>>>>>> b/w stub as I am not sure how widely they are in use (so, in
>>>>>>>>>>>>>>> essence,
>>>>>>>>>>>>> preserve semantic versioning). It would be good if there is a
>>>>>>>>>>>>> followup
>>>>>>>>>>>>> JIRA
>>>>>>>>>>>>>
>>>>>>>>>>>>>> that takes a look at what other operators can be moved to
>>>>>>>>>>>>>>> contrib
>>>>>>>>>>>>> with
>>>>>>>>>>> the
>>>>>>>>>>>>>> advent of the newer frameworks and understanding.
>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, Jul 14, 2017 at 9:44 AM, Thomas Weise <
>> thw@apache.org
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>> Most of the operators evolve, as is quite clearly indicated
>>>>>>>>>>>>>
>>>>>>>>>>>>>> by
>>>>>>>>>>>>>> @Evolving
>>>>>>>>>> annotations. So while perhaps 0.x.x would be a more
>>>>>>>>>>>>>>> appropriate
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> version
>>>>>>>>>>> number, I don't think you can change that.
>>>>>>>>>>>>>>> Thomas
>>>>>>>>>>>>>>>> On Fri, Jul 14, 2017 at 9:35 AM, Vlad Rozov <
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> v.rozov@datatorrent.com
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> If entire library is not stable, its version should be
>>>>>>>>>>>>>>>> 0.x.x
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> and
>>>>>>>>>> there
>>>>>>>>>>>>> should not be any semantic versioning enabled or implied.
>>>>>>>>>>>>>>>> It
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> evolves.
>>>>>>>>>> If
>>>>>>>>>>>>>>> some operators are stable as 3.8.x version suggests, the
>>>>>>>>>>>>>>>> library
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> should
>>>>>>>>>>>>> follow semantic versioning and it is not OK to make a
>>>>>>>>>>>>>>>> major
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> change
>>>>>>>>>> that
>>>>>>>>>>>>>> breaks semantic versioning across the entire library. It
>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> not a
>>>>>>>>>> finding
>>>>>>>>>>>>> an excuse not to make a change. For me semantic versioning
>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> backward
>>>>>>>>>>> compatibility is more important than a proper package
>>>>>>>>>>>>>>>> name.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thank you,
>>>>>>>>>> Vlad
>>>>>>>>>>>>>>>>> On 7/14/17 09:11, Thomas Weise wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Semantic versioning makes sense when you have a stable
>>>>>>>>>>>>>>>>> baseline.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> As
>>>>>>>>>>>>> long
>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>> frequent fixes need to be made to address basic issues,
>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>> makes
>>>>>>>>>> no
>>>>>>>>>>>>> sense
>>>>>>>>>>>>>>> to declare operators stable, which is why they are marked
>>>>>>>>>>>>>>>>> evolving.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Instead of finding reasons to avoid changes that should
>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>> been
>>>>>>>>>>> made a
>>>>>>>>>>>>>> long time ago, why not discuss what a "stable operator"
>>>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> and
>>>>>>>>>> which
>>>>>>>>>>>>> ones
>>>>>>>>>>>>>>> deserve to be in that category. Those are the ones that
>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> get
>>>>>>>>>>> the
>>>>>>>>>>>>> backward compatibility stub.
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>> Thomas
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Fri, Jul 14, 2017 at 8:34 AM, Vlad Rozov <
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> v.rozov@datatorrent.com>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>> My preference is to agree that the next Malhar release is
>>>>>>>>>>>>>>>>>> 4.0.0
>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>> make
>>>>>>>>>>>>>>> the proposed changes when 3.8.0 goes out. Otherwise why
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>> keep
>>>>>>>>>> semantic
>>>>>>>>>>>>> versioning checks on Malhar in case there is no version
>>>>>>>>>>>>>>>>>> compatibility.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thank you,
>>>>>>>>>>>>>>>>> Vlad
>>>>>>>>>>>>>>>>>>> On 7/14/17 08:11, Thomas Weise wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> next release is 3.8.0
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Fri, Jul 14, 2017 at 7:55 AM, Vlad Rozov <
>>>>>>>>>>>>>>>>>>>> v.rozov@datatorrent.com>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>> next release - 3.9.0 or 4.0.0?
>>>>>>>>>>>>>>>>>>>> Thank you,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Vlad
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On 7/13/17 22:25, Thomas Weise wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> It is time to resurrect this thread and get going with
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> work.
>>>>>>>>>>> For the next release, I will sign up to do the
>>>>>>>>>>>>>>> package
>>>>>>>>>>>>>>>>>>>> move
>>>>>>>>>> in
>>>>>>>>>>>> Malhar:
>>>>>>>>>>>>>> https://issues.apache.org/
>>>>>>>>>>>>>>>>>> jira/browse/APEXMALHAR-2517
>>>>>>>>>>>>>>>>>>>> In general this will be straightforward; most classes
>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>> Malhar
>>>>>>>>>>> are
>>>>>>>>>>>>>>> marked
>>>>>>>>>>>>>>>>> evolving and it is trivial for users to change import
>>>>>>>>>>>>>>>>>>>>>> statements.
>>>>>>>>>>>>>>>>>>>> However,
>>>>>>>>>>>>>>>> I would suggest to discuss if there are selected
>>>>>>>>>>>>>>>>>>>>>> operators
>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>> are
>>>>>>>>>>>>>>> worth
>>>>>>>>>>>>>>>>> keeping a backward compatibility stub in the old
>>>>>>>>>>>>>>>>>>>>>> location.
>>>>>>>>>>>>>>>>>>>> Here is my plan:
>>>>>>>>>>>>> 1. Relocate all classes in *lib* and *contrib* within
>>>>>>>>>>>>>>>>>>>>>> one
>>>>>>>>>>>>>>>>>>>> PR -
>>>>>>>>>>> this
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>>> IDE automated work
>>>>>>>>>>>>>>>>>>>>>> 2. Add backward compatibility classes, if, any in
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> separate
>>>>>>>>>>>>>>>>>>>> PR
>>>>>>>>>>>> 3. Create PR for Megh library to reflect moved
>>>>>>>>>>>>>> classes
>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>> Thomas
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Wed, Mar 1, 2017 at 2:24 PM, Pramod Immaneni <
>>>>>>>>>>>>>>>>>>>>>> pramod@datatorrent.com
>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Inline
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Mon, Feb 27, 2017 at 11:03 PM, Thomas Weise <
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> thw@apache.org>
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> -->
>>>>>>>>>>>>>>>>>>>>>>> On Mon, Feb 27, 2017 at 1:27 PM, Pramod Immaneni <
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> pramod@datatorrent.com
>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> For malhar, for existing operators, I prefer we do
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> this as
>>>>>>>>>>>>>>>>>>>>>> part
>>>>>>>>>>>>> of
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> planned refactoring for breaking the monolith
>>>>>>>>>>>>>>>>>>>>>>>> modules
>>>>>>>>>>>>>>>>>>>>>> into
>>>>>>>>>> baby
>>>>>>>>>>>>> packages
>>>>>>>>>>>>>>>> and would also prefer deprecating the existing
>>>>>>>>>>>>>>>>>>>>>>>> operators
>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>> place.
>>>>>>>>>>>>>> Refactor into smaller modules was discussed for
>>>>>>>>>>>>>>>>> malhar-contrib
>>>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>> given
>>>>>>>>>>>>>>>>> the overall state of that module I think it is OK
>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>> defer
>>>>>>>>>> package
>>>>>>>>>>>>> renaming
>>>>>>>>>>>>>>>>> there. I do however prefer to see the package rename
>>>>>>>>>>>>>>>>>>>>>>> addressed
>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>> other
>>>>>>>>>>>>>>>>> modules, especially for the main library module.
>>>>>>>>>>>>>>>>>>>>>>>> Should we consider breaking the library into
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> smaller
>>>>>>>>>>>>>>>>>>>>>> modules
>>>>>>>>>> as
>>>>>>>>>>>>>> well,
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> file/block operators for example probably can be in
>>>>>>>>>>>>>>>>>>>>>>> their
>>>>>>>>>>>>>>>>>>>>> own
>>>>>>>>>>>> module
>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>>>> just an organizational perspective.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> This
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> will help us achieve two things. First, the user
>>>>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> see
>>>>>>>>>> all
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>> changes at once as opposed to dealing with it
>>>>>>>>>>>>>>>>>>>>>>>>> twice
>>>>>>>>>>>>>>>>>>>>>>> (with
>>>>>>>>>> package
>>>>>>>>>>>>> rename
>>>>>>>>>>>>>>>>> and dependency changes) and second it will allow
>>>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>>> a
>>>>>>>>> smoother
>>>>>>>>>>> transition
>>>>>>>>>>>>>>>> as the existing code will still work in a
>>>>>>>>>>>>>>>>>>>>>>>> deprecated
>>>>>>>>>>>>>>>>>>>>>> state.
>>>>>>>>>> It
>>>>>>>>>>>>> will
>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>> give a more consistent structure to malhar. For new
>>>>>>>>>>>>>>>>>>>>>>>> operators,
>>>>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>> go
>>>>>>>>>>>>>>>>>>>>>>>> with the new package path but we need to ensure
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> they
>>>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>> get
>>>>>>>>>>>>> moved
>>>>>>>>>>>>>>> into
>>>>>>>>>>>>>>>>>> the baby packages as well.
>>>>>>>>>>>>>>>>>>>>>>>> I think existing operators should be renamed so
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>>> git
>>>>>>>>>> history
>>>>>>>>>>>>> remains. A
>>>>>>>>>>>>>>>> possible solution for backward compatibility could
>>>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>> subsequently
>>>>>>>>>>> add
>>>>>>>>>>>>>>>>>> empty subclasses in the previous location (for
>>>>>>>>>>>>>>>>>>>>>>>> existing
>>>> Thank you,
>>>>
>>>> Vlad
>>>>
>>
>> Thank you,
>>
>> Vlad
>>
Thank you,
Vlad
Re: Java packages: legacy -> org.apache.apex
Posted by Pramod Immaneni <pr...@datatorrent.com>.
It will be consistent but there is something to be said about preserving
some history in the present where it allows itself.
On Sat, Aug 19, 2017 at 11:05 AM Vlad Rozov <vr...@apache.org> wrote:
> Yes, consistency.
>
> Thank you,
>
> Vlad
>
> On 8/19/17 10:57, Pramod Immaneni wrote:
> > Is there any significant gain by doing this?
> >
> > Thanks
> >
> > On Sat, Aug 19, 2017 at 10:36 AM, Vlad Rozov <vr...@apache.org> wrote:
> >
> >> The parent level pom artifact Id is not referenced by applications, it
> >> exists simply to group modules together under current malhar project.
> The
> >> name can be changed to library-parent(root) or
> apex-library-parent(root) or
> >> something similar. Note that the proposal includes rename of the project
> >> from apex-malhar to apex-library along with the version change.
> >>
> >> Thank you,
> >>
> >> Vlad
> >>
> >>
> >> On 8/19/17 10:10, Pramod Immaneni wrote:
> >>
> >>> Would library module then become apex-library-library? Why change the
> >>> malhar artifact id as the project is called apex-malhar.
> >>>
> >>> Thanks
> >>>
> >>> On Sat, Aug 19, 2017 at 10:03 AM, Vlad Rozov <vr...@apache.org>
> wrote:
> >>>
> >>> Overall I support the proposal and this is what I was in favor of
> during
> >>>> the last discussion. I'd also propose changing the name of the
> artifact
> >>>> id
> >>>> and git repo from malhar to apex-library as part of the major version
> >>>> change. It will make it more consistent with the entire project and
> will
> >>>> also allow using 1.0 (1.0-SNAPSHOT) version that more closely reflects
> >>>> maturity of the library.
> >>>>
> >>>> Thank you,
> >>>>
> >>>> Vlad
> >>>>
> >>>> On 8/18/17 17:04, Thomas Weise wrote:
> >>>>
> >>>> The proposed change does NOT require changes to existing apps. It
> follows
> >>>>> what you see for example in the JDK where older classes are retired
> >>>>> without
> >>>>> making breaking major release changes.
> >>>>>
> >>>>> Even though not required technically, it looks like the preference
> is to
> >>>>> make this change with a 4.0 release. I will therefore propose to
> change
> >>>>> the
> >>>>> master branch to 4.0-SNAPSHOT and make this and other changes deemed
> >>>>> worthwhile.
> >>>>>
> >>>>> To me it is more important that to newcomers the codebase does not
> look
> >>>>> like an outdated legacy situation and therefore if I have to make a
> >>>>> choice
> >>>>> than pretty much any version number will do for that. So far I have
> seen
> >>>>> very little interest and initiative in addressing maintenance work
> like
> >>>>> the
> >>>>> packages, checkstyle, CI stability, documentation etc. So it would be
> >>>>> great
> >>>>> if people interested would step up and contribute.
> >>>>>
> >>>>> Here is the next proposal how to proceed:
> >>>>>
> >>>>> - create release-3.8 branch for last 3.8 release right away
> >>>>> - update master to 4.0-SNAPSHOT as part of the open PR #664
> >>>>> - identify further cleanup work for master
> >>>>> - release 4.0
> >>>>>
> >>>>> Thanks,
> >>>>> Thomas
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Thu, Aug 17, 2017 at 4:48 PM, Amol Kekre <am...@datatorrent.com>
> >>>>> wrote:
> >>>>>
> >>>>> This following pull request should be taken up in 4.0.0. See my
> comments
> >>>>>
> >>>>>> in
> >>>>>> https://github.com/apache/apex-malhar/pull/664
> >>>>>> <https://www.google.com/url?q=https%3A%2F%2Fgithub.com%
> >>>>>> 2Fapache%2Fapex-malhar%2Fpull%2F664&sa=D&ust=1503020269444000&usg=
> >>>>>> AFQjCNHDPZIg2e6I33Jb1XjB5Ir1FkdURQ>
> >>>>>> https://github.com/apache/apex-malhar/pull/662
> >>>>>> <https://www.google.com/url?q=https%3A%2F%2Fgithub.com%
> >>>>>> 2Fapache%2Fapex-malhar%2Fpull%2F662&sa=D&ust=1503020269444000&usg=
> >>>>>> AFQjCNF7Xde7X55M3qmc8z-D5y3ZwNg7Fg>
> >>>>>>
> >>>>>> This merge should not be done without a consensus. This will require
> >>>>>> code
> >>>>>> changes to existing apps.
> >>>>>>
> >>>>>> Thks,
> >>>>>> Amol
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> E:amol@datatorrent.com | M: 510-449-2606 | Twitter: @*amolhkekre*
> >>>>>>
> >>>>>> www.datatorrent.com
> >>>>>>
> >>>>>>
> >>>>>> On Mon, Aug 14, 2017 at 7:48 PM, Thomas Weise <th...@apache.org>
> wrote:
> >>>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>>> I opened following PRs for the package change:
> >>>>>>>
> >>>>>>> https://github.com/apache/apex-malhar/pull/662
> >>>>>>>
> >>>>>>> Moves all classes with history retained (hence 2 commits). Also
> >>>>>>> contains
> >>>>>>> checkstyle and other mechanical changes.
> >>>>>>>
> >>>>>>> https://github.com/apache/apex-malhar/pull/664
> >>>>>>>
> >>>>>>> Adds backward compatibility jar.
> >>>>>>>
> >>>>>>> Once above PRs are merged the new artifact can be deployed and
> >>>>>>> introduced
> >>>>>>> as dependency in malhar-library.
> >>>>>>>
> >>>>>>> Please review.
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>> Thomas
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On Sun, Jul 16, 2017 at 7:04 AM, Thomas Weise <th...@apache.org>
> wrote:
> >>>>>>>
> >>>>>>> My original list of work items contained the b/w compatibility
> aspect,
> >>>>>>> I
> >>>>>>> don't think there should be any confusion of whether it will be
> >>>>>>> covered
> >>>>>>>
> >>>>>>>> here or not.
> >>>>>>>>
> >>>>>>>> The proposed shading will provide the old classes and they will be
> >>>>>>>>
> >>>>>>>> frozen
> >>>>>>> as of release 3.7. That's the same as making a copy of the code and
> >>>>>>> never
> >>>>>>> again making changes to the original classes. This cannot be
> >>>>>>> accomplished
> >>>>>>> by using the older 3.7 release in your project because you cannot
> use
> >>>>>>> 2
> >>>>>>>
> >>>>>>>> different versions of Malhar in parallel unless you apply shading.
> >>>>>>>>
> >>>>>>>> The shaded artifact will only expose the com.datatorrent classes,
> and
> >>>>>>>>
> >>>>>>>> they
> >>>>>>> will be self-contained as the rest of the classes that they may
> depend
> >>>>>>>> on
> >>>>>>> are shaded. The shaded artifact does not evolve, there are not more
> >>>>>>> changes
> >>>>>>>
> >>>>>>> to com.datatorrent classes after they are relocated in master.
> >>>>>>>> Thanks,
> >>>>>>>> Thomas
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Sun, Jul 16, 2017 at 2:00 AM, Pramod Immaneni <
> >>>>>>>>
> >>>>>>>> pramod@datatorrent.com
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> I don't think we can limit the topic strictly to relocation
> without
> >>>>>>>> having
> >>>>>>>> a good b/w compatibility story or at least one that goes far
> enough.
> >>>>>>>>
> >>>>>>>>> The shading idea sounds interesting. Why not let the shaded
> version
> >>>>>>>>>
> >>>>>>>>> move
> >>>>>>> forward with each release till we hit a major release. If it is
> going
> >>>>>>>
> >>>>>>>> to
> >>>>>>>>
> >>>>>>> remain pegged at 3.7.0, why shade in the first place as the regular
> >>>>>>>
> >>>>>>>> 3.7.0
> >>>>>>>> release would do the same job and it would be same as the loss of
> b/w
> >>>>>>>>
> >>>>>>>>> compatibility with newer releases.
> >>>>>>>>>
> >>>>>>>>> Thanks
> >>>>>>>>>
> >>>>>>>>> On Sat, Jul 15, 2017 at 7:57 AM, Thomas Weise <th...@apache.org>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>> Discussing what in the future might become stable needs to be a
> >>>>>>>>> separate
> >>>>>>>>>
> >>>>>>>> thread, it will be a much bigger discussion.
> >>>>>>>>
> >>>>>>>>> The topic here is to relocate the packages. With a few exceptions
> >>>>>>>>>> relocation won't affect the semantic versioning. Semantic
> >>>>>>>>>> versioning
> >>>>>>>>>>
> >>>>>>>>>> is
> >>>>>>>> essentially not effective for Malhar because almost everything is
> >>>>>>>>
> >>>>>>>>> @Evolving
> >>>>>>>>>
> >>>>>>>>> (and there are reasons for that.. -> separate topic)
> >>>>>>>>>> I don't really like the idea of creating bw compatibility stubs
> for
> >>>>>>>>>>
> >>>>>>>>>> the
> >>>>>>>> follow up PR. It creates even more clutter in the source tree than
> >>>>>>>>
> >>>>>>>>> there
> >>>>>>>>>
> >>>>>>>> already is and so here is an alternative suggestion:
> >>>>>>>>
> >>>>>>>>> https://github.com/tweise/apex-malhar/blob/malhar37-
> >>>>>>>>>> compat/shaded-malhar37/pom.xml
> >>>>>>>>>>
> >>>>>>>>>> Create a shaded artifact that provides the old com.datatorrent.*
> >>>>>>>>>>
> >>>>>>>>>> classes as
> >>>>>>>>> of release 3.7. Users can include that artifact if they don't
> want
> >>>>>>>>>> to
> >>>>>>>> change import statements. At the same time they have an incentive
> to
> >>>>>>>> switch
> >>>>>>>>> to the relocated classes to take advantage of bug fixes and new
> >>>>>>>>>> functionality.
> >>>>>>>>>>
> >>>>>>>>>> I will work on the first PR that does the relocate. In the
> >>>>>>>>>> meantime,
> >>>>>>>>>>
> >>>>>>>>>> we
> >>>>>>>> can
> >>>>>>>>
> >>>>>>>>> finalize what backward compatibility support we want to provide
> and
> >>>>>>>>>> how.
> >>>>>>>> Thanks,
> >>>>>>>>
> >>>>>>>>> Thomas
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Fri, Jul 14, 2017 at 11:33 AM, Pramod Immaneni <
> >>>>>>>>>>
> >>>>>>>>>> pramod@datatorrent.com>
> >>>>>>>>> wrote:
> >>>>>>>>>> How about coming up with a list of @Evolving operators that we
> >>>>>>>>>> would
> >>>>>>>>>>
> >>>>>>>>> like
> >>>>>>>> to see in the eventual stable list and move those along with the
> >>>>>>>>>> not
> >>>>>>>>>>
> >>>>>>>>> @Evolving ones in org.apache.apex with b/w stubs and leave the
> >>>>>>>> rest
> >>>>>>>>> as
> >>>>>>> they
> >>>>>>>>> are. Then have a follow up JIRA for the rest to be moved over to
> >>>>>>>>>>> contrib
> >>>>>>>>>> and be deprecated.
> >>>>>>>>>>
> >>>>>>>>>>> Thanks
> >>>>>>>>>>>
> >>>>>>>>>>> On Fri, Jul 14, 2017 at 10:37 AM, Thomas Weise <
> >>>>>>>>>>>
> >>>>>>>>>>> thomas.weise@gmail.com>
> >>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> We need to keep the discussion here on topic, if other things
> >>>>>>>>>>> are
> >>>>>>>>>>>
> >>>>>>>>>> piled
> >>>>>>>> on
> >>>>>>>>>>> top then nothing gets done.
> >>>>>>>>>>>> Most operators today are not stable, they are @Evolving. So
> >>>>>>>>>>>>
> >>>>>>>>>>>> perhaps
> >>>>>>>>>> it
> >>>>>>>>> is
> >>>>>>>>>> necessary to have a separate discussion about that, outside of
> >>>>>>>>>>> this
> >>>>>>>>>>>
> >>>>>>>>>> thread.
> >>>>>>>>> My guess is that there are only a few operators that we could
> >>>>>>>>>>>> declare
> >>>>>>>>>> stable. Specifically, under contrib the closest one might have
> >>>>>>>>>>
> >>>>>>>>>>> been
> >>>>>>>>>>>
> >>>>>>>>>> Kafka,
> >>>>>>>>> but that is already superseded by the newer versions.
> >>>>>>>>>>>> Thomas
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Fri, Jul 14, 2017 at 10:21 AM, Pramod Immaneni <
> >>>>>>>>>>>>
> >>>>>>>>>>>> pramod@datatorrent.com>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>> We had a discussion a while back, agreed to relegate
> >>>>>>>>>>>> non-stable
> >>>>>>>>>>>>
> >>>>>>>>>>> and
> >>>>>>>> experimental operators to contrib and also added this to
> >>>>>>>>>>> contribution
> >>>>>>>>>>> guidelines. We aexecuted on this and cleaned up the repo by
> >>>>>>>>>>> moving
> >>>>>>>>>>> operators deemed non-suitable for production use at that time
> >>>>>>>>> to
> >>>>>>>>>>> contrib.
> >>>>>>>> So I wouldn't say the operators that are at the top level
> >>>>>>>>>>>> today
> >>>>>>>>>>>>
> >>>>>>>>>>> or
> >>>>>>> the
> >>>>>>>>> ones
> >>>>>>>>>>>> in library are 0.x.x quality. Granted, we may need to do one
> >>>>>>>>>>>>> more
> >>>>>>>>>>> pass
> >>>>>>>>> as
> >>>>>>>>>>> some of the operators may no longer be considered the right
> >>>>>>>>>>>> implementations
> >>>>>>>>>>>>
> >>>>>>>>>>>> with the advent of the windowed operator.
> >>>>>>>>>>>>> Since contrib used to be the place that housed operators that
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> required
> >>>>>>>>>>> third party libraries, there are some production quality
> >>>>>>>>>>>
> >>>>>>>>>>>> operators in
> >>>>>>>>>>>>
> >>>>>>>>>>> there
> >>>>>>>>>>> that need to be pulled out into top level modules.
> >>>>>>>>>>>>> I think we are in agreement that for operators that we
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> consider
> >>>>>>>>>>> stable,
> >>>>>>>> we
> >>>>>>>>>>>> should provide a b/w stub. I would add that we strongly
> >>>>>>>>>>>>> consider
> >>>>>>>>>>> creating
> >>>>>>>> the org.apache.apex counterparts of any stable operators that
> >>>>>>>>>>>> are
> >>>>>>>>>>>>
> >>>>>>>>>>> in
> >>>>>>>>> contrib out in top level modules and have the com.datatorrent
> >>>>>>>>>>> stubs
> >>>>>>>>>>> in
> >>>>>>>>>> contrib.
> >>>>>>>>>>>> For the operators not considered stable, I would prefer we
> >>>>>>>>>>>>> either
> >>>>>>>>>>> leave
> >>>>>>>>> the
> >>>>>>>>>>>> current package path as is or if we change the package path
> >>>>>>>>>>>>> then
> >>>>>>>>>>> create
> >>>>>>>> the
> >>>>>>>>>>>> b/w stub as I am not sure how widely they are in use (so, in
> >>>>>>>>>>>>> essence,
> >>>>>>>>>>> preserve semantic versioning). It would be good if there is a
> >>>>>>>>>>> followup
> >>>>>>>>>>> JIRA
> >>>>>>>>>>>
> >>>>>>>>>>>> that takes a look at what other operators can be moved to
> >>>>>>>>>>>>> contrib
> >>>>>>>>>>> with
> >>>>>>>>> the
> >>>>>>>>>>>> advent of the newer frameworks and understanding.
> >>>>>>>>>>>>> Thanks
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Fri, Jul 14, 2017 at 9:44 AM, Thomas Weise <
> thw@apache.org
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>> Most of the operators evolve, as is quite clearly indicated
> >>>>>>>>>>>
> >>>>>>>>>>>> by
> >>>>>>>>>>>> @Evolving
> >>>>>>>> annotations. So while perhaps 0.x.x would be a more
> >>>>>>>>>>>>> appropriate
> >>>>>>>>>>>>>
> >>>>>>>>>>>> version
> >>>>>>>>> number, I don't think you can change that.
> >>>>>>>>>>>>> Thomas
> >>>>>>>>>>>>>> On Fri, Jul 14, 2017 at 9:35 AM, Vlad Rozov <
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> v.rozov@datatorrent.com
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>> If entire library is not stable, its version should be
> >>>>>>>>>>>>>> 0.x.x
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>> and
> >>>>>>>> there
> >>>>>>>>>>> should not be any semantic versioning enabled or implied.
> >>>>>>>>>>>>>> It
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>> evolves.
> >>>>>>>> If
> >>>>>>>>>>>>> some operators are stable as 3.8.x version suggests, the
> >>>>>>>>>>>>>> library
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>> should
> >>>>>>>>>>> follow semantic versioning and it is not OK to make a
> >>>>>>>>>>>>>> major
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>> change
> >>>>>>>> that
> >>>>>>>>>>>> breaks semantic versioning across the entire library. It
> >>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>> not a
> >>>>>>>> finding
> >>>>>>>>>>> an excuse not to make a change. For me semantic versioning
> >>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>> backward
> >>>>>>>>> compatibility is more important than a proper package
> >>>>>>>>>>>>>> name.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>> Thank you,
> >>>>>>>> Vlad
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On 7/14/17 09:11, Thomas Weise wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Semantic versioning makes sense when you have a stable
> >>>>>>>>>>>>>>> baseline.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>> As
> >>>>>>>>>>> long
> >>>>>>>>>>>>> as
> >>>>>>>>>>>>>>> frequent fixes need to be made to address basic issues,
> >>>>>>>>>>>>>>>> it
> >>>>>>>>>>>>>> makes
> >>>>>>>> no
> >>>>>>>>>>> sense
> >>>>>>>>>>>>> to declare operators stable, which is why they are marked
> >>>>>>>>>>>>>>> evolving.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Instead of finding reasons to avoid changes that should
> >>>>>>>>>>>>> have
> >>>>>>>>>>>>>> been
> >>>>>>>>> made a
> >>>>>>>>>>>> long time ago, why not discuss what a "stable operator"
> >>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>> and
> >>>>>>>> which
> >>>>>>>>>>> ones
> >>>>>>>>>>>>> deserve to be in that category. Those are the ones that
> >>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>> get
> >>>>>>>>> the
> >>>>>>>>>>> backward compatibility stub.
> >>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>> Thomas
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Fri, Jul 14, 2017 at 8:34 AM, Vlad Rozov <
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> v.rozov@datatorrent.com>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>> My preference is to agree that the next Malhar release is
> >>>>>>>>>>>>>>>> 4.0.0
> >>>>>>>>>>>>>> and
> >>>>>>>>>>> make
> >>>>>>>>>>>>> the proposed changes when 3.8.0 goes out. Otherwise why
> >>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>> keep
> >>>>>>>> semantic
> >>>>>>>>>>> versioning checks on Malhar in case there is no version
> >>>>>>>>>>>>>>>> compatibility.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thank you,
> >>>>>>>>>>>>>>> Vlad
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On 7/14/17 08:11, Thomas Weise wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> next release is 3.8.0
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Fri, Jul 14, 2017 at 7:55 AM, Vlad Rozov <
> >>>>>>>>>>>>>>>>>> v.rozov@datatorrent.com>
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>> next release - 3.9.0 or 4.0.0?
> >>>>>>>>>>>>>>>>>> Thank you,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Vlad
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On 7/13/17 22:25, Thomas Weise wrote:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> It is time to resurrect this thread and get going with
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>> work.
> >>>>>>>>> For the next release, I will sign up to do the
> >>>>>>>>>>>>> package
> >>>>>>>>>>>>>>>>>> move
> >>>>>>>> in
> >>>>>>>>>> Malhar:
> >>>>>>>>>>>> https://issues.apache.org/
> >>>>>>>>>>>>>>>> jira/browse/APEXMALHAR-2517
> >>>>>>>>>>>>>>>>>> In general this will be straightforward; most classes
> >>>>>>>> in
> >>>>>>>>>>>>>>>>>> Malhar
> >>>>>>>>> are
> >>>>>>>>>>>>> marked
> >>>>>>>>>>>>>>> evolving and it is trivial for users to change import
> >>>>>>>>>>>>>>>>>>>> statements.
> >>>>>>>>>>>>>>>>>> However,
> >>>>>>>>>>>>>> I would suggest to discuss if there are selected
> >>>>>>>>>>>>>>>>>>>> operators
> >>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>> are
> >>>>>>>>>>>>> worth
> >>>>>>>>>>>>>>> keeping a backward compatibility stub in the old
> >>>>>>>>>>>>>>>>>>>> location.
> >>>>>>>>>>>>>>>>>> Here is my plan:
> >>>>>>>>>>> 1. Relocate all classes in *lib* and *contrib* within
> >>>>>>>>>>>>>>>>>>>> one
> >>>>>>>>>>>>>>>>>> PR -
> >>>>>>>>> this
> >>>>>>>>>>>> is
> >>>>>>>>>>>>>> all
> >>>>>>>>>>>>>>>> IDE automated work
> >>>>>>>>>>>>>>>>>>>> 2. Add backward compatibility classes, if, any in
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> separate
> >>>>>>>>>>>>>>>>>> PR
> >>>>>>>>>> 3. Create PR for Megh library to reflect moved
> >>>>>>>>>>>> classes
> >>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>> Thomas
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Wed, Mar 1, 2017 at 2:24 PM, Pramod Immaneni <
> >>>>>>>>>>>>>>>>>>>> pramod@datatorrent.com
> >>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Inline
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Mon, Feb 27, 2017 at 11:03 PM, Thomas Weise <
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> thw@apache.org>
> >>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>> -->
> >>>>>>>>>>>>>>>>>>>>> On Mon, Feb 27, 2017 at 1:27 PM, Pramod Immaneni <
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> pramod@datatorrent.com
> >>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> For malhar, for existing operators, I prefer we do
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> this as
> >>>>>>>>>>>>>>>>>>>> part
> >>>>>>>>>>> of
> >>>>>>>>>>>>> the
> >>>>>>>>>>>>>>> planned refactoring for breaking the monolith
> >>>>>>>>>>>>>>>>>>>>>> modules
> >>>>>>>>>>>>>>>>>>>> into
> >>>>>>>> baby
> >>>>>>>>>>> packages
> >>>>>>>>>>>>>> and would also prefer deprecating the existing
> >>>>>>>>>>>>>>>>>>>>>> operators
> >>>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>> place.
> >>>>>>>>>>>> Refactor into smaller modules was discussed for
> >>>>>>>>>>>>>>> malhar-contrib
> >>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>> given
> >>>>>>>>>>>>>>> the overall state of that module I think it is OK
> >>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>> defer
> >>>>>>>> package
> >>>>>>>>>>> renaming
> >>>>>>>>>>>>>>> there. I do however prefer to see the package rename
> >>>>>>>>>>>>>>>>>>>>> addressed
> >>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>> other
> >>>>>>>>>>>>>>> modules, especially for the main library module.
> >>>>>>>>>>>>>>>>>>>>>> Should we consider breaking the library into
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> smaller
> >>>>>>>>>>>>>>>>>>>> modules
> >>>>>>>> as
> >>>>>>>>>>>> well,
> >>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>> file/block operators for example probably can be in
> >>>>>>>>>>>>>>>>>>>>> their
> >>>>>>>>>>>>>>>>>>> own
> >>>>>>>>>> module
> >>>>>>>>>>>> from
> >>>>>>>>>>>>>>>> just an organizational perspective.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> This
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> will help us achieve two things. First, the user
> >>>>>>>>>>>>>>>>>>>>> will
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> see
> >>>>>>>> all
> >>>>>>>>>>> the
> >>>>>>>>>>>>> new
> >>>>>>>>>>>>>>> changes at once as opposed to dealing with it
> >>>>>>>>>>>>>>>>>>>>>>> twice
> >>>>>>>>>>>>>>>>>>>>> (with
> >>>>>>>> package
> >>>>>>>>>>> rename
> >>>>>>>>>>>>>>> and dependency changes) and second it will allow
> >>>>>>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>> a
> >>>>>>> smoother
> >>>>>>>>> transition
> >>>>>>>>>>>>>> as the existing code will still work in a
> >>>>>>>>>>>>>>>>>>>>>> deprecated
> >>>>>>>>>>>>>>>>>>>> state.
> >>>>>>>> It
> >>>>>>>>>>> will
> >>>>>>>>>>>>> also
> >>>>>>>>>>>>>>>> give a more consistent structure to malhar. For new
> >>>>>>>>>>>>>>>>>>>>>> operators,
> >>>>>>>>>>>>>>>>>>>> we
> >>>>>>>>>>>>> can
> >>>>>>>>>>>>>>> go
> >>>>>>>>>>>>>>>>>>>>>> with the new package path but we need to ensure
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> they
> >>>>>>>>>>>>>>>>>>>> will
> >>>>>>>> get
> >>>>>>>>>>> moved
> >>>>>>>>>>>>> into
> >>>>>>>>>>>>>>>> the baby packages as well.
> >>>>>>>>>>>>>>>>>>>>>> I think existing operators should be renamed so
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>>>> git
> >>>>>>>> history
> >>>>>>>>>>> remains. A
> >>>>>>>>>>>>>> possible solution for backward compatibility could
> >>>>>>>>>>>>>>>>>>>>> be
> >>>>>>>>>>>>>>>>>>> to
> >>>>>>> subsequently
> >>>>>>>>> add
> >>>>>>>>>>>>>>>> empty subclasses in the previous location (for
> >>>>>>>>>>>>>>>>>>>>>> existing
> >> Thank you,
> >>
> >> Vlad
> >>
>
>
> Thank you,
>
> Vlad
>
Re: Java packages: legacy -> org.apache.apex
Posted by Vlad Rozov <vr...@apache.org>.
Yes, consistency.
Thank you,
Vlad
On 8/19/17 10:57, Pramod Immaneni wrote:
> Is there any significant gain by doing this?
>
> Thanks
>
> On Sat, Aug 19, 2017 at 10:36 AM, Vlad Rozov <vr...@apache.org> wrote:
>
>> The parent level pom artifact Id is not referenced by applications, it
>> exists simply to group modules together under current malhar project. The
>> name can be changed to library-parent(root) or apex-library-parent(root) or
>> something similar. Note that the proposal includes rename of the project
>> from apex-malhar to apex-library along with the version change.
>>
>> Thank you,
>>
>> Vlad
>>
>>
>> On 8/19/17 10:10, Pramod Immaneni wrote:
>>
>>> Would library module then become apex-library-library? Why change the
>>> malhar artifact id as the project is called apex-malhar.
>>>
>>> Thanks
>>>
>>> On Sat, Aug 19, 2017 at 10:03 AM, Vlad Rozov <vr...@apache.org> wrote:
>>>
>>> Overall I support the proposal and this is what I was in favor of during
>>>> the last discussion. I'd also propose changing the name of the artifact
>>>> id
>>>> and git repo from malhar to apex-library as part of the major version
>>>> change. It will make it more consistent with the entire project and will
>>>> also allow using 1.0 (1.0-SNAPSHOT) version that more closely reflects
>>>> maturity of the library.
>>>>
>>>> Thank you,
>>>>
>>>> Vlad
>>>>
>>>> On 8/18/17 17:04, Thomas Weise wrote:
>>>>
>>>> The proposed change does NOT require changes to existing apps. It follows
>>>>> what you see for example in the JDK where older classes are retired
>>>>> without
>>>>> making breaking major release changes.
>>>>>
>>>>> Even though not required technically, it looks like the preference is to
>>>>> make this change with a 4.0 release. I will therefore propose to change
>>>>> the
>>>>> master branch to 4.0-SNAPSHOT and make this and other changes deemed
>>>>> worthwhile.
>>>>>
>>>>> To me it is more important that to newcomers the codebase does not look
>>>>> like an outdated legacy situation and therefore if I have to make a
>>>>> choice
>>>>> than pretty much any version number will do for that. So far I have seen
>>>>> very little interest and initiative in addressing maintenance work like
>>>>> the
>>>>> packages, checkstyle, CI stability, documentation etc. So it would be
>>>>> great
>>>>> if people interested would step up and contribute.
>>>>>
>>>>> Here is the next proposal how to proceed:
>>>>>
>>>>> - create release-3.8 branch for last 3.8 release right away
>>>>> - update master to 4.0-SNAPSHOT as part of the open PR #664
>>>>> - identify further cleanup work for master
>>>>> - release 4.0
>>>>>
>>>>> Thanks,
>>>>> Thomas
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Aug 17, 2017 at 4:48 PM, Amol Kekre <am...@datatorrent.com>
>>>>> wrote:
>>>>>
>>>>> This following pull request should be taken up in 4.0.0. See my comments
>>>>>
>>>>>> in
>>>>>> https://github.com/apache/apex-malhar/pull/664
>>>>>> <https://www.google.com/url?q=https%3A%2F%2Fgithub.com%
>>>>>> 2Fapache%2Fapex-malhar%2Fpull%2F664&sa=D&ust=1503020269444000&usg=
>>>>>> AFQjCNHDPZIg2e6I33Jb1XjB5Ir1FkdURQ>
>>>>>> https://github.com/apache/apex-malhar/pull/662
>>>>>> <https://www.google.com/url?q=https%3A%2F%2Fgithub.com%
>>>>>> 2Fapache%2Fapex-malhar%2Fpull%2F662&sa=D&ust=1503020269444000&usg=
>>>>>> AFQjCNF7Xde7X55M3qmc8z-D5y3ZwNg7Fg>
>>>>>>
>>>>>> This merge should not be done without a consensus. This will require
>>>>>> code
>>>>>> changes to existing apps.
>>>>>>
>>>>>> Thks,
>>>>>> Amol
>>>>>>
>>>>>>
>>>>>>
>>>>>> E:amol@datatorrent.com | M: 510-449-2606 | Twitter: @*amolhkekre*
>>>>>>
>>>>>> www.datatorrent.com
>>>>>>
>>>>>>
>>>>>> On Mon, Aug 14, 2017 at 7:48 PM, Thomas Weise <th...@apache.org> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>>> I opened following PRs for the package change:
>>>>>>>
>>>>>>> https://github.com/apache/apex-malhar/pull/662
>>>>>>>
>>>>>>> Moves all classes with history retained (hence 2 commits). Also
>>>>>>> contains
>>>>>>> checkstyle and other mechanical changes.
>>>>>>>
>>>>>>> https://github.com/apache/apex-malhar/pull/664
>>>>>>>
>>>>>>> Adds backward compatibility jar.
>>>>>>>
>>>>>>> Once above PRs are merged the new artifact can be deployed and
>>>>>>> introduced
>>>>>>> as dependency in malhar-library.
>>>>>>>
>>>>>>> Please review.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Thomas
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Sun, Jul 16, 2017 at 7:04 AM, Thomas Weise <th...@apache.org> wrote:
>>>>>>>
>>>>>>> My original list of work items contained the b/w compatibility aspect,
>>>>>>> I
>>>>>>> don't think there should be any confusion of whether it will be
>>>>>>> covered
>>>>>>>
>>>>>>>> here or not.
>>>>>>>>
>>>>>>>> The proposed shading will provide the old classes and they will be
>>>>>>>>
>>>>>>>> frozen
>>>>>>> as of release 3.7. That's the same as making a copy of the code and
>>>>>>> never
>>>>>>> again making changes to the original classes. This cannot be
>>>>>>> accomplished
>>>>>>> by using the older 3.7 release in your project because you cannot use
>>>>>>> 2
>>>>>>>
>>>>>>>> different versions of Malhar in parallel unless you apply shading.
>>>>>>>>
>>>>>>>> The shaded artifact will only expose the com.datatorrent classes, and
>>>>>>>>
>>>>>>>> they
>>>>>>> will be self-contained as the rest of the classes that they may depend
>>>>>>>> on
>>>>>>> are shaded. The shaded artifact does not evolve, there are not more
>>>>>>> changes
>>>>>>>
>>>>>>> to com.datatorrent classes after they are relocated in master.
>>>>>>>> Thanks,
>>>>>>>> Thomas
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sun, Jul 16, 2017 at 2:00 AM, Pramod Immaneni <
>>>>>>>>
>>>>>>>> pramod@datatorrent.com
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I don't think we can limit the topic strictly to relocation without
>>>>>>>> having
>>>>>>>> a good b/w compatibility story or at least one that goes far enough.
>>>>>>>>
>>>>>>>>> The shading idea sounds interesting. Why not let the shaded version
>>>>>>>>>
>>>>>>>>> move
>>>>>>> forward with each release till we hit a major release. If it is going
>>>>>>>
>>>>>>>> to
>>>>>>>>
>>>>>>> remain pegged at 3.7.0, why shade in the first place as the regular
>>>>>>>
>>>>>>>> 3.7.0
>>>>>>>> release would do the same job and it would be same as the loss of b/w
>>>>>>>>
>>>>>>>>> compatibility with newer releases.
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>>
>>>>>>>>> On Sat, Jul 15, 2017 at 7:57 AM, Thomas Weise <th...@apache.org>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Discussing what in the future might become stable needs to be a
>>>>>>>>> separate
>>>>>>>>>
>>>>>>>> thread, it will be a much bigger discussion.
>>>>>>>>
>>>>>>>>> The topic here is to relocate the packages. With a few exceptions
>>>>>>>>>> relocation won't affect the semantic versioning. Semantic
>>>>>>>>>> versioning
>>>>>>>>>>
>>>>>>>>>> is
>>>>>>>> essentially not effective for Malhar because almost everything is
>>>>>>>>
>>>>>>>>> @Evolving
>>>>>>>>>
>>>>>>>>> (and there are reasons for that.. -> separate topic)
>>>>>>>>>> I don't really like the idea of creating bw compatibility stubs for
>>>>>>>>>>
>>>>>>>>>> the
>>>>>>>> follow up PR. It creates even more clutter in the source tree than
>>>>>>>>
>>>>>>>>> there
>>>>>>>>>
>>>>>>>> already is and so here is an alternative suggestion:
>>>>>>>>
>>>>>>>>> https://github.com/tweise/apex-malhar/blob/malhar37-
>>>>>>>>>> compat/shaded-malhar37/pom.xml
>>>>>>>>>>
>>>>>>>>>> Create a shaded artifact that provides the old com.datatorrent.*
>>>>>>>>>>
>>>>>>>>>> classes as
>>>>>>>>> of release 3.7. Users can include that artifact if they don't want
>>>>>>>>>> to
>>>>>>>> change import statements. At the same time they have an incentive to
>>>>>>>> switch
>>>>>>>>> to the relocated classes to take advantage of bug fixes and new
>>>>>>>>>> functionality.
>>>>>>>>>>
>>>>>>>>>> I will work on the first PR that does the relocate. In the
>>>>>>>>>> meantime,
>>>>>>>>>>
>>>>>>>>>> we
>>>>>>>> can
>>>>>>>>
>>>>>>>>> finalize what backward compatibility support we want to provide and
>>>>>>>>>> how.
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>>> Thomas
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Fri, Jul 14, 2017 at 11:33 AM, Pramod Immaneni <
>>>>>>>>>>
>>>>>>>>>> pramod@datatorrent.com>
>>>>>>>>> wrote:
>>>>>>>>>> How about coming up with a list of @Evolving operators that we
>>>>>>>>>> would
>>>>>>>>>>
>>>>>>>>> like
>>>>>>>> to see in the eventual stable list and move those along with the
>>>>>>>>>> not
>>>>>>>>>>
>>>>>>>>> @Evolving ones in org.apache.apex with b/w stubs and leave the
>>>>>>>> rest
>>>>>>>>> as
>>>>>>> they
>>>>>>>>> are. Then have a follow up JIRA for the rest to be moved over to
>>>>>>>>>>> contrib
>>>>>>>>>> and be deprecated.
>>>>>>>>>>
>>>>>>>>>>> Thanks
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Jul 14, 2017 at 10:37 AM, Thomas Weise <
>>>>>>>>>>>
>>>>>>>>>>> thomas.weise@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> We need to keep the discussion here on topic, if other things
>>>>>>>>>>> are
>>>>>>>>>>>
>>>>>>>>>> piled
>>>>>>>> on
>>>>>>>>>>> top then nothing gets done.
>>>>>>>>>>>> Most operators today are not stable, they are @Evolving. So
>>>>>>>>>>>>
>>>>>>>>>>>> perhaps
>>>>>>>>>> it
>>>>>>>>> is
>>>>>>>>>> necessary to have a separate discussion about that, outside of
>>>>>>>>>>> this
>>>>>>>>>>>
>>>>>>>>>> thread.
>>>>>>>>> My guess is that there are only a few operators that we could
>>>>>>>>>>>> declare
>>>>>>>>>> stable. Specifically, under contrib the closest one might have
>>>>>>>>>>
>>>>>>>>>>> been
>>>>>>>>>>>
>>>>>>>>>> Kafka,
>>>>>>>>> but that is already superseded by the newer versions.
>>>>>>>>>>>> Thomas
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Jul 14, 2017 at 10:21 AM, Pramod Immaneni <
>>>>>>>>>>>>
>>>>>>>>>>>> pramod@datatorrent.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>> We had a discussion a while back, agreed to relegate
>>>>>>>>>>>> non-stable
>>>>>>>>>>>>
>>>>>>>>>>> and
>>>>>>>> experimental operators to contrib and also added this to
>>>>>>>>>>> contribution
>>>>>>>>>>> guidelines. We aexecuted on this and cleaned up the repo by
>>>>>>>>>>> moving
>>>>>>>>>>> operators deemed non-suitable for production use at that time
>>>>>>>>> to
>>>>>>>>>>> contrib.
>>>>>>>> So I wouldn't say the operators that are at the top level
>>>>>>>>>>>> today
>>>>>>>>>>>>
>>>>>>>>>>> or
>>>>>>> the
>>>>>>>>> ones
>>>>>>>>>>>> in library are 0.x.x quality. Granted, we may need to do one
>>>>>>>>>>>>> more
>>>>>>>>>>> pass
>>>>>>>>> as
>>>>>>>>>>> some of the operators may no longer be considered the right
>>>>>>>>>>>> implementations
>>>>>>>>>>>>
>>>>>>>>>>>> with the advent of the windowed operator.
>>>>>>>>>>>>> Since contrib used to be the place that housed operators that
>>>>>>>>>>>>>
>>>>>>>>>>>>> required
>>>>>>>>>>> third party libraries, there are some production quality
>>>>>>>>>>>
>>>>>>>>>>>> operators in
>>>>>>>>>>>>
>>>>>>>>>>> there
>>>>>>>>>>> that need to be pulled out into top level modules.
>>>>>>>>>>>>> I think we are in agreement that for operators that we
>>>>>>>>>>>>>
>>>>>>>>>>>>> consider
>>>>>>>>>>> stable,
>>>>>>>> we
>>>>>>>>>>>> should provide a b/w stub. I would add that we strongly
>>>>>>>>>>>>> consider
>>>>>>>>>>> creating
>>>>>>>> the org.apache.apex counterparts of any stable operators that
>>>>>>>>>>>> are
>>>>>>>>>>>>
>>>>>>>>>>> in
>>>>>>>>> contrib out in top level modules and have the com.datatorrent
>>>>>>>>>>> stubs
>>>>>>>>>>> in
>>>>>>>>>> contrib.
>>>>>>>>>>>> For the operators not considered stable, I would prefer we
>>>>>>>>>>>>> either
>>>>>>>>>>> leave
>>>>>>>>> the
>>>>>>>>>>>> current package path as is or if we change the package path
>>>>>>>>>>>>> then
>>>>>>>>>>> create
>>>>>>>> the
>>>>>>>>>>>> b/w stub as I am not sure how widely they are in use (so, in
>>>>>>>>>>>>> essence,
>>>>>>>>>>> preserve semantic versioning). It would be good if there is a
>>>>>>>>>>> followup
>>>>>>>>>>> JIRA
>>>>>>>>>>>
>>>>>>>>>>>> that takes a look at what other operators can be moved to
>>>>>>>>>>>>> contrib
>>>>>>>>>>> with
>>>>>>>>> the
>>>>>>>>>>>> advent of the newer frameworks and understanding.
>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Jul 14, 2017 at 9:44 AM, Thomas Weise <thw@apache.org
>>>>>>>>>>>>>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>> Most of the operators evolve, as is quite clearly indicated
>>>>>>>>>>>
>>>>>>>>>>>> by
>>>>>>>>>>>> @Evolving
>>>>>>>> annotations. So while perhaps 0.x.x would be a more
>>>>>>>>>>>>> appropriate
>>>>>>>>>>>>>
>>>>>>>>>>>> version
>>>>>>>>> number, I don't think you can change that.
>>>>>>>>>>>>> Thomas
>>>>>>>>>>>>>> On Fri, Jul 14, 2017 at 9:35 AM, Vlad Rozov <
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> v.rozov@datatorrent.com
>>>>>>>>>>>> wrote:
>>>>>>>>>>>> If entire library is not stable, its version should be
>>>>>>>>>>>>>> 0.x.x
>>>>>>>>>>>>>>
>>>>>>>>>>>>> and
>>>>>>>> there
>>>>>>>>>>> should not be any semantic versioning enabled or implied.
>>>>>>>>>>>>>> It
>>>>>>>>>>>>>>
>>>>>>>>>>>>> evolves.
>>>>>>>> If
>>>>>>>>>>>>> some operators are stable as 3.8.x version suggests, the
>>>>>>>>>>>>>> library
>>>>>>>>>>>>>>
>>>>>>>>>>>>> should
>>>>>>>>>>> follow semantic versioning and it is not OK to make a
>>>>>>>>>>>>>> major
>>>>>>>>>>>>>>
>>>>>>>>>>>>> change
>>>>>>>> that
>>>>>>>>>>>> breaks semantic versioning across the entire library. It
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>
>>>>>>>>>>>>> not a
>>>>>>>> finding
>>>>>>>>>>> an excuse not to make a change. For me semantic versioning
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>
>>>>>>>>>>>>> backward
>>>>>>>>> compatibility is more important than a proper package
>>>>>>>>>>>>>> name.
>>>>>>>>>>>>>>
>>>>>>>>>>>>> Thank you,
>>>>>>>> Vlad
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 7/14/17 09:11, Thomas Weise wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Semantic versioning makes sense when you have a stable
>>>>>>>>>>>>>>> baseline.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> As
>>>>>>>>>>> long
>>>>>>>>>>>>> as
>>>>>>>>>>>>>>> frequent fixes need to be made to address basic issues,
>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>> makes
>>>>>>>> no
>>>>>>>>>>> sense
>>>>>>>>>>>>> to declare operators stable, which is why they are marked
>>>>>>>>>>>>>>> evolving.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Instead of finding reasons to avoid changes that should
>>>>>>>>>>>>> have
>>>>>>>>>>>>>> been
>>>>>>>>> made a
>>>>>>>>>>>> long time ago, why not discuss what a "stable operator"
>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> and
>>>>>>>> which
>>>>>>>>>>> ones
>>>>>>>>>>>>> deserve to be in that category. Those are the ones that
>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> get
>>>>>>>>> the
>>>>>>>>>>> backward compatibility stub.
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>> Thomas
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Fri, Jul 14, 2017 at 8:34 AM, Vlad Rozov <
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> v.rozov@datatorrent.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> My preference is to agree that the next Malhar release is
>>>>>>>>>>>>>>>> 4.0.0
>>>>>>>>>>>>>> and
>>>>>>>>>>> make
>>>>>>>>>>>>> the proposed changes when 3.8.0 goes out. Otherwise why
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> keep
>>>>>>>> semantic
>>>>>>>>>>> versioning checks on Malhar in case there is no version
>>>>>>>>>>>>>>>> compatibility.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thank you,
>>>>>>>>>>>>>>> Vlad
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On 7/14/17 08:11, Thomas Weise wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> next release is 3.8.0
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Fri, Jul 14, 2017 at 7:55 AM, Vlad Rozov <
>>>>>>>>>>>>>>>>>> v.rozov@datatorrent.com>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> next release - 3.9.0 or 4.0.0?
>>>>>>>>>>>>>>>>>> Thank you,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Vlad
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On 7/13/17 22:25, Thomas Weise wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> It is time to resurrect this thread and get going with
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> work.
>>>>>>>>> For the next release, I will sign up to do the
>>>>>>>>>>>>> package
>>>>>>>>>>>>>>>>>> move
>>>>>>>> in
>>>>>>>>>> Malhar:
>>>>>>>>>>>> https://issues.apache.org/
>>>>>>>>>>>>>>>> jira/browse/APEXMALHAR-2517
>>>>>>>>>>>>>>>>>> In general this will be straightforward; most classes
>>>>>>>> in
>>>>>>>>>>>>>>>>>> Malhar
>>>>>>>>> are
>>>>>>>>>>>>> marked
>>>>>>>>>>>>>>> evolving and it is trivial for users to change import
>>>>>>>>>>>>>>>>>>>> statements.
>>>>>>>>>>>>>>>>>> However,
>>>>>>>>>>>>>> I would suggest to discuss if there are selected
>>>>>>>>>>>>>>>>>>>> operators
>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>> are
>>>>>>>>>>>>> worth
>>>>>>>>>>>>>>> keeping a backward compatibility stub in the old
>>>>>>>>>>>>>>>>>>>> location.
>>>>>>>>>>>>>>>>>> Here is my plan:
>>>>>>>>>>> 1. Relocate all classes in *lib* and *contrib* within
>>>>>>>>>>>>>>>>>>>> one
>>>>>>>>>>>>>>>>>> PR -
>>>>>>>>> this
>>>>>>>>>>>> is
>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>> IDE automated work
>>>>>>>>>>>>>>>>>>>> 2. Add backward compatibility classes, if, any in
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> separate
>>>>>>>>>>>>>>>>>> PR
>>>>>>>>>> 3. Create PR for Megh library to reflect moved
>>>>>>>>>>>> classes
>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>> Thomas
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Wed, Mar 1, 2017 at 2:24 PM, Pramod Immaneni <
>>>>>>>>>>>>>>>>>>>> pramod@datatorrent.com
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Inline
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Mon, Feb 27, 2017 at 11:03 PM, Thomas Weise <
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> thw@apache.org>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>> -->
>>>>>>>>>>>>>>>>>>>>> On Mon, Feb 27, 2017 at 1:27 PM, Pramod Immaneni <
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> pramod@datatorrent.com
>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> For malhar, for existing operators, I prefer we do
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> this as
>>>>>>>>>>>>>>>>>>>> part
>>>>>>>>>>> of
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> planned refactoring for breaking the monolith
>>>>>>>>>>>>>>>>>>>>>> modules
>>>>>>>>>>>>>>>>>>>> into
>>>>>>>> baby
>>>>>>>>>>> packages
>>>>>>>>>>>>>> and would also prefer deprecating the existing
>>>>>>>>>>>>>>>>>>>>>> operators
>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>> place.
>>>>>>>>>>>> Refactor into smaller modules was discussed for
>>>>>>>>>>>>>>> malhar-contrib
>>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>> given
>>>>>>>>>>>>>>> the overall state of that module I think it is OK
>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> defer
>>>>>>>> package
>>>>>>>>>>> renaming
>>>>>>>>>>>>>>> there. I do however prefer to see the package rename
>>>>>>>>>>>>>>>>>>>>> addressed
>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>> other
>>>>>>>>>>>>>>> modules, especially for the main library module.
>>>>>>>>>>>>>>>>>>>>>> Should we consider breaking the library into
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> smaller
>>>>>>>>>>>>>>>>>>>> modules
>>>>>>>> as
>>>>>>>>>>>> well,
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> file/block operators for example probably can be in
>>>>>>>>>>>>>>>>>>>>> their
>>>>>>>>>>>>>>>>>>> own
>>>>>>>>>> module
>>>>>>>>>>>> from
>>>>>>>>>>>>>>>> just an organizational perspective.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> This
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> will help us achieve two things. First, the user
>>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> see
>>>>>>>> all
>>>>>>>>>>> the
>>>>>>>>>>>>> new
>>>>>>>>>>>>>>> changes at once as opposed to dealing with it
>>>>>>>>>>>>>>>>>>>>>>> twice
>>>>>>>>>>>>>>>>>>>>> (with
>>>>>>>> package
>>>>>>>>>>> rename
>>>>>>>>>>>>>>> and dependency changes) and second it will allow
>>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>> a
>>>>>>> smoother
>>>>>>>>> transition
>>>>>>>>>>>>>> as the existing code will still work in a
>>>>>>>>>>>>>>>>>>>>>> deprecated
>>>>>>>>>>>>>>>>>>>> state.
>>>>>>>> It
>>>>>>>>>>> will
>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>> give a more consistent structure to malhar. For new
>>>>>>>>>>>>>>>>>>>>>> operators,
>>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>> can
>>>>>>>>>>>>>>> go
>>>>>>>>>>>>>>>>>>>>>> with the new package path but we need to ensure
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> they
>>>>>>>>>>>>>>>>>>>> will
>>>>>>>> get
>>>>>>>>>>> moved
>>>>>>>>>>>>> into
>>>>>>>>>>>>>>>> the baby packages as well.
>>>>>>>>>>>>>>>>>>>>>> I think existing operators should be renamed so
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>> git
>>>>>>>> history
>>>>>>>>>>> remains. A
>>>>>>>>>>>>>> possible solution for backward compatibility could
>>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>> to
>>>>>>> subsequently
>>>>>>>>> add
>>>>>>>>>>>>>>>> empty subclasses in the previous location (for
>>>>>>>>>>>>>>>>>>>>>> existing
>> Thank you,
>>
>> Vlad
>>
Thank you,
Vlad
Re: Java packages: legacy -> org.apache.apex
Posted by Pramod Immaneni <pr...@datatorrent.com>.
Is there any significant gain by doing this?
Thanks
On Sat, Aug 19, 2017 at 10:36 AM, Vlad Rozov <vr...@apache.org> wrote:
> The parent level pom artifact Id is not referenced by applications, it
> exists simply to group modules together under current malhar project. The
> name can be changed to library-parent(root) or apex-library-parent(root) or
> something similar. Note that the proposal includes rename of the project
> from apex-malhar to apex-library along with the version change.
>
> Thank you,
>
> Vlad
>
>
> On 8/19/17 10:10, Pramod Immaneni wrote:
>
>> Would library module then become apex-library-library? Why change the
>> malhar artifact id as the project is called apex-malhar.
>>
>> Thanks
>>
>> On Sat, Aug 19, 2017 at 10:03 AM, Vlad Rozov <vr...@apache.org> wrote:
>>
>> Overall I support the proposal and this is what I was in favor of during
>>> the last discussion. I'd also propose changing the name of the artifact
>>> id
>>> and git repo from malhar to apex-library as part of the major version
>>> change. It will make it more consistent with the entire project and will
>>> also allow using 1.0 (1.0-SNAPSHOT) version that more closely reflects
>>> maturity of the library.
>>>
>>> Thank you,
>>>
>>> Vlad
>>>
>>> On 8/18/17 17:04, Thomas Weise wrote:
>>>
>>> The proposed change does NOT require changes to existing apps. It follows
>>>> what you see for example in the JDK where older classes are retired
>>>> without
>>>> making breaking major release changes.
>>>>
>>>> Even though not required technically, it looks like the preference is to
>>>> make this change with a 4.0 release. I will therefore propose to change
>>>> the
>>>> master branch to 4.0-SNAPSHOT and make this and other changes deemed
>>>> worthwhile.
>>>>
>>>> To me it is more important that to newcomers the codebase does not look
>>>> like an outdated legacy situation and therefore if I have to make a
>>>> choice
>>>> than pretty much any version number will do for that. So far I have seen
>>>> very little interest and initiative in addressing maintenance work like
>>>> the
>>>> packages, checkstyle, CI stability, documentation etc. So it would be
>>>> great
>>>> if people interested would step up and contribute.
>>>>
>>>> Here is the next proposal how to proceed:
>>>>
>>>> - create release-3.8 branch for last 3.8 release right away
>>>> - update master to 4.0-SNAPSHOT as part of the open PR #664
>>>> - identify further cleanup work for master
>>>> - release 4.0
>>>>
>>>> Thanks,
>>>> Thomas
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, Aug 17, 2017 at 4:48 PM, Amol Kekre <am...@datatorrent.com>
>>>> wrote:
>>>>
>>>> This following pull request should be taken up in 4.0.0. See my comments
>>>>
>>>>> in
>>>>> https://github.com/apache/apex-malhar/pull/664
>>>>> <https://www.google.com/url?q=https%3A%2F%2Fgithub.com%
>>>>> 2Fapache%2Fapex-malhar%2Fpull%2F664&sa=D&ust=1503020269444000&usg=
>>>>> AFQjCNHDPZIg2e6I33Jb1XjB5Ir1FkdURQ>
>>>>> https://github.com/apache/apex-malhar/pull/662
>>>>> <https://www.google.com/url?q=https%3A%2F%2Fgithub.com%
>>>>> 2Fapache%2Fapex-malhar%2Fpull%2F662&sa=D&ust=1503020269444000&usg=
>>>>> AFQjCNF7Xde7X55M3qmc8z-D5y3ZwNg7Fg>
>>>>>
>>>>> This merge should not be done without a consensus. This will require
>>>>> code
>>>>> changes to existing apps.
>>>>>
>>>>> Thks,
>>>>> Amol
>>>>>
>>>>>
>>>>>
>>>>> E:amol@datatorrent.com | M: 510-449-2606 | Twitter: @*amolhkekre*
>>>>>
>>>>> www.datatorrent.com
>>>>>
>>>>>
>>>>> On Mon, Aug 14, 2017 at 7:48 PM, Thomas Weise <th...@apache.org> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>>> I opened following PRs for the package change:
>>>>>>
>>>>>> https://github.com/apache/apex-malhar/pull/662
>>>>>>
>>>>>> Moves all classes with history retained (hence 2 commits). Also
>>>>>> contains
>>>>>> checkstyle and other mechanical changes.
>>>>>>
>>>>>> https://github.com/apache/apex-malhar/pull/664
>>>>>>
>>>>>> Adds backward compatibility jar.
>>>>>>
>>>>>> Once above PRs are merged the new artifact can be deployed and
>>>>>> introduced
>>>>>> as dependency in malhar-library.
>>>>>>
>>>>>> Please review.
>>>>>>
>>>>>> Thanks,
>>>>>> Thomas
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Sun, Jul 16, 2017 at 7:04 AM, Thomas Weise <th...@apache.org> wrote:
>>>>>>
>>>>>> My original list of work items contained the b/w compatibility aspect,
>>>>>> I
>>>>>> don't think there should be any confusion of whether it will be
>>>>>> covered
>>>>>>
>>>>>>> here or not.
>>>>>>>
>>>>>>> The proposed shading will provide the old classes and they will be
>>>>>>>
>>>>>>> frozen
>>>>>> as of release 3.7. That's the same as making a copy of the code and
>>>>>> never
>>>>>> again making changes to the original classes. This cannot be
>>>>>> accomplished
>>>>>> by using the older 3.7 release in your project because you cannot use
>>>>>> 2
>>>>>>
>>>>>>> different versions of Malhar in parallel unless you apply shading.
>>>>>>>
>>>>>>> The shaded artifact will only expose the com.datatorrent classes, and
>>>>>>>
>>>>>>> they
>>>>>>
>>>>>> will be self-contained as the rest of the classes that they may depend
>>>>>>>
>>>>>>> on
>>>>>> are shaded. The shaded artifact does not evolve, there are not more
>>>>>> changes
>>>>>>
>>>>>> to com.datatorrent classes after they are relocated in master.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Thomas
>>>>>>>
>>>>>>>
>>>>>>> On Sun, Jul 16, 2017 at 2:00 AM, Pramod Immaneni <
>>>>>>>
>>>>>>> pramod@datatorrent.com
>>>>>> wrote:
>>>>>>
>>>>>>> I don't think we can limit the topic strictly to relocation without
>>>>>>> having
>>>>>>> a good b/w compatibility story or at least one that goes far enough.
>>>>>>>
>>>>>>>> The shading idea sounds interesting. Why not let the shaded version
>>>>>>>>
>>>>>>>> move
>>>>>>>
>>>>>> forward with each release till we hit a major release. If it is going
>>>>>>
>>>>>>> to
>>>>>>>
>>>>>> remain pegged at 3.7.0, why shade in the first place as the regular
>>>>>>
>>>>>>> 3.7.0
>>>>>>> release would do the same job and it would be same as the loss of b/w
>>>>>>>
>>>>>>>> compatibility with newer releases.
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>>
>>>>>>>> On Sat, Jul 15, 2017 at 7:57 AM, Thomas Weise <th...@apache.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Discussing what in the future might become stable needs to be a
>>>>>>>> separate
>>>>>>>>
>>>>>>> thread, it will be a much bigger discussion.
>>>>>>>
>>>>>>>> The topic here is to relocate the packages. With a few exceptions
>>>>>>>>> relocation won't affect the semantic versioning. Semantic
>>>>>>>>> versioning
>>>>>>>>>
>>>>>>>>> is
>>>>>>>>
>>>>>>> essentially not effective for Malhar because almost everything is
>>>>>>>
>>>>>>>> @Evolving
>>>>>>>>
>>>>>>>> (and there are reasons for that.. -> separate topic)
>>>>>>>>>
>>>>>>>>> I don't really like the idea of creating bw compatibility stubs for
>>>>>>>>>
>>>>>>>>> the
>>>>>>>>
>>>>>>> follow up PR. It creates even more clutter in the source tree than
>>>>>>>
>>>>>>>> there
>>>>>>>>
>>>>>>> already is and so here is an alternative suggestion:
>>>>>>>
>>>>>>>> https://github.com/tweise/apex-malhar/blob/malhar37-
>>>>>>>>> compat/shaded-malhar37/pom.xml
>>>>>>>>>
>>>>>>>>> Create a shaded artifact that provides the old com.datatorrent.*
>>>>>>>>>
>>>>>>>>> classes as
>>>>>>>>
>>>>>>>> of release 3.7. Users can include that artifact if they don't want
>>>>>>>>>
>>>>>>>>> to
>>>>>>>>
>>>>>>> change import statements. At the same time they have an incentive to
>>>>>>
>>>>>>> switch
>>>>>>>>
>>>>>>>> to the relocated classes to take advantage of bug fixes and new
>>>>>>>>> functionality.
>>>>>>>>>
>>>>>>>>> I will work on the first PR that does the relocate. In the
>>>>>>>>> meantime,
>>>>>>>>>
>>>>>>>>> we
>>>>>>>>
>>>>>>> can
>>>>>>>
>>>>>>>> finalize what backward compatibility support we want to provide and
>>>>>>>>>
>>>>>>>>> how.
>>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>>> Thomas
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, Jul 14, 2017 at 11:33 AM, Pramod Immaneni <
>>>>>>>>>
>>>>>>>>> pramod@datatorrent.com>
>>>>>>>>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> How about coming up with a list of @Evolving operators that we
>>>>>>>>> would
>>>>>>>>>
>>>>>>>> like
>>>>>>
>>>>>>> to see in the eventual stable list and move those along with the
>>>>>>>>> not
>>>>>>>>>
>>>>>>>> @Evolving ones in org.apache.apex with b/w stubs and leave the
>>>>>>
>>>>>>> rest
>>>>>>>>>
>>>>>>>> as
>>>>>>
>>>>>> they
>>>>>>>
>>>>>>>> are. Then have a follow up JIRA for the rest to be moved over to
>>>>>>>>>>
>>>>>>>>>> contrib
>>>>>>>>> and be deprecated.
>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>>
>>>>>>>>>> On Fri, Jul 14, 2017 at 10:37 AM, Thomas Weise <
>>>>>>>>>>
>>>>>>>>>> thomas.weise@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> We need to keep the discussion here on topic, if other things
>>>>>>>>>> are
>>>>>>>>>>
>>>>>>>>> piled
>>>>>>
>>>>>>> on
>>>>>>>>>
>>>>>>>>>> top then nothing gets done.
>>>>>>>>>>>
>>>>>>>>>>> Most operators today are not stable, they are @Evolving. So
>>>>>>>>>>>
>>>>>>>>>>> perhaps
>>>>>>>>>>
>>>>>>>>> it
>>>>>>>
>>>>>>>> is
>>>>>>>>>
>>>>>>>>> necessary to have a separate discussion about that, outside of
>>>>>>>>>> this
>>>>>>>>>>
>>>>>>>>> thread.
>>>>>>>
>>>>>>>> My guess is that there are only a few operators that we could
>>>>>>>>>>>
>>>>>>>>>>> declare
>>>>>>>>>>
>>>>>>>>> stable. Specifically, under contrib the closest one might have
>>>>>>>>>
>>>>>>>>>> been
>>>>>>>>>>
>>>>>>>>> Kafka,
>>>>>>>
>>>>>>>> but that is already superseded by the newer versions.
>>>>>>>>>>>
>>>>>>>>>>> Thomas
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Jul 14, 2017 at 10:21 AM, Pramod Immaneni <
>>>>>>>>>>>
>>>>>>>>>>> pramod@datatorrent.com>
>>>>>>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> We had a discussion a while back, agreed to relegate
>>>>>>>>>>> non-stable
>>>>>>>>>>>
>>>>>>>>>> and
>>>>>>
>>>>>>> experimental operators to contrib and also added this to
>>>>>>>>>
>>>>>>>>>> contribution
>>>>>>>>>>>
>>>>>>>>>> guidelines. We aexecuted on this and cleaned up the repo by
>>>>>>>>>
>>>>>>>>>> moving
>>>>>>>>>>>
>>>>>>>>>> operators deemed non-suitable for production use at that time
>>>>>>>
>>>>>>>> to
>>>>>>>>>>>
>>>>>>>>>> contrib.
>>>>>>
>>>>>>> So I wouldn't say the operators that are at the top level
>>>>>>>>>>> today
>>>>>>>>>>>
>>>>>>>>>> or
>>>>>>
>>>>>> the
>>>>>>>
>>>>>>>> ones
>>>>>>>>>>
>>>>>>>>>>> in library are 0.x.x quality. Granted, we may need to do one
>>>>>>>>>>>>
>>>>>>>>>>>> more
>>>>>>>>>>>
>>>>>>>>>> pass
>>>>>>>
>>>>>>>> as
>>>>>>>>>>
>>>>>>>>>> some of the operators may no longer be considered the right
>>>>>>>>>>> implementations
>>>>>>>>>>>
>>>>>>>>>>> with the advent of the windowed operator.
>>>>>>>>>>>>
>>>>>>>>>>>> Since contrib used to be the place that housed operators that
>>>>>>>>>>>>
>>>>>>>>>>>> required
>>>>>>>>>>>
>>>>>>>>>> third party libraries, there are some production quality
>>>>>>>>>>
>>>>>>>>>>> operators in
>>>>>>>>>>>
>>>>>>>>>> there
>>>>>>>>>
>>>>>>>>>> that need to be pulled out into top level modules.
>>>>>>>>>>>>
>>>>>>>>>>>> I think we are in agreement that for operators that we
>>>>>>>>>>>>
>>>>>>>>>>>> consider
>>>>>>>>>>>
>>>>>>>>>> stable,
>>>>>>
>>>>>>> we
>>>>>>>>>>
>>>>>>>>>>> should provide a b/w stub. I would add that we strongly
>>>>>>>>>>>>
>>>>>>>>>>>> consider
>>>>>>>>>>>
>>>>>>>>>> creating
>>>>>>
>>>>>>> the org.apache.apex counterparts of any stable operators that
>>>>>>>>>>> are
>>>>>>>>>>>
>>>>>>>>>> in
>>>>>>>
>>>>>>>> contrib out in top level modules and have the com.datatorrent
>>>>>>>>>
>>>>>>>>>> stubs
>>>>>>>>>>>
>>>>>>>>>> in
>>>>>>>>>
>>>>>>>>> contrib.
>>>>>>>>>>
>>>>>>>>>>> For the operators not considered stable, I would prefer we
>>>>>>>>>>>>
>>>>>>>>>>>> either
>>>>>>>>>>>
>>>>>>>>>> leave
>>>>>>>
>>>>>>>> the
>>>>>>>>>>
>>>>>>>>>>> current package path as is or if we change the package path
>>>>>>>>>>>>
>>>>>>>>>>>> then
>>>>>>>>>>>
>>>>>>>>>> create
>>>>>>
>>>>>>> the
>>>>>>>>>>
>>>>>>>>>>> b/w stub as I am not sure how widely they are in use (so, in
>>>>>>>>>>>>
>>>>>>>>>>>> essence,
>>>>>>>>>>>
>>>>>>>>>> preserve semantic versioning). It would be good if there is a
>>>>>>>>>
>>>>>>>>>> followup
>>>>>>>>>>>
>>>>>>>>>> JIRA
>>>>>>>>>>
>>>>>>>>>>> that takes a look at what other operators can be moved to
>>>>>>>>>>>>
>>>>>>>>>>>> contrib
>>>>>>>>>>>
>>>>>>>>>> with
>>>>>>>
>>>>>>>> the
>>>>>>>>>>
>>>>>>>>>>> advent of the newer frameworks and understanding.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Jul 14, 2017 at 9:44 AM, Thomas Weise <thw@apache.org
>>>>>>>>>>>>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>> Most of the operators evolve, as is quite clearly indicated
>>>>>>>>>>
>>>>>>>>>>> by
>>>>>>>>>>>>
>>>>>>>>>>> @Evolving
>>>>>>
>>>>>>> annotations. So while perhaps 0.x.x would be a more
>>>>>>>>>>>> appropriate
>>>>>>>>>>>>
>>>>>>>>>>> version
>>>>>>>
>>>>>>>> number, I don't think you can change that.
>>>>>>>>>>>
>>>>>>>>>>>> Thomas
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Jul 14, 2017 at 9:35 AM, Vlad Rozov <
>>>>>>>>>>>>>
>>>>>>>>>>>>> v.rozov@datatorrent.com
>>>>>>>>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> If entire library is not stable, its version should be
>>>>>>>>>>>>> 0.x.x
>>>>>>>>>>>>>
>>>>>>>>>>>> and
>>>>>>
>>>>>>> there
>>>>>>>>>
>>>>>>>>>> should not be any semantic versioning enabled or implied.
>>>>>>>>>>>>
>>>>>>>>>>>>> It
>>>>>>>>>>>>>
>>>>>>>>>>>> evolves.
>>>>>>
>>>>>>> If
>>>>>>>>>>>
>>>>>>>>>>>> some operators are stable as 3.8.x version suggests, the
>>>>>>>>>>>>> library
>>>>>>>>>>>>>
>>>>>>>>>>>> should
>>>>>>>>>
>>>>>>>>>> follow semantic versioning and it is not OK to make a
>>>>>>>>>>>>
>>>>>>>>>>>>> major
>>>>>>>>>>>>>
>>>>>>>>>>>> change
>>>>>>
>>>>>>> that
>>>>>>>>>>
>>>>>>>>>>> breaks semantic versioning across the entire library. It
>>>>>>>>>>>>
>>>>>>>>>>>>> is
>>>>>>>>>>>>>
>>>>>>>>>>>> not a
>>>>>>
>>>>>>> finding
>>>>>>>>>
>>>>>>>>>> an excuse not to make a change. For me semantic versioning
>>>>>>>>>>>>> and
>>>>>>>>>>>>>
>>>>>>>>>>>> backward
>>>>>>>
>>>>>>>> compatibility is more important than a proper package
>>>>>>>>>>>>
>>>>>>>>>>>>> name.
>>>>>>>>>>>>>
>>>>>>>>>>>> Thank you,
>>>>>>
>>>>>>> Vlad
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 7/14/17 09:11, Thomas Weise wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Semantic versioning makes sense when you have a stable
>>>>>>>>>>>>>> baseline.
>>>>>>>>>>>>>>
>>>>>>>>>>>>> As
>>>>>>>>>
>>>>>>>>>> long
>>>>>>>>>>>
>>>>>>>>>>>> as
>>>>>>>>>>>>>
>>>>>>>>>>>>>> frequent fixes need to be made to address basic issues,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>
>>>>>>>>>>>>> makes
>>>>>>
>>>>>>> no
>>>>>>>>>
>>>>>>>>>> sense
>>>>>>>>>>>
>>>>>>>>>>>> to declare operators stable, which is why they are marked
>>>>>>>>>>>>>> evolving.
>>>>>>>>>>>>>>
>>>>>>>>>>>>> Instead of finding reasons to avoid changes that should
>>>>>>>>>>>
>>>>>>>>>>>> have
>>>>>>>>>>>>>>
>>>>>>>>>>>>> been
>>>>>>>
>>>>>>>> made a
>>>>>>>>>>
>>>>>>>>>>> long time ago, why not discuss what a "stable operator"
>>>>>>>>>>>>>
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>
>>>>>>>>>>>>> and
>>>>>>
>>>>>>> which
>>>>>>>>>
>>>>>>>>>> ones
>>>>>>>>>>>
>>>>>>>>>>>> deserve to be in that category. Those are the ones that
>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>
>>>>>>>>>>>>> get
>>>>>>>
>>>>>>>> the
>>>>>>>>>
>>>>>>>>>> backward compatibility stub.
>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Thomas
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, Jul 14, 2017 at 8:34 AM, Vlad Rozov <
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> v.rozov@datatorrent.com>
>>>>>>>>>>>>>>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> My preference is to agree that the next Malhar release is
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 4.0.0
>>>>>>>>>>>>>>
>>>>>>>>>>>>> and
>>>>>>>>>
>>>>>>>>>> make
>>>>>>>>>>>
>>>>>>>>>>>> the proposed changes when 3.8.0 goes out. Otherwise why
>>>>>>>>>>>>>
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> keep
>>>>>>
>>>>>>> semantic
>>>>>>>>>
>>>>>>>>>> versioning checks on Malhar in case there is no version
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> compatibility.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thank you,
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Vlad
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 7/14/17 08:11, Thomas Weise wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> next release is 3.8.0
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Fri, Jul 14, 2017 at 7:55 AM, Vlad Rozov <
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> v.rozov@datatorrent.com>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> next release - 3.9.0 or 4.0.0?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thank you,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Vlad
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On 7/13/17 22:25, Thomas Weise wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> It is time to resurrect this thread and get going with
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> work.
>>>>>>>
>>>>>>>> For the next release, I will sign up to do the
>>>>>>>>>>>
>>>>>>>>>>>> package
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> move
>>>>>>
>>>>>>> in
>>>>>>>>>
>>>>>>>>> Malhar:
>>>>>>>>>>
>>>>>>>>>>> https://issues.apache.org/
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> jira/browse/APEXMALHAR-2517
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> In general this will be straightforward; most classes
>>>>>>
>>>>>>> in
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Malhar
>>>>>>>
>>>>>>>> are
>>>>>>>>>>>
>>>>>>>>>>>> marked
>>>>>>>>>>>>>
>>>>>>>>>>>>>> evolving and it is trivial for users to change import
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> statements.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> However,
>>>>>>>>>>>>
>>>>>>>>>>>>> I would suggest to discuss if there are selected
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> operators
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> that
>>>>>>>>>
>>>>>>>>>> are
>>>>>>>>>>>
>>>>>>>>>>>> worth
>>>>>>>>>>>>>
>>>>>>>>>>>>>> keeping a backward compatibility stub in the old
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> location.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Here is my plan:
>>>>>>>>>
>>>>>>>>>> 1. Relocate all classes in *lib* and *contrib* within
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> one
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> PR -
>>>>>>>
>>>>>>>> this
>>>>>>>>>>
>>>>>>>>>>> is
>>>>>>>>>>>>>
>>>>>>>>>>>>> all
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> IDE automated work
>>>>>>>>>>>>>>>>>>> 2. Add backward compatibility classes, if, any in
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> separate
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> PR
>>>>>>>>>
>>>>>>>>> 3. Create PR for Megh library to reflect moved
>>>>>>>>>>
>>>>>>>>>>> classes
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>
>>>>>>> Thomas
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Wed, Mar 1, 2017 at 2:24 PM, Pramod Immaneni <
>>>>>>>>>>>>>>>>>>> pramod@datatorrent.com
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Inline
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Mon, Feb 27, 2017 at 11:03 PM, Thomas Weise <
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> thw@apache.org>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> -->
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Mon, Feb 27, 2017 at 1:27 PM, Pramod Immaneni <
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> pramod@datatorrent.com
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> For malhar, for existing operators, I prefer we do
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> this as
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> part
>>>>>>>>>
>>>>>>>>>> of
>>>>>>>>>>>>
>>>>>>>>>>>> the
>>>>>>>>>>>>>
>>>>>>>>>>>>>> planned refactoring for breaking the monolith
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> modules
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> into
>>>>>>
>>>>>>> baby
>>>>>>>>>
>>>>>>>>>> packages
>>>>>>>>>>>>
>>>>>>>>>>>>> and would also prefer deprecating the existing
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> operators
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>
>>>>>>>>> place.
>>>>>>>>>>
>>>>>>>>>>> Refactor into smaller modules was discussed for
>>>>>>>>>>>>>
>>>>>>>>>>>>>> malhar-contrib
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>
>>>>>>>>>>>> given
>>>>>>>>>>>>>
>>>>>>>>>>>>>> the overall state of that module I think it is OK
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> defer
>>>>>>
>>>>>>> package
>>>>>>>>>
>>>>>>>>>> renaming
>>>>>>>>>>>>>
>>>>>>>>>>>>>> there. I do however prefer to see the package rename
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> addressed
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>
>>>>>>>>>>>> other
>>>>>>>>>>>>>
>>>>>>>>>>>>>> modules, especially for the main library module.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Should we consider breaking the library into
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> smaller
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> modules
>>>>>>
>>>>>>> as
>>>>>>>>>>
>>>>>>>>>>> well,
>>>>>>>>>>>>
>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> file/block operators for example probably can be in
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> their
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> own
>>>>>>>>>
>>>>>>>>> module
>>>>>>>>>>
>>>>>>>>>>> from
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> just an organizational perspective.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> This
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> will help us achieve two things. First, the user
>>>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> see
>>>>>>
>>>>>>> all
>>>>>>>>>
>>>>>>>>>> the
>>>>>>>>>>>
>>>>>>>>>>>> new
>>>>>>>>>>>>>
>>>>>>>>>>>>>> changes at once as opposed to dealing with it
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> twice
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> (with
>>>>>>
>>>>>>> package
>>>>>>>>>
>>>>>>>>>> rename
>>>>>>>>>>>>>
>>>>>>>>>>>>>> and dependency changes) and second it will allow
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> a
>>>>>>
>>>>>> smoother
>>>>>>>
>>>>>>>> transition
>>>>>>>>>>>>
>>>>>>>>>>>>> as the existing code will still work in a
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> deprecated
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> state.
>>>>>>
>>>>>>> It
>>>>>>>>>>
>>>>>>>>>> will
>>>>>>>>>>>
>>>>>>>>>>>> also
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> give a more consistent structure to malhar. For new
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> operators,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>
>>>>>>>>>>>> can
>>>>>>>>>>>>>
>>>>>>>>>>>>>> go
>>>>>>>>>>>>>>>>>>>>> with the new package path but we need to ensure
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> they
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> will
>>>>>>
>>>>>>> get
>>>>>>>>>
>>>>>>>>>> moved
>>>>>>>>>>>
>>>>>>>>>>>> into
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> the baby packages as well.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I think existing operators should be renamed so
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> git
>>>>>>
>>>>>>> history
>>>>>>>>>
>>>>>>>>>> remains. A
>>>>>>>>>>>>
>>>>>>>>>>>>> possible solution for backward compatibility could
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> to
>>>>>>
>>>>>> subsequently
>>>>>>>
>>>>>>>> add
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> empty subclasses in the previous location (for
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> existing
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>
>
> Thank you,
>
> Vlad
>
Re: Java packages: legacy -> org.apache.apex
Posted by Vlad Rozov <vr...@apache.org>.
The parent level pom artifact Id is not referenced by applications, it
exists simply to group modules together under current malhar project.
The name can be changed to library-parent(root) or
apex-library-parent(root) or something similar. Note that the proposal
includes rename of the project from apex-malhar to apex-library along
with the version change.
Thank you,
Vlad
On 8/19/17 10:10, Pramod Immaneni wrote:
> Would library module then become apex-library-library? Why change the
> malhar artifact id as the project is called apex-malhar.
>
> Thanks
>
> On Sat, Aug 19, 2017 at 10:03 AM, Vlad Rozov <vr...@apache.org> wrote:
>
>> Overall I support the proposal and this is what I was in favor of during
>> the last discussion. I'd also propose changing the name of the artifact id
>> and git repo from malhar to apex-library as part of the major version
>> change. It will make it more consistent with the entire project and will
>> also allow using 1.0 (1.0-SNAPSHOT) version that more closely reflects
>> maturity of the library.
>>
>> Thank you,
>>
>> Vlad
>>
>> On 8/18/17 17:04, Thomas Weise wrote:
>>
>>> The proposed change does NOT require changes to existing apps. It follows
>>> what you see for example in the JDK where older classes are retired
>>> without
>>> making breaking major release changes.
>>>
>>> Even though not required technically, it looks like the preference is to
>>> make this change with a 4.0 release. I will therefore propose to change
>>> the
>>> master branch to 4.0-SNAPSHOT and make this and other changes deemed
>>> worthwhile.
>>>
>>> To me it is more important that to newcomers the codebase does not look
>>> like an outdated legacy situation and therefore if I have to make a choice
>>> than pretty much any version number will do for that. So far I have seen
>>> very little interest and initiative in addressing maintenance work like
>>> the
>>> packages, checkstyle, CI stability, documentation etc. So it would be
>>> great
>>> if people interested would step up and contribute.
>>>
>>> Here is the next proposal how to proceed:
>>>
>>> - create release-3.8 branch for last 3.8 release right away
>>> - update master to 4.0-SNAPSHOT as part of the open PR #664
>>> - identify further cleanup work for master
>>> - release 4.0
>>>
>>> Thanks,
>>> Thomas
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Thu, Aug 17, 2017 at 4:48 PM, Amol Kekre <am...@datatorrent.com> wrote:
>>>
>>> This following pull request should be taken up in 4.0.0. See my comments
>>>> in
>>>> https://github.com/apache/apex-malhar/pull/664
>>>> <https://www.google.com/url?q=https%3A%2F%2Fgithub.com%
>>>> 2Fapache%2Fapex-malhar%2Fpull%2F664&sa=D&ust=1503020269444000&usg=
>>>> AFQjCNHDPZIg2e6I33Jb1XjB5Ir1FkdURQ>
>>>> https://github.com/apache/apex-malhar/pull/662
>>>> <https://www.google.com/url?q=https%3A%2F%2Fgithub.com%
>>>> 2Fapache%2Fapex-malhar%2Fpull%2F662&sa=D&ust=1503020269444000&usg=
>>>> AFQjCNF7Xde7X55M3qmc8z-D5y3ZwNg7Fg>
>>>>
>>>> This merge should not be done without a consensus. This will require code
>>>> changes to existing apps.
>>>>
>>>> Thks,
>>>> Amol
>>>>
>>>>
>>>>
>>>> E:amol@datatorrent.com | M: 510-449-2606 | Twitter: @*amolhkekre*
>>>>
>>>> www.datatorrent.com
>>>>
>>>>
>>>> On Mon, Aug 14, 2017 at 7:48 PM, Thomas Weise <th...@apache.org> wrote:
>>>>
>>>> Hi,
>>>>> I opened following PRs for the package change:
>>>>>
>>>>> https://github.com/apache/apex-malhar/pull/662
>>>>>
>>>>> Moves all classes with history retained (hence 2 commits). Also contains
>>>>> checkstyle and other mechanical changes.
>>>>>
>>>>> https://github.com/apache/apex-malhar/pull/664
>>>>>
>>>>> Adds backward compatibility jar.
>>>>>
>>>>> Once above PRs are merged the new artifact can be deployed and
>>>>> introduced
>>>>> as dependency in malhar-library.
>>>>>
>>>>> Please review.
>>>>>
>>>>> Thanks,
>>>>> Thomas
>>>>>
>>>>>
>>>>>
>>>>> On Sun, Jul 16, 2017 at 7:04 AM, Thomas Weise <th...@apache.org> wrote:
>>>>>
>>>>> My original list of work items contained the b/w compatibility aspect,
>>>>> I
>>>>> don't think there should be any confusion of whether it will be covered
>>>>>> here or not.
>>>>>>
>>>>>> The proposed shading will provide the old classes and they will be
>>>>>>
>>>>> frozen
>>>>> as of release 3.7. That's the same as making a copy of the code and
>>>>> never
>>>>> again making changes to the original classes. This cannot be
>>>>> accomplished
>>>>> by using the older 3.7 release in your project because you cannot use 2
>>>>>> different versions of Malhar in parallel unless you apply shading.
>>>>>>
>>>>>> The shaded artifact will only expose the com.datatorrent classes, and
>>>>>>
>>>>> they
>>>>>
>>>>>> will be self-contained as the rest of the classes that they may depend
>>>>>>
>>>>> on
>>>>> are shaded. The shaded artifact does not evolve, there are not more
>>>>> changes
>>>>>
>>>>>> to com.datatorrent classes after they are relocated in master.
>>>>>>
>>>>>> Thanks,
>>>>>> Thomas
>>>>>>
>>>>>>
>>>>>> On Sun, Jul 16, 2017 at 2:00 AM, Pramod Immaneni <
>>>>>>
>>>>> pramod@datatorrent.com
>>>>> wrote:
>>>>>> I don't think we can limit the topic strictly to relocation without
>>>>>> having
>>>>>> a good b/w compatibility story or at least one that goes far enough.
>>>>>>> The shading idea sounds interesting. Why not let the shaded version
>>>>>>>
>>>>>> move
>>>>> forward with each release till we hit a major release. If it is going
>>>>>> to
>>>>> remain pegged at 3.7.0, why shade in the first place as the regular
>>>>>> 3.7.0
>>>>>> release would do the same job and it would be same as the loss of b/w
>>>>>>> compatibility with newer releases.
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>> On Sat, Jul 15, 2017 at 7:57 AM, Thomas Weise <th...@apache.org> wrote:
>>>>>>>
>>>>>>> Discussing what in the future might become stable needs to be a
>>>>>>> separate
>>>>>> thread, it will be a much bigger discussion.
>>>>>>>> The topic here is to relocate the packages. With a few exceptions
>>>>>>>> relocation won't affect the semantic versioning. Semantic versioning
>>>>>>>>
>>>>>>> is
>>>>>> essentially not effective for Malhar because almost everything is
>>>>>>> @Evolving
>>>>>>>
>>>>>>>> (and there are reasons for that.. -> separate topic)
>>>>>>>>
>>>>>>>> I don't really like the idea of creating bw compatibility stubs for
>>>>>>>>
>>>>>>> the
>>>>>> follow up PR. It creates even more clutter in the source tree than
>>>>>>> there
>>>>>> already is and so here is an alternative suggestion:
>>>>>>>> https://github.com/tweise/apex-malhar/blob/malhar37-
>>>>>>>> compat/shaded-malhar37/pom.xml
>>>>>>>>
>>>>>>>> Create a shaded artifact that provides the old com.datatorrent.*
>>>>>>>>
>>>>>>> classes as
>>>>>>>
>>>>>>>> of release 3.7. Users can include that artifact if they don't want
>>>>>>>>
>>>>>>> to
>>>>> change import statements. At the same time they have an incentive to
>>>>>>> switch
>>>>>>>
>>>>>>>> to the relocated classes to take advantage of bug fixes and new
>>>>>>>> functionality.
>>>>>>>>
>>>>>>>> I will work on the first PR that does the relocate. In the meantime,
>>>>>>>>
>>>>>>> we
>>>>>> can
>>>>>>>> finalize what backward compatibility support we want to provide and
>>>>>>>>
>>>>>>> how.
>>>>>> Thanks,
>>>>>>>> Thomas
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Jul 14, 2017 at 11:33 AM, Pramod Immaneni <
>>>>>>>>
>>>>>>> pramod@datatorrent.com>
>>>>>>>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> How about coming up with a list of @Evolving operators that we
>>>>>>>> would
>>>>> like
>>>>>>>> to see in the eventual stable list and move those along with the
>>>>>>>> not
>>>>> @Evolving ones in org.apache.apex with b/w stubs and leave the
>>>>>>>> rest
>>>>> as
>>>>>
>>>>>> they
>>>>>>>>> are. Then have a follow up JIRA for the rest to be moved over to
>>>>>>>>>
>>>>>>>> contrib
>>>>>>>> and be deprecated.
>>>>>>>>> Thanks
>>>>>>>>>
>>>>>>>>> On Fri, Jul 14, 2017 at 10:37 AM, Thomas Weise <
>>>>>>>>>
>>>>>>>> thomas.weise@gmail.com>
>>>>>>>> wrote:
>>>>>>>>> We need to keep the discussion here on topic, if other things
>>>>>>>>> are
>>>>> piled
>>>>>>>> on
>>>>>>>>>> top then nothing gets done.
>>>>>>>>>>
>>>>>>>>>> Most operators today are not stable, they are @Evolving. So
>>>>>>>>>>
>>>>>>>>> perhaps
>>>>>> it
>>>>>>>> is
>>>>>>>>
>>>>>>>>> necessary to have a separate discussion about that, outside of
>>>>>>>>> this
>>>>>> thread.
>>>>>>>>>> My guess is that there are only a few operators that we could
>>>>>>>>>>
>>>>>>>>> declare
>>>>>>>> stable. Specifically, under contrib the closest one might have
>>>>>>>>> been
>>>>>> Kafka,
>>>>>>>>>> but that is already superseded by the newer versions.
>>>>>>>>>>
>>>>>>>>>> Thomas
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Fri, Jul 14, 2017 at 10:21 AM, Pramod Immaneni <
>>>>>>>>>>
>>>>>>>>> pramod@datatorrent.com>
>>>>>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> We had a discussion a while back, agreed to relegate
>>>>>>>>>> non-stable
>>>>> and
>>>>>>>> experimental operators to contrib and also added this to
>>>>>>>>>> contribution
>>>>>>>> guidelines. We aexecuted on this and cleaned up the repo by
>>>>>>>>>> moving
>>>>>> operators deemed non-suitable for production use at that time
>>>>>>>>>> to
>>>>> contrib.
>>>>>>>>>> So I wouldn't say the operators that are at the top level
>>>>>>>>>> today
>>>>> or
>>>>>
>>>>>> the
>>>>>>>>> ones
>>>>>>>>>>> in library are 0.x.x quality. Granted, we may need to do one
>>>>>>>>>>>
>>>>>>>>>> more
>>>>>> pass
>>>>>>>>> as
>>>>>>>>>
>>>>>>>>>> some of the operators may no longer be considered the right
>>>>>>>>>> implementations
>>>>>>>>>>
>>>>>>>>>>> with the advent of the windowed operator.
>>>>>>>>>>>
>>>>>>>>>>> Since contrib used to be the place that housed operators that
>>>>>>>>>>>
>>>>>>>>>> required
>>>>>>>>> third party libraries, there are some production quality
>>>>>>>>>> operators in
>>>>>>>> there
>>>>>>>>>>> that need to be pulled out into top level modules.
>>>>>>>>>>>
>>>>>>>>>>> I think we are in agreement that for operators that we
>>>>>>>>>>>
>>>>>>>>>> consider
>>>>> stable,
>>>>>>>>> we
>>>>>>>>>>> should provide a b/w stub. I would add that we strongly
>>>>>>>>>>>
>>>>>>>>>> consider
>>>>> creating
>>>>>>>>>> the org.apache.apex counterparts of any stable operators that
>>>>>>>>>> are
>>>>>> in
>>>>>>>> contrib out in top level modules and have the com.datatorrent
>>>>>>>>>> stubs
>>>>>>>> in
>>>>>>>>
>>>>>>>>> contrib.
>>>>>>>>>>> For the operators not considered stable, I would prefer we
>>>>>>>>>>>
>>>>>>>>>> either
>>>>>> leave
>>>>>>>>> the
>>>>>>>>>>> current package path as is or if we change the package path
>>>>>>>>>>>
>>>>>>>>>> then
>>>>> create
>>>>>>>>> the
>>>>>>>>>>> b/w stub as I am not sure how widely they are in use (so, in
>>>>>>>>>>>
>>>>>>>>>> essence,
>>>>>>>> preserve semantic versioning). It would be good if there is a
>>>>>>>>>> followup
>>>>>>>>> JIRA
>>>>>>>>>>> that takes a look at what other operators can be moved to
>>>>>>>>>>>
>>>>>>>>>> contrib
>>>>>> with
>>>>>>>>> the
>>>>>>>>>>> advent of the newer frameworks and understanding.
>>>>>>>>>>>
>>>>>>>>>>> Thanks
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Jul 14, 2017 at 9:44 AM, Thomas Weise <thw@apache.org
>>>>>>>>>>>
>>>>>>>>>> wrote:
>>>>>>>>> Most of the operators evolve, as is quite clearly indicated
>>>>>>>>>>> by
>>>>> @Evolving
>>>>>>>>>>> annotations. So while perhaps 0.x.x would be a more
>>>>>>>>>>> appropriate
>>>>>> version
>>>>>>>>>> number, I don't think you can change that.
>>>>>>>>>>>> Thomas
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Jul 14, 2017 at 9:35 AM, Vlad Rozov <
>>>>>>>>>>>>
>>>>>>>>>>> v.rozov@datatorrent.com
>>>>>>>>> wrote:
>>>>>>>>>>>> If entire library is not stable, its version should be
>>>>>>>>>>>> 0.x.x
>>>>> and
>>>>>>>> there
>>>>>>>>>>> should not be any semantic versioning enabled or implied.
>>>>>>>>>>>> It
>>>>> evolves.
>>>>>>>>>> If
>>>>>>>>>>>> some operators are stable as 3.8.x version suggests, the
>>>>>>>>>>>> library
>>>>>>>> should
>>>>>>>>>>> follow semantic versioning and it is not OK to make a
>>>>>>>>>>>> major
>>>>> change
>>>>>>>>> that
>>>>>>>>>>> breaks semantic versioning across the entire library. It
>>>>>>>>>>>> is
>>>>> not a
>>>>>>>> finding
>>>>>>>>>>>> an excuse not to make a change. For me semantic versioning
>>>>>>>>>>>> and
>>>>>> backward
>>>>>>>>>>> compatibility is more important than a proper package
>>>>>>>>>>>> name.
>>>>> Thank you,
>>>>>>>>>>>>> Vlad
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 7/14/17 09:11, Thomas Weise wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Semantic versioning makes sense when you have a stable
>>>>>>>>>>>>> baseline.
>>>>>>>> As
>>>>>>>>>> long
>>>>>>>>>>>> as
>>>>>>>>>>>>>> frequent fixes need to be made to address basic issues,
>>>>>>>>>>>>>>
>>>>>>>>>>>>> it
>>>>> makes
>>>>>>>> no
>>>>>>>>>> sense
>>>>>>>>>>>>> to declare operators stable, which is why they are marked
>>>>>>>>>>>>> evolving.
>>>>>>>>>> Instead of finding reasons to avoid changes that should
>>>>>>>>>>>>> have
>>>>>> been
>>>>>>>>> made a
>>>>>>>>>>>> long time ago, why not discuss what a "stable operator"
>>>>>>>>>>>>> is
>>>>> and
>>>>>>>> which
>>>>>>>>>> ones
>>>>>>>>>>>>> deserve to be in that category. Those are the ones that
>>>>>>>>>>>>> will
>>>>>> get
>>>>>>>> the
>>>>>>>>>> backward compatibility stub.
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> Thomas
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Jul 14, 2017 at 8:34 AM, Vlad Rozov <
>>>>>>>>>>>>>>
>>>>>>>>>>>>> v.rozov@datatorrent.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> My preference is to agree that the next Malhar release is
>>>>>>>>>>>>>>
>>>>>>>>>>>>> 4.0.0
>>>>>>>> and
>>>>>>>>>> make
>>>>>>>>>>>> the proposed changes when 3.8.0 goes out. Otherwise why
>>>>>>>>>>>>>> to
>>>>> keep
>>>>>>>> semantic
>>>>>>>>>>>>> versioning checks on Malhar in case there is no version
>>>>>>>>>>>>>> compatibility.
>>>>>>>>>>>> Thank you,
>>>>>>>>>>>>>>> Vlad
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 7/14/17 08:11, Thomas Weise wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> next release is 3.8.0
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Fri, Jul 14, 2017 at 7:55 AM, Vlad Rozov <
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> v.rozov@datatorrent.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>> next release - 3.9.0 or 4.0.0?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thank you,
>>>>>>>>>>>>>>>>> Vlad
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On 7/13/17 22:25, Thomas Weise wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> It is time to resurrect this thread and get going with
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> the
>>>>>> work.
>>>>>>>>>> For the next release, I will sign up to do the
>>>>>>>>>>>>>>>>> package
>>>>> move
>>>>>>>> in
>>>>>>>>
>>>>>>>>> Malhar:
>>>>>>>>>>>>> https://issues.apache.org/
>>>>>>>>>>>>>>>>> jira/browse/APEXMALHAR-2517
>>>>> In general this will be straightforward; most classes
>>>>>>>>>>>>>>>>> in
>>>>>> Malhar
>>>>>>>>>> are
>>>>>>>>>>>> marked
>>>>>>>>>>>>>>>>>> evolving and it is trivial for users to change import
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> statements.
>>>>>>>>>>> However,
>>>>>>>>>>>>>>>>>> I would suggest to discuss if there are selected
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> operators
>>>>>>>> that
>>>>>>>>>> are
>>>>>>>>>>>> worth
>>>>>>>>>>>>>>>>>> keeping a backward compatibility stub in the old
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> location.
>>>>>>>> Here is my plan:
>>>>>>>>>>>>>>>>>> 1. Relocate all classes in *lib* and *contrib* within
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> one
>>>>>> PR -
>>>>>>>>> this
>>>>>>>>>>>> is
>>>>>>>>>>>>
>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>>> IDE automated work
>>>>>>>>>>>>>>>>>> 2. Add backward compatibility classes, if, any in
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> separate
>>>>>>>> PR
>>>>>>>>
>>>>>>>>> 3. Create PR for Megh library to reflect moved
>>>>>>>>>>>>>>>>> classes
>>>>> Thanks,
>>>>>>>>>>>>>>>>>> Thomas
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Wed, Mar 1, 2017 at 2:24 PM, Pramod Immaneni <
>>>>>>>>>>>>>>>>>> pramod@datatorrent.com
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Inline
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Mon, Feb 27, 2017 at 11:03 PM, Thomas Weise <
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> thw@apache.org>
>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>> -->
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Mon, Feb 27, 2017 at 1:27 PM, Pramod Immaneni <
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> pramod@datatorrent.com
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> For malhar, for existing operators, I prefer we do
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> this as
>>>>>>>> part
>>>>>>>>>>> of
>>>>>>>>>>>
>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>> planned refactoring for breaking the monolith
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> modules
>>>>> into
>>>>>>>> baby
>>>>>>>>>>> packages
>>>>>>>>>>>>>>>>>>>>> and would also prefer deprecating the existing
>>>>>>>>>>>>>>>>>>> operators
>>>>>>>> in
>>>>>>>>
>>>>>>>>> place.
>>>>>>>>>>>> Refactor into smaller modules was discussed for
>>>>>>>>>>>>>>>>>>> malhar-contrib
>>>>>>>>>> and
>>>>>>>>>>>> given
>>>>>>>>>>>>>>>>>>>> the overall state of that module I think it is OK
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> to
>>>>> defer
>>>>>>>> package
>>>>>>>>>>>> renaming
>>>>>>>>>>>>>>>>>>>> there. I do however prefer to see the package rename
>>>>>>>>>>>>>>>>>> addressed
>>>>>>>>>> for
>>>>>>>>>>>> other
>>>>>>>>>>>>>>>>>>>> modules, especially for the main library module.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Should we consider breaking the library into
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> smaller
>>>>> modules
>>>>>>>>> as
>>>>>>>>>>> well,
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> file/block operators for example probably can be in
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> their
>>>>>>>> own
>>>>>>>>
>>>>>>>>> module
>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>>>>> just an organizational perspective.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> This
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> will help us achieve two things. First, the user
>>>>>>>>>>>>>>>>>>> will
>>>>> see
>>>>>>>> all
>>>>>>>>>> the
>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>>>> changes at once as opposed to dealing with it
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> twice
>>>>> (with
>>>>>>>> package
>>>>>>>>>>>> rename
>>>>>>>>>>>>>>>>>>>>> and dependency changes) and second it will allow
>>>>>>>>>>>>>>>>>>> for
>>>>> a
>>>>>
>>>>>> smoother
>>>>>>>>>>> transition
>>>>>>>>>>>>>>>>>>>> as the existing code will still work in a
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> deprecated
>>>>> state.
>>>>>>>>> It
>>>>>>>>>
>>>>>>>>>> will
>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>>>>> give a more consistent structure to malhar. For new
>>>>>>>>>>>>>>>>>>> operators,
>>>>>>>>>> we
>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>>> go
>>>>>>>>>>>>>>>>>>>> with the new package path but we need to ensure
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> they
>>>>> will
>>>>>>>> get
>>>>>>>>>> moved
>>>>>>>>>>>>> into
>>>>>>>>>>>>>>>>>>>> the baby packages as well.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I think existing operators should be renamed so
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> that
>>>>> git
>>>>>>>> history
>>>>>>>>>>> remains. A
>>>>>>>>>>>>>>>>>>>> possible solution for backward compatibility could
>>>>>>>>>>>>>>>>>> be
>>>>> to
>>>>>
>>>>>> subsequently
>>>>>>>>>>>>> add
>>>>>>>>>>>>>>>>>>>> empty subclasses in the previous location (for
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> existing
>>>>
Thank you,
Vlad
Re: Java packages: legacy -> org.apache.apex
Posted by Pramod Immaneni <pr...@datatorrent.com>.
Would library module then become apex-library-library? Why change the
malhar artifact id as the project is called apex-malhar.
Thanks
On Sat, Aug 19, 2017 at 10:03 AM, Vlad Rozov <vr...@apache.org> wrote:
> Overall I support the proposal and this is what I was in favor of during
> the last discussion. I'd also propose changing the name of the artifact id
> and git repo from malhar to apex-library as part of the major version
> change. It will make it more consistent with the entire project and will
> also allow using 1.0 (1.0-SNAPSHOT) version that more closely reflects
> maturity of the library.
>
> Thank you,
>
> Vlad
>
> On 8/18/17 17:04, Thomas Weise wrote:
>
>> The proposed change does NOT require changes to existing apps. It follows
>> what you see for example in the JDK where older classes are retired
>> without
>> making breaking major release changes.
>>
>> Even though not required technically, it looks like the preference is to
>> make this change with a 4.0 release. I will therefore propose to change
>> the
>> master branch to 4.0-SNAPSHOT and make this and other changes deemed
>> worthwhile.
>>
>> To me it is more important that to newcomers the codebase does not look
>> like an outdated legacy situation and therefore if I have to make a choice
>> than pretty much any version number will do for that. So far I have seen
>> very little interest and initiative in addressing maintenance work like
>> the
>> packages, checkstyle, CI stability, documentation etc. So it would be
>> great
>> if people interested would step up and contribute.
>>
>> Here is the next proposal how to proceed:
>>
>> - create release-3.8 branch for last 3.8 release right away
>> - update master to 4.0-SNAPSHOT as part of the open PR #664
>> - identify further cleanup work for master
>> - release 4.0
>>
>> Thanks,
>> Thomas
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> On Thu, Aug 17, 2017 at 4:48 PM, Amol Kekre <am...@datatorrent.com> wrote:
>>
>> This following pull request should be taken up in 4.0.0. See my comments
>>> in
>>> https://github.com/apache/apex-malhar/pull/664
>>> <https://www.google.com/url?q=https%3A%2F%2Fgithub.com%
>>> 2Fapache%2Fapex-malhar%2Fpull%2F664&sa=D&ust=1503020269444000&usg=
>>> AFQjCNHDPZIg2e6I33Jb1XjB5Ir1FkdURQ>
>>> https://github.com/apache/apex-malhar/pull/662
>>> <https://www.google.com/url?q=https%3A%2F%2Fgithub.com%
>>> 2Fapache%2Fapex-malhar%2Fpull%2F662&sa=D&ust=1503020269444000&usg=
>>> AFQjCNF7Xde7X55M3qmc8z-D5y3ZwNg7Fg>
>>>
>>> This merge should not be done without a consensus. This will require code
>>> changes to existing apps.
>>>
>>> Thks,
>>> Amol
>>>
>>>
>>>
>>> E:amol@datatorrent.com | M: 510-449-2606 | Twitter: @*amolhkekre*
>>>
>>> www.datatorrent.com
>>>
>>>
>>> On Mon, Aug 14, 2017 at 7:48 PM, Thomas Weise <th...@apache.org> wrote:
>>>
>>> Hi,
>>>>
>>>> I opened following PRs for the package change:
>>>>
>>>> https://github.com/apache/apex-malhar/pull/662
>>>>
>>>> Moves all classes with history retained (hence 2 commits). Also contains
>>>> checkstyle and other mechanical changes.
>>>>
>>>> https://github.com/apache/apex-malhar/pull/664
>>>>
>>>> Adds backward compatibility jar.
>>>>
>>>> Once above PRs are merged the new artifact can be deployed and
>>>> introduced
>>>> as dependency in malhar-library.
>>>>
>>>> Please review.
>>>>
>>>> Thanks,
>>>> Thomas
>>>>
>>>>
>>>>
>>>> On Sun, Jul 16, 2017 at 7:04 AM, Thomas Weise <th...@apache.org> wrote:
>>>>
>>>> My original list of work items contained the b/w compatibility aspect,
>>>>>
>>>> I
>>>
>>>> don't think there should be any confusion of whether it will be covered
>>>>> here or not.
>>>>>
>>>>> The proposed shading will provide the old classes and they will be
>>>>>
>>>> frozen
>>>
>>>> as of release 3.7. That's the same as making a copy of the code and
>>>>>
>>>> never
>>>
>>>> again making changes to the original classes. This cannot be
>>>>>
>>>> accomplished
>>>
>>>> by using the older 3.7 release in your project because you cannot use 2
>>>>> different versions of Malhar in parallel unless you apply shading.
>>>>>
>>>>> The shaded artifact will only expose the com.datatorrent classes, and
>>>>>
>>>> they
>>>>
>>>>> will be self-contained as the rest of the classes that they may depend
>>>>>
>>>> on
>>>
>>>> are shaded. The shaded artifact does not evolve, there are not more
>>>>>
>>>> changes
>>>>
>>>>> to com.datatorrent classes after they are relocated in master.
>>>>>
>>>>> Thanks,
>>>>> Thomas
>>>>>
>>>>>
>>>>> On Sun, Jul 16, 2017 at 2:00 AM, Pramod Immaneni <
>>>>>
>>>> pramod@datatorrent.com
>>>
>>>> wrote:
>>>>>
>>>>> I don't think we can limit the topic strictly to relocation without
>>>>>>
>>>>> having
>>>>
>>>>> a good b/w compatibility story or at least one that goes far enough.
>>>>>>
>>>>>> The shading idea sounds interesting. Why not let the shaded version
>>>>>>
>>>>> move
>>>
>>>> forward with each release till we hit a major release. If it is going
>>>>>>
>>>>> to
>>>
>>>> remain pegged at 3.7.0, why shade in the first place as the regular
>>>>>>
>>>>> 3.7.0
>>>>
>>>>> release would do the same job and it would be same as the loss of b/w
>>>>>> compatibility with newer releases.
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> On Sat, Jul 15, 2017 at 7:57 AM, Thomas Weise <th...@apache.org> wrote:
>>>>>>
>>>>>> Discussing what in the future might become stable needs to be a
>>>>>>>
>>>>>> separate
>>>>
>>>>> thread, it will be a much bigger discussion.
>>>>>>>
>>>>>>> The topic here is to relocate the packages. With a few exceptions
>>>>>>> relocation won't affect the semantic versioning. Semantic versioning
>>>>>>>
>>>>>> is
>>>>
>>>>> essentially not effective for Malhar because almost everything is
>>>>>>>
>>>>>> @Evolving
>>>>>>
>>>>>>> (and there are reasons for that.. -> separate topic)
>>>>>>>
>>>>>>> I don't really like the idea of creating bw compatibility stubs for
>>>>>>>
>>>>>> the
>>>>
>>>>> follow up PR. It creates even more clutter in the source tree than
>>>>>>>
>>>>>> there
>>>>
>>>>> already is and so here is an alternative suggestion:
>>>>>>>
>>>>>>> https://github.com/tweise/apex-malhar/blob/malhar37-
>>>>>>> compat/shaded-malhar37/pom.xml
>>>>>>>
>>>>>>> Create a shaded artifact that provides the old com.datatorrent.*
>>>>>>>
>>>>>> classes as
>>>>>>
>>>>>>> of release 3.7. Users can include that artifact if they don't want
>>>>>>>
>>>>>> to
>>>
>>>> change import statements. At the same time they have an incentive to
>>>>>>>
>>>>>> switch
>>>>>>
>>>>>>> to the relocated classes to take advantage of bug fixes and new
>>>>>>> functionality.
>>>>>>>
>>>>>>> I will work on the first PR that does the relocate. In the meantime,
>>>>>>>
>>>>>> we
>>>>
>>>>> can
>>>>>>
>>>>>>> finalize what backward compatibility support we want to provide and
>>>>>>>
>>>>>> how.
>>>>
>>>>> Thanks,
>>>>>>> Thomas
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Jul 14, 2017 at 11:33 AM, Pramod Immaneni <
>>>>>>>
>>>>>> pramod@datatorrent.com>
>>>>>>
>>>>>>> wrote:
>>>>>>>
>>>>>>> How about coming up with a list of @Evolving operators that we
>>>>>>>>
>>>>>>> would
>>>
>>>> like
>>>>>>
>>>>>>> to see in the eventual stable list and move those along with the
>>>>>>>>
>>>>>>> not
>>>
>>>> @Evolving ones in org.apache.apex with b/w stubs and leave the
>>>>>>>>
>>>>>>> rest
>>>
>>>> as
>>>>
>>>>> they
>>>>>>>
>>>>>>>> are. Then have a follow up JIRA for the rest to be moved over to
>>>>>>>>
>>>>>>> contrib
>>>>>>
>>>>>>> and be deprecated.
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>>
>>>>>>>> On Fri, Jul 14, 2017 at 10:37 AM, Thomas Weise <
>>>>>>>>
>>>>>>> thomas.weise@gmail.com>
>>>>>>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> We need to keep the discussion here on topic, if other things
>>>>>>>>>
>>>>>>>> are
>>>
>>>> piled
>>>>>>
>>>>>>> on
>>>>>>>>
>>>>>>>>> top then nothing gets done.
>>>>>>>>>
>>>>>>>>> Most operators today are not stable, they are @Evolving. So
>>>>>>>>>
>>>>>>>> perhaps
>>>>
>>>>> it
>>>>>>
>>>>>>> is
>>>>>>>
>>>>>>>> necessary to have a separate discussion about that, outside of
>>>>>>>>>
>>>>>>>> this
>>>>
>>>>> thread.
>>>>>>>>
>>>>>>>>> My guess is that there are only a few operators that we could
>>>>>>>>>
>>>>>>>> declare
>>>>>>
>>>>>>> stable. Specifically, under contrib the closest one might have
>>>>>>>>>
>>>>>>>> been
>>>>
>>>>> Kafka,
>>>>>>>>
>>>>>>>>> but that is already superseded by the newer versions.
>>>>>>>>>
>>>>>>>>> Thomas
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, Jul 14, 2017 at 10:21 AM, Pramod Immaneni <
>>>>>>>>>
>>>>>>>> pramod@datatorrent.com>
>>>>>>>>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> We had a discussion a while back, agreed to relegate
>>>>>>>>>>
>>>>>>>>> non-stable
>>>
>>>> and
>>>>>>
>>>>>>> experimental operators to contrib and also added this to
>>>>>>>>>>
>>>>>>>>> contribution
>>>>>>
>>>>>>> guidelines. We aexecuted on this and cleaned up the repo by
>>>>>>>>>>
>>>>>>>>> moving
>>>>
>>>>> operators deemed non-suitable for production use at that time
>>>>>>>>>>
>>>>>>>>> to
>>>
>>>> contrib.
>>>>>>>>
>>>>>>>>> So I wouldn't say the operators that are at the top level
>>>>>>>>>>
>>>>>>>>> today
>>>
>>>> or
>>>>
>>>>> the
>>>>>>>
>>>>>>>> ones
>>>>>>>>>
>>>>>>>>>> in library are 0.x.x quality. Granted, we may need to do one
>>>>>>>>>>
>>>>>>>>> more
>>>>
>>>>> pass
>>>>>>>
>>>>>>>> as
>>>>>>>>
>>>>>>>>> some of the operators may no longer be considered the right
>>>>>>>>>>
>>>>>>>>> implementations
>>>>>>>>>
>>>>>>>>>> with the advent of the windowed operator.
>>>>>>>>>>
>>>>>>>>>> Since contrib used to be the place that housed operators that
>>>>>>>>>>
>>>>>>>>> required
>>>>>>>
>>>>>>>> third party libraries, there are some production quality
>>>>>>>>>>
>>>>>>>>> operators in
>>>>>>
>>>>>>> there
>>>>>>>>>
>>>>>>>>>> that need to be pulled out into top level modules.
>>>>>>>>>>
>>>>>>>>>> I think we are in agreement that for operators that we
>>>>>>>>>>
>>>>>>>>> consider
>>>
>>>> stable,
>>>>>>>
>>>>>>>> we
>>>>>>>>>
>>>>>>>>>> should provide a b/w stub. I would add that we strongly
>>>>>>>>>>
>>>>>>>>> consider
>>>
>>>> creating
>>>>>>>>
>>>>>>>>> the org.apache.apex counterparts of any stable operators that
>>>>>>>>>>
>>>>>>>>> are
>>>>
>>>>> in
>>>>>>
>>>>>>> contrib out in top level modules and have the com.datatorrent
>>>>>>>>>>
>>>>>>>>> stubs
>>>>>>
>>>>>>> in
>>>>>>>
>>>>>>>> contrib.
>>>>>>>>>>
>>>>>>>>>> For the operators not considered stable, I would prefer we
>>>>>>>>>>
>>>>>>>>> either
>>>>
>>>>> leave
>>>>>>>
>>>>>>>> the
>>>>>>>>>
>>>>>>>>>> current package path as is or if we change the package path
>>>>>>>>>>
>>>>>>>>> then
>>>
>>>> create
>>>>>>>
>>>>>>>> the
>>>>>>>>>
>>>>>>>>>> b/w stub as I am not sure how widely they are in use (so, in
>>>>>>>>>>
>>>>>>>>> essence,
>>>>>>
>>>>>>> preserve semantic versioning). It would be good if there is a
>>>>>>>>>>
>>>>>>>>> followup
>>>>>>>
>>>>>>>> JIRA
>>>>>>>>>
>>>>>>>>>> that takes a look at what other operators can be moved to
>>>>>>>>>>
>>>>>>>>> contrib
>>>>
>>>>> with
>>>>>>>
>>>>>>>> the
>>>>>>>>>
>>>>>>>>>> advent of the newer frameworks and understanding.
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>>
>>>>>>>>>> On Fri, Jul 14, 2017 at 9:44 AM, Thomas Weise <thw@apache.org
>>>>>>>>>>
>>>>>>>>> wrote:
>>>>>>>
>>>>>>>> Most of the operators evolve, as is quite clearly indicated
>>>>>>>>>>>
>>>>>>>>>> by
>>>
>>>> @Evolving
>>>>>>>>>
>>>>>>>>>> annotations. So while perhaps 0.x.x would be a more
>>>>>>>>>>>
>>>>>>>>>> appropriate
>>>>
>>>>> version
>>>>>>>>
>>>>>>>>> number, I don't think you can change that.
>>>>>>>>>>>
>>>>>>>>>>> Thomas
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Jul 14, 2017 at 9:35 AM, Vlad Rozov <
>>>>>>>>>>>
>>>>>>>>>> v.rozov@datatorrent.com
>>>>>>>
>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> If entire library is not stable, its version should be
>>>>>>>>>>>>
>>>>>>>>>>> 0.x.x
>>>
>>>> and
>>>>>>
>>>>>>> there
>>>>>>>>>
>>>>>>>>>> should not be any semantic versioning enabled or implied.
>>>>>>>>>>>>
>>>>>>>>>>> It
>>>
>>>> evolves.
>>>>>>>>
>>>>>>>>> If
>>>>>>>>>>
>>>>>>>>>>> some operators are stable as 3.8.x version suggests, the
>>>>>>>>>>>>
>>>>>>>>>>> library
>>>>>>
>>>>>>> should
>>>>>>>>>
>>>>>>>>>> follow semantic versioning and it is not OK to make a
>>>>>>>>>>>>
>>>>>>>>>>> major
>>>
>>>> change
>>>>>>>
>>>>>>>> that
>>>>>>>>>
>>>>>>>>>> breaks semantic versioning across the entire library. It
>>>>>>>>>>>>
>>>>>>>>>>> is
>>>
>>>> not a
>>>>>>
>>>>>>> finding
>>>>>>>>>>
>>>>>>>>>>> an excuse not to make a change. For me semantic versioning
>>>>>>>>>>>>
>>>>>>>>>>> and
>>>>
>>>>> backward
>>>>>>>>>
>>>>>>>>>> compatibility is more important than a proper package
>>>>>>>>>>>>
>>>>>>>>>>> name.
>>>
>>>> Thank you,
>>>>>>>>>>>>
>>>>>>>>>>>> Vlad
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 7/14/17 09:11, Thomas Weise wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Semantic versioning makes sense when you have a stable
>>>>>>>>>>>>>
>>>>>>>>>>>> baseline.
>>>>>>
>>>>>>> As
>>>>>>>>
>>>>>>>>> long
>>>>>>>>>>
>>>>>>>>>>> as
>>>>>>>>>>>>> frequent fixes need to be made to address basic issues,
>>>>>>>>>>>>>
>>>>>>>>>>>> it
>>>
>>>> makes
>>>>>>
>>>>>>> no
>>>>>>>>
>>>>>>>>> sense
>>>>>>>>>>>
>>>>>>>>>>>> to declare operators stable, which is why they are marked
>>>>>>>>>>>>>
>>>>>>>>>>>> evolving.
>>>>>>>>
>>>>>>>>> Instead of finding reasons to avoid changes that should
>>>>>>>>>>>>>
>>>>>>>>>>>> have
>>>>
>>>>> been
>>>>>>>
>>>>>>>> made a
>>>>>>>>>>
>>>>>>>>>>> long time ago, why not discuss what a "stable operator"
>>>>>>>>>>>>>
>>>>>>>>>>>> is
>>>
>>>> and
>>>>>>
>>>>>>> which
>>>>>>>>
>>>>>>>>> ones
>>>>>>>>>>>
>>>>>>>>>>>> deserve to be in that category. Those are the ones that
>>>>>>>>>>>>>
>>>>>>>>>>>> will
>>>>
>>>>> get
>>>>>>
>>>>>>> the
>>>>>>>>
>>>>>>>>> backward compatibility stub.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Thomas
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Jul 14, 2017 at 8:34 AM, Vlad Rozov <
>>>>>>>>>>>>>
>>>>>>>>>>>> v.rozov@datatorrent.com>
>>>>>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> My preference is to agree that the next Malhar release is
>>>>>>>>>>>>>
>>>>>>>>>>>> 4.0.0
>>>>>>
>>>>>>> and
>>>>>>>>
>>>>>>>>> make
>>>>>>>>>>
>>>>>>>>>>> the proposed changes when 3.8.0 goes out. Otherwise why
>>>>>>>>>>>>>>
>>>>>>>>>>>>> to
>>>
>>>> keep
>>>>>>
>>>>>>> semantic
>>>>>>>>>>>
>>>>>>>>>>>> versioning checks on Malhar in case there is no version
>>>>>>>>>>>>>>
>>>>>>>>>>>>> compatibility.
>>>>>>>>>>
>>>>>>>>>>> Thank you,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Vlad
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 7/14/17 08:11, Thomas Weise wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> next release is 3.8.0
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, Jul 14, 2017 at 7:55 AM, Vlad Rozov <
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> v.rozov@datatorrent.com>
>>>>>>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> next release - 3.9.0 or 4.0.0?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thank you,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Vlad
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 7/13/17 22:25, Thomas Weise wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> It is time to resurrect this thread and get going with
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> the
>>>>
>>>>> work.
>>>>>>>>
>>>>>>>>> For the next release, I will sign up to do the
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> package
>>>
>>>> move
>>>>>>
>>>>>>> in
>>>>>>>
>>>>>>>> Malhar:
>>>>>>>>>>>
>>>>>>>>>>>> https://issues.apache.org/
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> jira/browse/APEXMALHAR-2517
>>>
>>>> In general this will be straightforward; most classes
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> in
>>>>
>>>>> Malhar
>>>>>>>>
>>>>>>>>> are
>>>>>>>>>>
>>>>>>>>>>> marked
>>>>>>>>>>>>>>>>> evolving and it is trivial for users to change import
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> statements.
>>>>>>>>>
>>>>>>>>>> However,
>>>>>>>>>>>>>>>>> I would suggest to discuss if there are selected
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> operators
>>>>>>
>>>>>>> that
>>>>>>>>
>>>>>>>>> are
>>>>>>>>>>
>>>>>>>>>>> worth
>>>>>>>>>>>>>>>>> keeping a backward compatibility stub in the old
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> location.
>>>>>>
>>>>>>> Here is my plan:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 1. Relocate all classes in *lib* and *contrib* within
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> one
>>>>
>>>>> PR -
>>>>>>>
>>>>>>>> this
>>>>>>>>>>
>>>>>>>>>>> is
>>>>>>>>>>>
>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> IDE automated work
>>>>>>>>>>>>>>>>> 2. Add backward compatibility classes, if, any in
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> separate
>>>>>>
>>>>>>> PR
>>>>>>>
>>>>>>>> 3. Create PR for Megh library to reflect moved
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> classes
>>>
>>>> Thanks,
>>>>>>>>>>>>>>>>> Thomas
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Wed, Mar 1, 2017 at 2:24 PM, Pramod Immaneni <
>>>>>>>>>>>>>>>>> pramod@datatorrent.com
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Inline
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Mon, Feb 27, 2017 at 11:03 PM, Thomas Weise <
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> thw@apache.org>
>>>>>>>>
>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> -->
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Mon, Feb 27, 2017 at 1:27 PM, Pramod Immaneni <
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> pramod@datatorrent.com
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> For malhar, for existing operators, I prefer we do
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> this as
>>>>>>
>>>>>>> part
>>>>>>>>>
>>>>>>>>>> of
>>>>>>>>>>
>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> planned refactoring for breaking the monolith
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> modules
>>>
>>>> into
>>>>>>
>>>>>>> baby
>>>>>>>>>
>>>>>>>>>> packages
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> and would also prefer deprecating the existing
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> operators
>>>>>>
>>>>>>> in
>>>>>>>
>>>>>>>> place.
>>>>>>>>>>
>>>>>>>>>>> Refactor into smaller modules was discussed for
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> malhar-contrib
>>>>>>>>
>>>>>>>>> and
>>>>>>>>>>
>>>>>>>>>>> given
>>>>>>>>>>>>>>>>>>> the overall state of that module I think it is OK
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> to
>>>
>>>> defer
>>>>>>
>>>>>>> package
>>>>>>>>>>
>>>>>>>>>>> renaming
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> there. I do however prefer to see the package rename
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> addressed
>>>>>>>>
>>>>>>>>> for
>>>>>>>>>>
>>>>>>>>>>> other
>>>>>>>>>>>>>>>>>>> modules, especially for the main library module.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Should we consider breaking the library into
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> smaller
>>>
>>>> modules
>>>>>>>
>>>>>>>> as
>>>>>>>>>
>>>>>>>>>> well,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> file/block operators for example probably can be in
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> their
>>>>>>
>>>>>>> own
>>>>>>>
>>>>>>>> module
>>>>>>>>>>>
>>>>>>>>>>>> from
>>>>>>>>>>>>>>>>>> just an organizational perspective.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> This
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> will help us achieve two things. First, the user
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> will
>>>
>>>> see
>>>>>>
>>>>>>> all
>>>>>>>>
>>>>>>>>> the
>>>>>>>>>>
>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>>> changes at once as opposed to dealing with it
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> twice
>>>
>>>> (with
>>>>>>
>>>>>>> package
>>>>>>>>>>
>>>>>>>>>>> rename
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> and dependency changes) and second it will allow
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> for
>>>
>>>> a
>>>>
>>>>> smoother
>>>>>>>>>
>>>>>>>>>> transition
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> as the existing code will still work in a
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> deprecated
>>>
>>>> state.
>>>>>>>
>>>>>>>> It
>>>>>>>>
>>>>>>>>> will
>>>>>>>>>>>
>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> give a more consistent structure to malhar. For new
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> operators,
>>>>>>>>
>>>>>>>>> we
>>>>>>>>>>
>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>> go
>>>>>>>>>>>>>>>>>>> with the new package path but we need to ensure
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> they
>>>
>>>> will
>>>>>>
>>>>>>> get
>>>>>>>>
>>>>>>>>> moved
>>>>>>>>>>>
>>>>>>>>>>>> into
>>>>>>>>>>>>>>>>>>> the baby packages as well.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I think existing operators should be renamed so
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> that
>>>
>>>> git
>>>>>>
>>>>>>> history
>>>>>>>>>
>>>>>>>>>> remains. A
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> possible solution for backward compatibility could
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> be
>>>
>>>> to
>>>>
>>>>> subsequently
>>>>>>>>>>>
>>>>>>>>>>>> add
>>>>>>>>>>>>>>>>>>> empty subclasses in the previous location (for
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> existing
>>>
>>>
Re: Java packages: legacy -> org.apache.apex
Posted by Vlad Rozov <vr...@apache.org>.
Overall I support the proposal and this is what I was in favor of during
the last discussion. I'd also propose changing the name of the artifact
id and git repo from malhar to apex-library as part of the major version
change. It will make it more consistent with the entire project and will
also allow using 1.0 (1.0-SNAPSHOT) version that more closely reflects
maturity of the library.
Thank you,
Vlad
On 8/18/17 17:04, Thomas Weise wrote:
> The proposed change does NOT require changes to existing apps. It follows
> what you see for example in the JDK where older classes are retired without
> making breaking major release changes.
>
> Even though not required technically, it looks like the preference is to
> make this change with a 4.0 release. I will therefore propose to change the
> master branch to 4.0-SNAPSHOT and make this and other changes deemed
> worthwhile.
>
> To me it is more important that to newcomers the codebase does not look
> like an outdated legacy situation and therefore if I have to make a choice
> than pretty much any version number will do for that. So far I have seen
> very little interest and initiative in addressing maintenance work like the
> packages, checkstyle, CI stability, documentation etc. So it would be great
> if people interested would step up and contribute.
>
> Here is the next proposal how to proceed:
>
> - create release-3.8 branch for last 3.8 release right away
> - update master to 4.0-SNAPSHOT as part of the open PR #664
> - identify further cleanup work for master
> - release 4.0
>
> Thanks,
> Thomas
>
>
>
>
>
>
>
>
>
> On Thu, Aug 17, 2017 at 4:48 PM, Amol Kekre <am...@datatorrent.com> wrote:
>
>> This following pull request should be taken up in 4.0.0. See my comments in
>> https://github.com/apache/apex-malhar/pull/664
>> <https://www.google.com/url?q=https%3A%2F%2Fgithub.com%
>> 2Fapache%2Fapex-malhar%2Fpull%2F664&sa=D&ust=1503020269444000&usg=
>> AFQjCNHDPZIg2e6I33Jb1XjB5Ir1FkdURQ>
>> https://github.com/apache/apex-malhar/pull/662
>> <https://www.google.com/url?q=https%3A%2F%2Fgithub.com%
>> 2Fapache%2Fapex-malhar%2Fpull%2F662&sa=D&ust=1503020269444000&usg=
>> AFQjCNF7Xde7X55M3qmc8z-D5y3ZwNg7Fg>
>>
>> This merge should not be done without a consensus. This will require code
>> changes to existing apps.
>>
>> Thks,
>> Amol
>>
>>
>>
>> E:amol@datatorrent.com | M: 510-449-2606 | Twitter: @*amolhkekre*
>>
>> www.datatorrent.com
>>
>>
>> On Mon, Aug 14, 2017 at 7:48 PM, Thomas Weise <th...@apache.org> wrote:
>>
>>> Hi,
>>>
>>> I opened following PRs for the package change:
>>>
>>> https://github.com/apache/apex-malhar/pull/662
>>>
>>> Moves all classes with history retained (hence 2 commits). Also contains
>>> checkstyle and other mechanical changes.
>>>
>>> https://github.com/apache/apex-malhar/pull/664
>>>
>>> Adds backward compatibility jar.
>>>
>>> Once above PRs are merged the new artifact can be deployed and introduced
>>> as dependency in malhar-library.
>>>
>>> Please review.
>>>
>>> Thanks,
>>> Thomas
>>>
>>>
>>>
>>> On Sun, Jul 16, 2017 at 7:04 AM, Thomas Weise <th...@apache.org> wrote:
>>>
>>>> My original list of work items contained the b/w compatibility aspect,
>> I
>>>> don't think there should be any confusion of whether it will be covered
>>>> here or not.
>>>>
>>>> The proposed shading will provide the old classes and they will be
>> frozen
>>>> as of release 3.7. That's the same as making a copy of the code and
>> never
>>>> again making changes to the original classes. This cannot be
>> accomplished
>>>> by using the older 3.7 release in your project because you cannot use 2
>>>> different versions of Malhar in parallel unless you apply shading.
>>>>
>>>> The shaded artifact will only expose the com.datatorrent classes, and
>>> they
>>>> will be self-contained as the rest of the classes that they may depend
>> on
>>>> are shaded. The shaded artifact does not evolve, there are not more
>>> changes
>>>> to com.datatorrent classes after they are relocated in master.
>>>>
>>>> Thanks,
>>>> Thomas
>>>>
>>>>
>>>> On Sun, Jul 16, 2017 at 2:00 AM, Pramod Immaneni <
>> pramod@datatorrent.com
>>>> wrote:
>>>>
>>>>> I don't think we can limit the topic strictly to relocation without
>>> having
>>>>> a good b/w compatibility story or at least one that goes far enough.
>>>>>
>>>>> The shading idea sounds interesting. Why not let the shaded version
>> move
>>>>> forward with each release till we hit a major release. If it is going
>> to
>>>>> remain pegged at 3.7.0, why shade in the first place as the regular
>>> 3.7.0
>>>>> release would do the same job and it would be same as the loss of b/w
>>>>> compatibility with newer releases.
>>>>>
>>>>> Thanks
>>>>>
>>>>> On Sat, Jul 15, 2017 at 7:57 AM, Thomas Weise <th...@apache.org> wrote:
>>>>>
>>>>>> Discussing what in the future might become stable needs to be a
>>> separate
>>>>>> thread, it will be a much bigger discussion.
>>>>>>
>>>>>> The topic here is to relocate the packages. With a few exceptions
>>>>>> relocation won't affect the semantic versioning. Semantic versioning
>>> is
>>>>>> essentially not effective for Malhar because almost everything is
>>>>> @Evolving
>>>>>> (and there are reasons for that.. -> separate topic)
>>>>>>
>>>>>> I don't really like the idea of creating bw compatibility stubs for
>>> the
>>>>>> follow up PR. It creates even more clutter in the source tree than
>>> there
>>>>>> already is and so here is an alternative suggestion:
>>>>>>
>>>>>> https://github.com/tweise/apex-malhar/blob/malhar37-
>>>>>> compat/shaded-malhar37/pom.xml
>>>>>>
>>>>>> Create a shaded artifact that provides the old com.datatorrent.*
>>>>> classes as
>>>>>> of release 3.7. Users can include that artifact if they don't want
>> to
>>>>>> change import statements. At the same time they have an incentive to
>>>>> switch
>>>>>> to the relocated classes to take advantage of bug fixes and new
>>>>>> functionality.
>>>>>>
>>>>>> I will work on the first PR that does the relocate. In the meantime,
>>> we
>>>>> can
>>>>>> finalize what backward compatibility support we want to provide and
>>> how.
>>>>>> Thanks,
>>>>>> Thomas
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Jul 14, 2017 at 11:33 AM, Pramod Immaneni <
>>>>> pramod@datatorrent.com>
>>>>>> wrote:
>>>>>>
>>>>>>> How about coming up with a list of @Evolving operators that we
>> would
>>>>> like
>>>>>>> to see in the eventual stable list and move those along with the
>> not
>>>>>>> @Evolving ones in org.apache.apex with b/w stubs and leave the
>> rest
>>> as
>>>>>> they
>>>>>>> are. Then have a follow up JIRA for the rest to be moved over to
>>>>> contrib
>>>>>>> and be deprecated.
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>> On Fri, Jul 14, 2017 at 10:37 AM, Thomas Weise <
>>>>> thomas.weise@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> We need to keep the discussion here on topic, if other things
>> are
>>>>> piled
>>>>>>> on
>>>>>>>> top then nothing gets done.
>>>>>>>>
>>>>>>>> Most operators today are not stable, they are @Evolving. So
>>> perhaps
>>>>> it
>>>>>> is
>>>>>>>> necessary to have a separate discussion about that, outside of
>>> this
>>>>>>> thread.
>>>>>>>> My guess is that there are only a few operators that we could
>>>>> declare
>>>>>>>> stable. Specifically, under contrib the closest one might have
>>> been
>>>>>>> Kafka,
>>>>>>>> but that is already superseded by the newer versions.
>>>>>>>>
>>>>>>>> Thomas
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Jul 14, 2017 at 10:21 AM, Pramod Immaneni <
>>>>>>> pramod@datatorrent.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> We had a discussion a while back, agreed to relegate
>> non-stable
>>>>> and
>>>>>>>>> experimental operators to contrib and also added this to
>>>>> contribution
>>>>>>>>> guidelines. We aexecuted on this and cleaned up the repo by
>>> moving
>>>>>>>>> operators deemed non-suitable for production use at that time
>> to
>>>>>>> contrib.
>>>>>>>>> So I wouldn't say the operators that are at the top level
>> today
>>> or
>>>>>> the
>>>>>>>> ones
>>>>>>>>> in library are 0.x.x quality. Granted, we may need to do one
>>> more
>>>>>> pass
>>>>>>> as
>>>>>>>>> some of the operators may no longer be considered the right
>>>>>>>> implementations
>>>>>>>>> with the advent of the windowed operator.
>>>>>>>>>
>>>>>>>>> Since contrib used to be the place that housed operators that
>>>>>> required
>>>>>>>>> third party libraries, there are some production quality
>>>>> operators in
>>>>>>>> there
>>>>>>>>> that need to be pulled out into top level modules.
>>>>>>>>>
>>>>>>>>> I think we are in agreement that for operators that we
>> consider
>>>>>> stable,
>>>>>>>> we
>>>>>>>>> should provide a b/w stub. I would add that we strongly
>> consider
>>>>>>> creating
>>>>>>>>> the org.apache.apex counterparts of any stable operators that
>>> are
>>>>> in
>>>>>>>>> contrib out in top level modules and have the com.datatorrent
>>>>> stubs
>>>>>> in
>>>>>>>>> contrib.
>>>>>>>>>
>>>>>>>>> For the operators not considered stable, I would prefer we
>>> either
>>>>>> leave
>>>>>>>> the
>>>>>>>>> current package path as is or if we change the package path
>> then
>>>>>> create
>>>>>>>> the
>>>>>>>>> b/w stub as I am not sure how widely they are in use (so, in
>>>>> essence,
>>>>>>>>> preserve semantic versioning). It would be good if there is a
>>>>>> followup
>>>>>>>> JIRA
>>>>>>>>> that takes a look at what other operators can be moved to
>>> contrib
>>>>>> with
>>>>>>>> the
>>>>>>>>> advent of the newer frameworks and understanding.
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>>
>>>>>>>>> On Fri, Jul 14, 2017 at 9:44 AM, Thomas Weise <thw@apache.org
>>>>>> wrote:
>>>>>>>>>> Most of the operators evolve, as is quite clearly indicated
>> by
>>>>>>>> @Evolving
>>>>>>>>>> annotations. So while perhaps 0.x.x would be a more
>>> appropriate
>>>>>>> version
>>>>>>>>>> number, I don't think you can change that.
>>>>>>>>>>
>>>>>>>>>> Thomas
>>>>>>>>>>
>>>>>>>>>> On Fri, Jul 14, 2017 at 9:35 AM, Vlad Rozov <
>>>>>> v.rozov@datatorrent.com
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> If entire library is not stable, its version should be
>> 0.x.x
>>>>> and
>>>>>>>> there
>>>>>>>>>>> should not be any semantic versioning enabled or implied.
>> It
>>>>>>> evolves.
>>>>>>>>> If
>>>>>>>>>>> some operators are stable as 3.8.x version suggests, the
>>>>> library
>>>>>>>> should
>>>>>>>>>>> follow semantic versioning and it is not OK to make a
>> major
>>>>>> change
>>>>>>>> that
>>>>>>>>>>> breaks semantic versioning across the entire library. It
>> is
>>>>> not a
>>>>>>>>> finding
>>>>>>>>>>> an excuse not to make a change. For me semantic versioning
>>> and
>>>>>>>> backward
>>>>>>>>>>> compatibility is more important than a proper package
>> name.
>>>>>>>>>>> Thank you,
>>>>>>>>>>>
>>>>>>>>>>> Vlad
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 7/14/17 09:11, Thomas Weise wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Semantic versioning makes sense when you have a stable
>>>>> baseline.
>>>>>>> As
>>>>>>>>> long
>>>>>>>>>>>> as
>>>>>>>>>>>> frequent fixes need to be made to address basic issues,
>> it
>>>>> makes
>>>>>>> no
>>>>>>>>>> sense
>>>>>>>>>>>> to declare operators stable, which is why they are marked
>>>>>>> evolving.
>>>>>>>>>>>> Instead of finding reasons to avoid changes that should
>>> have
>>>>>> been
>>>>>>>>> made a
>>>>>>>>>>>> long time ago, why not discuss what a "stable operator"
>> is
>>>>> and
>>>>>>> which
>>>>>>>>>> ones
>>>>>>>>>>>> deserve to be in that category. Those are the ones that
>>> will
>>>>> get
>>>>>>> the
>>>>>>>>>>>> backward compatibility stub.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Thomas
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Jul 14, 2017 at 8:34 AM, Vlad Rozov <
>>>>>>>> v.rozov@datatorrent.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> My preference is to agree that the next Malhar release is
>>>>> 4.0.0
>>>>>>> and
>>>>>>>>> make
>>>>>>>>>>>>> the proposed changes when 3.8.0 goes out. Otherwise why
>> to
>>>>> keep
>>>>>>>>>> semantic
>>>>>>>>>>>>> versioning checks on Malhar in case there is no version
>>>>>>>>> compatibility.
>>>>>>>>>>>>> Thank you,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Vlad
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 7/14/17 08:11, Thomas Weise wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> next release is 3.8.0
>>>>>>>>>>>>>> On Fri, Jul 14, 2017 at 7:55 AM, Vlad Rozov <
>>>>>>>>> v.rozov@datatorrent.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> next release - 3.9.0 or 4.0.0?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thank you,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Vlad
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 7/13/17 22:25, Thomas Weise wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> It is time to resurrect this thread and get going with
>>> the
>>>>>>> work.
>>>>>>>>>>>>>>>> For the next release, I will sign up to do the
>> package
>>>>> move
>>>>>> in
>>>>>>>>>> Malhar:
>>>>>>>>>>>>>>>> https://issues.apache.org/
>> jira/browse/APEXMALHAR-2517
>>>>>>>>>>>>>>>> In general this will be straightforward; most classes
>>> in
>>>>>>> Malhar
>>>>>>>>> are
>>>>>>>>>>>>>>>> marked
>>>>>>>>>>>>>>>> evolving and it is trivial for users to change import
>>>>>>>> statements.
>>>>>>>>>>>>>>>> However,
>>>>>>>>>>>>>>>> I would suggest to discuss if there are selected
>>>>> operators
>>>>>>> that
>>>>>>>>> are
>>>>>>>>>>>>>>>> worth
>>>>>>>>>>>>>>>> keeping a backward compatibility stub in the old
>>>>> location.
>>>>>>>>>>>>>>>> Here is my plan:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 1. Relocate all classes in *lib* and *contrib* within
>>> one
>>>>>> PR -
>>>>>>>>> this
>>>>>>>>>> is
>>>>>>>>>>>>>>>> all
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> IDE automated work
>>>>>>>>>>>>>>>> 2. Add backward compatibility classes, if, any in
>>>>> separate
>>>>>> PR
>>>>>>>>>>>>>>>> 3. Create PR for Megh library to reflect moved
>> classes
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>> Thomas
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Wed, Mar 1, 2017 at 2:24 PM, Pramod Immaneni <
>>>>>>>>>>>>>>>> pramod@datatorrent.com
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Inline
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Mon, Feb 27, 2017 at 11:03 PM, Thomas Weise <
>>>>>>> thw@apache.org>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> -->
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Mon, Feb 27, 2017 at 1:27 PM, Pramod Immaneni <
>>>>>>>>>>>>>>>>>> pramod@datatorrent.com
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> For malhar, for existing operators, I prefer we do
>>>>> this as
>>>>>>>> part
>>>>>>>>> of
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> planned refactoring for breaking the monolith
>> modules
>>>>> into
>>>>>>>> baby
>>>>>>>>>>>>>>>>>>> packages
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> and would also prefer deprecating the existing
>>>>> operators
>>>>>> in
>>>>>>>>> place.
>>>>>>>>>>>>>>>>>> Refactor into smaller modules was discussed for
>>>>>>> malhar-contrib
>>>>>>>>> and
>>>>>>>>>>>>>>>>>> given
>>>>>>>>>>>>>>>>>> the overall state of that module I think it is OK
>> to
>>>>> defer
>>>>>>>>> package
>>>>>>>>>>>>>>>>>> renaming
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> there. I do however prefer to see the package rename
>>>>>>> addressed
>>>>>>>>> for
>>>>>>>>>>>>>>>>>> other
>>>>>>>>>>>>>>>>>> modules, especially for the main library module.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Should we consider breaking the library into
>> smaller
>>>>>> modules
>>>>>>>> as
>>>>>>>>>>>>>>>>>> well,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> file/block operators for example probably can be in
>>>>> their
>>>>>> own
>>>>>>>>>> module
>>>>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>>> just an organizational perspective.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> This
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> will help us achieve two things. First, the user
>> will
>>>>> see
>>>>>>> all
>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> new
>>>>>>>>>>>>>>>>>>> changes at once as opposed to dealing with it
>> twice
>>>>> (with
>>>>>>>>> package
>>>>>>>>>>>>>>>>>>> rename
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> and dependency changes) and second it will allow
>> for
>>> a
>>>>>>>> smoother
>>>>>>>>>>>>>>>>>> transition
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> as the existing code will still work in a
>> deprecated
>>>>>> state.
>>>>>>> It
>>>>>>>>>> will
>>>>>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> give a more consistent structure to malhar. For new
>>>>>>> operators,
>>>>>>>>> we
>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>> go
>>>>>>>>>>>>>>>>>> with the new package path but we need to ensure
>> they
>>>>> will
>>>>>>> get
>>>>>>>>>> moved
>>>>>>>>>>>>>>>>>> into
>>>>>>>>>>>>>>>>>> the baby packages as well.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I think existing operators should be renamed so
>> that
>>>>> git
>>>>>>>> history
>>>>>>>>>>>>>>>>>>> remains. A
>>>>>>>>>>>>>>>>> possible solution for backward compatibility could
>> be
>>> to
>>>>>>>>>> subsequently
>>>>>>>>>>>>>>>>>> add
>>>>>>>>>>>>>>>>>> empty subclasses in the previous location (for
>>> existing
>>>>>>>> concrete
>>>>>>>>>>>>>>>>>> operators
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> that we know are actually in use) to simplify
>>> migration
>>>>> for
>>>>>>>>> users.
>>>>>>>>>>>>>>>>>> Yes we can do that.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> For demos, we can modify the paths as the apps are
>>>>>> typically
>>>>>>>>> used
>>>>>>>>>>>>>>>>> wholesale
>>>>>>>>>>>>>>>>>> and the interface is typically manual interaction.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> For core, if we are adding new api subsystems,
>> like
>>>>> the
>>>>>>>>> launcher
>>>>>>>>>>>>>>>>>>> api
>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>> added recently for example, we can go with new
>>> package
>>>>>> path
>>>>>>>> but
>>>>>>>>>> if
>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> making incremental additions to existing
>>>>> functionality, I
>>>>>>> feel
>>>>>>>>> it
>>>>>>>>>> is
>>>>>>>>>>>>>>>>>> better
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> to keep it in the same package. I also prefer we
>> keep
>>>>> the
>>>>>>>>> package
>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> implementation classes consistent with api, for
>>>>>>>>> understandability
>>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> readability of the code. So, for example, we don't
>>>>> change
>>>>>>>>> package
>>>>>>>>>>>>>>>>>>> path
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> LogicalPlan as it is an implementation of DAG. It
>> is
>>>>>>>> subjective,
>>>>>>>>>> but
>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> will be good if we can also do the same with
>> classes
>>>>>> closely
>>>>>>>>>> related
>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> implementation classes as well. Maybe we can moving
>>>>> these
>>>>>>> on a
>>>>>>>>>>>>>>>>>> package
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> package basis, like everything in
>>>>>>> com.datatorrent.stram.engine
>>>>>>>>>> could
>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> moved. For completely internal components like
>> buffer
>>>>>>> server,
>>>>>>>> we
>>>>>>>>>> can
>>>>>>>>>>>>>>>>>>> move
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> them wholesale. We can consider moving all api and
>>>>>> classes,
>>>>>>>> when
>>>>>>>>>> we
>>>>>>>>>>>>>>>>>> go
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> next major release but would like to see if we can
>>>>> find a
>>>>>>> way
>>>>>>>> to
>>>>>>>>>>>>>>>>>> support
>>>>>>>>>>>>>>>>>> existing api for one more major release in
>> deprecated
>>>>>> mode.
>>>>>>>>>>>>>>>>>> The point of the major release is to enable
>> backward
>>>>>>>>> incompatible
>>>>>>>>>>>>>>>>>>> changes
>>>>>>>>>>>>>>>>>> and I don't think it is realistic to support the
>>>>> existing
>>>>>>> API
>>>>>>>>> for
>>>>>>>>>>>>>>>>>> another
>>>>>>>>>>>>>>>>>> major release. IMO it is also not necessary as most
>>>>>> existing
>>>>>>>>>>>>>>>>>> application
>>>>>>>>>>>>>>>>>> code refers to operators, attributes and the
>>>>> application
>>>>>>>>>> interface.
>>>>>>>>>>>>>>>>>> Perhaps
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> it is possible to keep those around as interface
>>>>> extensions
>>>>>>> to
>>>>>>>>> help
>>>>>>>>>>>>>>>>>> migration. Custom operators may need to be migrated
>>> to
>>>>>>> reflect
>>>>>>>>> API
>>>>>>>>>>>>>>>>>> changes,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> and I would consider that a reasonable task for
>>> operator
>>>>>>>>> developers
>>>>>>>>>>>>>>>>> as
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> part
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> of a major upgrade.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> It would be good if we can keep them as deprecated
>>>>>> interface
>>>>>>>>>>>>>>>>>> extensions
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>> one release to provide a smoother transition.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> API and implementation in engine are kept separate
>>>>>>>> intentionally.
>>>>>>>>>>>>>>>>> They
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> reside in different packages today, so I don't see a
>>>>>> problem
>>>>>>>>>> renaming
>>>>>>>>>>>>>>>>>> com.datatorrent.stram.engine as you say, even when
>>> the
>>>>> API
>>>>>>>>> cannot
>>>>>>>>>> be
>>>>>>>>>>>>>>>>>> touched right away.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> They are different packages but sharing a common
>>> prefix
>>>>>> with
>>>>>>>> api
>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> helpful to someone new to codebase in terms of
>>>>>> readability.
>>>>>>>> Not
>>>>>>>>> a
>>>>>>>>>>>>>>>>> big
>>>>>>>>>>>>>>>>> deal
>>>>>>>>>>>>>>>>> and can be changed.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Mon, Feb 27, 2017 at 7:39 AM, Thomas Weise <
>>>>>>> thw@apache.org>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> This topic has come up on several PRs and I think
>> it
>>>>>>>> warrants a
>>>>>>>>>>>>>>>>>>>> broader
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> discussion.
>>>>>>>>>>>>>>>>>> At the time of incubation, the decision was to
>> defer
>>>>>> change
>>>>>>> of
>>>>>>>>>> Java
>>>>>>>>>>>>>>>>>>>> packages from com.datatorrent to org.apache.apex
>>> till
>>>>>> next
>>>>>>>>> major
>>>>>>>>>>>>>>>>>>>> release
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> ensure backward compatibility for users.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Unfortunately that has lead to some confusion, as
>>>>>>>> contributors
>>>>>>>>>>>>>>>>>>>> continue
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> add new code under legacy packages.
>>>>>>>>>>>>>>>>>>>> It is also a wider issue that examples for using
>>> Apex
>>>>>>>> continue
>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> refer
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> com.datatorrent packages, nearly one year after
>>>>>> graduation.
>>>>>>>> More
>>>>>>>>>> and
>>>>>>>>>>>>>>>>>>>> more
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> user code is being built on top of something that
>>>>> needs
>>>>>> to
>>>>>>>>>> change,
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>>>>> is being kicked down the road and users will face
>>> more
>>>>>>> changes
>>>>>>>>>>>>>>>>>>> later.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I would like to propose the following:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> 1. All new code has to be submitted under
>>>>>> org.apache.apex
>>>>>>>>>> packages
>>>>>>>>>>>>>>>>>>>> 2. Not all code is under backward compatibility
>>>>>>> restriction
>>>>>>>>> and
>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>> those
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> cases we can rename the packages right away.
>>> Examples:
>>>>>>> buffer
>>>>>>>>>>>>>>>>>>> server,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> engine, demos/examples, benchmarks
>>>>>>>>>>>>>>>>>>>> 3. Discuss when the core API and operators can be
>>>>>> changed.
>>>>>>>> For
>>>>>>>>>>>>>>>>>>>> operators
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> have a bit more freedom to do changes before a
>> major
>>>>>>> release
>>>>>>>> as
>>>>>>>>>>>>>>>>>>> most
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> them are marked @Evolving and users have the
>> ability
>>>>> to
>>>>>>>>> continue
>>>>>>>>>>>>>>>>>> using
>>>>>>>>>>>>>>>>>>> prior version of Malhar with newer engine due to
>>>>> engine
>>>>>>>>> backward
>>>>>>>>>>>>>>>>>> compatibility guarantee.
>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>> Thomas
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>
Thank you,
Vlad
Re: Java packages: legacy -> org.apache.apex
Posted by Thomas Weise <th...@apache.org>.
+ 1 for release-3 branch instead of release-3.8
I would suggest we vote on two proposals (4.x as well as the option to
change names and start at 1.0). Any other suggestions? I'm planning to
start a VOTE thread by EOD.
Thanks,
Thomas
On Sat, Aug 19, 2017 at 1:56 PM, Pramod Immaneni <pr...@datatorrent.com>
wrote:
> I think if there is a will to allow older apps to continue to benefit from
> improvements to operators, without changes, it can be done along with the
> proposed move, especially since we are not taking about all operators.
>
> However, since that is not the case, I am fine with starting a 4.0 release
> line as long as we don't seal the future of 3.x now. What I would prefer is
> to create a release-3 branch and not a release-3.8 branch now and let the
> future determine if 3.8 will be the last 3.x minor release.
>
> Thanks
>
> On Fri, Aug 18, 2017 at 5:04 PM Thomas Weise <th...@apache.org> wrote:
>
> > The proposed change does NOT require changes to existing apps. It follows
> > what you see for example in the JDK where older classes are retired
> without
> > making breaking major release changes.
> >
> > Even though not required technically, it looks like the preference is to
> > make this change with a 4.0 release. I will therefore propose to change
> the
> > master branch to 4.0-SNAPSHOT and make this and other changes deemed
> > worthwhile.
> >
> > To me it is more important that to newcomers the codebase does not look
> > like an outdated legacy situation and therefore if I have to make a
> choice
> > than pretty much any version number will do for that. So far I have seen
> > very little interest and initiative in addressing maintenance work like
> the
> > packages, checkstyle, CI stability, documentation etc. So it would be
> great
> > if people interested would step up and contribute.
> >
> > Here is the next proposal how to proceed:
> >
> > - create release-3.8 branch for last 3.8 release right away
> > - update master to 4.0-SNAPSHOT as part of the open PR #664
> > - identify further cleanup work for master
> > - release 4.0
> >
> > Thanks,
> > Thomas
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Thu, Aug 17, 2017 at 4:48 PM, Amol Kekre <am...@datatorrent.com>
> wrote:
> >
> > > This following pull request should be taken up in 4.0.0. See my
> comments
> > in
> > > https://github.com/apache/apex-malhar/pull/664
> > > <https://www.google.com/url?q=https%3A%2F%2Fgithub.com%
> > > 2Fapache%2Fapex-malhar%2Fpull%2F664&sa=D&ust=1503020269444000&usg=
> > > AFQjCNHDPZIg2e6I33Jb1XjB5Ir1FkdURQ>
> > > https://github.com/apache/apex-malhar/pull/662
> > > <https://www.google.com/url?q=https%3A%2F%2Fgithub.com%
> > > 2Fapache%2Fapex-malhar%2Fpull%2F662&sa=D&ust=1503020269444000&usg=
> > > AFQjCNF7Xde7X55M3qmc8z-D5y3ZwNg7Fg>
> > >
> > > This merge should not be done without a consensus. This will require
> code
> > > changes to existing apps.
> > >
> > > Thks,
> > > Amol
> > >
> > >
> > >
> > > E:amol@datatorrent.com | M: 510-449-2606 | Twitter: @*amolhkekre*
> > >
> > > www.datatorrent.com
> > >
> > >
> > > On Mon, Aug 14, 2017 at 7:48 PM, Thomas Weise <th...@apache.org> wrote:
> > >
> > > > Hi,
> > > >
> > > > I opened following PRs for the package change:
> > > >
> > > > https://github.com/apache/apex-malhar/pull/662
> > > >
> > > > Moves all classes with history retained (hence 2 commits). Also
> > contains
> > > > checkstyle and other mechanical changes.
> > > >
> > > > https://github.com/apache/apex-malhar/pull/664
> > > >
> > > > Adds backward compatibility jar.
> > > >
> > > > Once above PRs are merged the new artifact can be deployed and
> > introduced
> > > > as dependency in malhar-library.
> > > >
> > > > Please review.
> > > >
> > > > Thanks,
> > > > Thomas
> > > >
> > > >
> > > >
> > > > On Sun, Jul 16, 2017 at 7:04 AM, Thomas Weise <th...@apache.org>
> wrote:
> > > >
> > > > > My original list of work items contained the b/w compatibility
> > aspect,
> > > I
> > > > > don't think there should be any confusion of whether it will be
> > covered
> > > > > here or not.
> > > > >
> > > > > The proposed shading will provide the old classes and they will be
> > > frozen
> > > > > as of release 3.7. That's the same as making a copy of the code and
> > > never
> > > > > again making changes to the original classes. This cannot be
> > > accomplished
> > > > > by using the older 3.7 release in your project because you cannot
> > use 2
> > > > > different versions of Malhar in parallel unless you apply shading.
> > > > >
> > > > > The shaded artifact will only expose the com.datatorrent classes,
> and
> > > > they
> > > > > will be self-contained as the rest of the classes that they may
> > depend
> > > on
> > > > > are shaded. The shaded artifact does not evolve, there are not more
> > > > changes
> > > > > to com.datatorrent classes after they are relocated in master.
> > > > >
> > > > > Thanks,
> > > > > Thomas
> > > > >
> > > > >
> > > > > On Sun, Jul 16, 2017 at 2:00 AM, Pramod Immaneni <
> > > pramod@datatorrent.com
> > > > >
> > > > > wrote:
> > > > >
> > > > >> I don't think we can limit the topic strictly to relocation
> without
> > > > having
> > > > >> a good b/w compatibility story or at least one that goes far
> enough.
> > > > >>
> > > > >> The shading idea sounds interesting. Why not let the shaded
> version
> > > move
> > > > >> forward with each release till we hit a major release. If it is
> > going
> > > to
> > > > >> remain pegged at 3.7.0, why shade in the first place as the
> regular
> > > > 3.7.0
> > > > >> release would do the same job and it would be same as the loss of
> > b/w
> > > > >> compatibility with newer releases.
> > > > >>
> > > > >> Thanks
> > > > >>
> > > > >> On Sat, Jul 15, 2017 at 7:57 AM, Thomas Weise <th...@apache.org>
> > wrote:
> > > > >>
> > > > >> > Discussing what in the future might become stable needs to be a
> > > > separate
> > > > >> > thread, it will be a much bigger discussion.
> > > > >> >
> > > > >> > The topic here is to relocate the packages. With a few
> exceptions
> > > > >> > relocation won't affect the semantic versioning. Semantic
> > versioning
> > > > is
> > > > >> > essentially not effective for Malhar because almost everything
> is
> > > > >> @Evolving
> > > > >> > (and there are reasons for that.. -> separate topic)
> > > > >> >
> > > > >> > I don't really like the idea of creating bw compatibility stubs
> > for
> > > > the
> > > > >> > follow up PR. It creates even more clutter in the source tree
> than
> > > > there
> > > > >> > already is and so here is an alternative suggestion:
> > > > >> >
> > > > >> > https://github.com/tweise/apex-malhar/blob/malhar37-
> > > > >> > compat/shaded-malhar37/pom.xml
> > > > >> >
> > > > >> > Create a shaded artifact that provides the old com.datatorrent.*
> > > > >> classes as
> > > > >> > of release 3.7. Users can include that artifact if they don't
> want
> > > to
> > > > >> > change import statements. At the same time they have an
> incentive
> > to
> > > > >> switch
> > > > >> > to the relocated classes to take advantage of bug fixes and new
> > > > >> > functionality.
> > > > >> >
> > > > >> > I will work on the first PR that does the relocate. In the
> > meantime,
> > > > we
> > > > >> can
> > > > >> > finalize what backward compatibility support we want to provide
> > and
> > > > how.
> > > > >> >
> > > > >> > Thanks,
> > > > >> > Thomas
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > On Fri, Jul 14, 2017 at 11:33 AM, Pramod Immaneni <
> > > > >> pramod@datatorrent.com>
> > > > >> > wrote:
> > > > >> >
> > > > >> > > How about coming up with a list of @Evolving operators that we
> > > would
> > > > >> like
> > > > >> > > to see in the eventual stable list and move those along with
> the
> > > not
> > > > >> > > @Evolving ones in org.apache.apex with b/w stubs and leave the
> > > rest
> > > > as
> > > > >> > they
> > > > >> > > are. Then have a follow up JIRA for the rest to be moved over
> to
> > > > >> contrib
> > > > >> > > and be deprecated.
> > > > >> > >
> > > > >> > > Thanks
> > > > >> > >
> > > > >> > > On Fri, Jul 14, 2017 at 10:37 AM, Thomas Weise <
> > > > >> thomas.weise@gmail.com>
> > > > >> > > wrote:
> > > > >> > >
> > > > >> > > > We need to keep the discussion here on topic, if other
> things
> > > are
> > > > >> piled
> > > > >> > > on
> > > > >> > > > top then nothing gets done.
> > > > >> > > >
> > > > >> > > > Most operators today are not stable, they are @Evolving. So
> > > > perhaps
> > > > >> it
> > > > >> > is
> > > > >> > > > necessary to have a separate discussion about that, outside
> of
> > > > this
> > > > >> > > thread.
> > > > >> > > > My guess is that there are only a few operators that we
> could
> > > > >> declare
> > > > >> > > > stable. Specifically, under contrib the closest one might
> have
> > > > been
> > > > >> > > Kafka,
> > > > >> > > > but that is already superseded by the newer versions.
> > > > >> > > >
> > > > >> > > > Thomas
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > On Fri, Jul 14, 2017 at 10:21 AM, Pramod Immaneni <
> > > > >> > > pramod@datatorrent.com>
> > > > >> > > > wrote:
> > > > >> > > >
> > > > >> > > > > We had a discussion a while back, agreed to relegate
> > > non-stable
> > > > >> and
> > > > >> > > > > experimental operators to contrib and also added this to
> > > > >> contribution
> > > > >> > > > > guidelines. We aexecuted on this and cleaned up the repo
> by
> > > > moving
> > > > >> > > > > operators deemed non-suitable for production use at that
> > time
> > > to
> > > > >> > > contrib.
> > > > >> > > > > So I wouldn't say the operators that are at the top level
> > > today
> > > > or
> > > > >> > the
> > > > >> > > > ones
> > > > >> > > > > in library are 0.x.x quality. Granted, we may need to do
> one
> > > > more
> > > > >> > pass
> > > > >> > > as
> > > > >> > > > > some of the operators may no longer be considered the
> right
> > > > >> > > > implementations
> > > > >> > > > > with the advent of the windowed operator.
> > > > >> > > > >
> > > > >> > > > > Since contrib used to be the place that housed operators
> > that
> > > > >> > required
> > > > >> > > > > third party libraries, there are some production quality
> > > > >> operators in
> > > > >> > > > there
> > > > >> > > > > that need to be pulled out into top level modules.
> > > > >> > > > >
> > > > >> > > > > I think we are in agreement that for operators that we
> > > consider
> > > > >> > stable,
> > > > >> > > > we
> > > > >> > > > > should provide a b/w stub. I would add that we strongly
> > > consider
> > > > >> > > creating
> > > > >> > > > > the org.apache.apex counterparts of any stable operators
> > that
> > > > are
> > > > >> in
> > > > >> > > > > contrib out in top level modules and have the
> > com.datatorrent
> > > > >> stubs
> > > > >> > in
> > > > >> > > > > contrib.
> > > > >> > > > >
> > > > >> > > > > For the operators not considered stable, I would prefer we
> > > > either
> > > > >> > leave
> > > > >> > > > the
> > > > >> > > > > current package path as is or if we change the package
> path
> > > then
> > > > >> > create
> > > > >> > > > the
> > > > >> > > > > b/w stub as I am not sure how widely they are in use (so,
> in
> > > > >> essence,
> > > > >> > > > > preserve semantic versioning). It would be good if there
> is
> > a
> > > > >> > followup
> > > > >> > > > JIRA
> > > > >> > > > > that takes a look at what other operators can be moved to
> > > > contrib
> > > > >> > with
> > > > >> > > > the
> > > > >> > > > > advent of the newer frameworks and understanding.
> > > > >> > > > >
> > > > >> > > > > Thanks
> > > > >> > > > >
> > > > >> > > > > On Fri, Jul 14, 2017 at 9:44 AM, Thomas Weise <
> > thw@apache.org
> > > >
> > > > >> > wrote:
> > > > >> > > > >
> > > > >> > > > > > Most of the operators evolve, as is quite clearly
> > indicated
> > > by
> > > > >> > > > @Evolving
> > > > >> > > > > > annotations. So while perhaps 0.x.x would be a more
> > > > appropriate
> > > > >> > > version
> > > > >> > > > > > number, I don't think you can change that.
> > > > >> > > > > >
> > > > >> > > > > > Thomas
> > > > >> > > > > >
> > > > >> > > > > > On Fri, Jul 14, 2017 at 9:35 AM, Vlad Rozov <
> > > > >> > v.rozov@datatorrent.com
> > > > >> > > >
> > > > >> > > > > > wrote:
> > > > >> > > > > >
> > > > >> > > > > > > If entire library is not stable, its version should be
> > > 0.x.x
> > > > >> and
> > > > >> > > > there
> > > > >> > > > > > > should not be any semantic versioning enabled or
> > implied.
> > > It
> > > > >> > > evolves.
> > > > >> > > > > If
> > > > >> > > > > > > some operators are stable as 3.8.x version suggests,
> the
> > > > >> library
> > > > >> > > > should
> > > > >> > > > > > > follow semantic versioning and it is not OK to make a
> > > major
> > > > >> > change
> > > > >> > > > that
> > > > >> > > > > > > breaks semantic versioning across the entire library.
> It
> > > is
> > > > >> not a
> > > > >> > > > > finding
> > > > >> > > > > > > an excuse not to make a change. For me semantic
> > versioning
> > > > and
> > > > >> > > > backward
> > > > >> > > > > > > compatibility is more important than a proper package
> > > name.
> > > > >> > > > > > >
> > > > >> > > > > > > Thank you,
> > > > >> > > > > > >
> > > > >> > > > > > > Vlad
> > > > >> > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > > > On 7/14/17 09:11, Thomas Weise wrote:
> > > > >> > > > > > >
> > > > >> > > > > > >> Semantic versioning makes sense when you have a
> stable
> > > > >> baseline.
> > > > >> > > As
> > > > >> > > > > long
> > > > >> > > > > > >> as
> > > > >> > > > > > >> frequent fixes need to be made to address basic
> issues,
> > > it
> > > > >> makes
> > > > >> > > no
> > > > >> > > > > > sense
> > > > >> > > > > > >> to declare operators stable, which is why they are
> > marked
> > > > >> > > evolving.
> > > > >> > > > > > >>
> > > > >> > > > > > >> Instead of finding reasons to avoid changes that
> should
> > > > have
> > > > >> > been
> > > > >> > > > > made a
> > > > >> > > > > > >> long time ago, why not discuss what a "stable
> operator"
> > > is
> > > > >> and
> > > > >> > > which
> > > > >> > > > > > ones
> > > > >> > > > > > >> deserve to be in that category. Those are the ones
> that
> > > > will
> > > > >> get
> > > > >> > > the
> > > > >> > > > > > >> backward compatibility stub.
> > > > >> > > > > > >>
> > > > >> > > > > > >> Thanks,
> > > > >> > > > > > >> Thomas
> > > > >> > > > > > >>
> > > > >> > > > > > >>
> > > > >> > > > > > >> On Fri, Jul 14, 2017 at 8:34 AM, Vlad Rozov <
> > > > >> > > > v.rozov@datatorrent.com>
> > > > >> > > > > > >> wrote:
> > > > >> > > > > > >>
> > > > >> > > > > > >> My preference is to agree that the next Malhar
> release
> > is
> > > > >> 4.0.0
> > > > >> > > and
> > > > >> > > > > make
> > > > >> > > > > > >>> the proposed changes when 3.8.0 goes out. Otherwise
> > why
> > > to
> > > > >> keep
> > > > >> > > > > > semantic
> > > > >> > > > > > >>> versioning checks on Malhar in case there is no
> > version
> > > > >> > > > > compatibility.
> > > > >> > > > > > >>>
> > > > >> > > > > > >>> Thank you,
> > > > >> > > > > > >>>
> > > > >> > > > > > >>> Vlad
> > > > >> > > > > > >>>
> > > > >> > > > > > >>>
> > > > >> > > > > > >>> On 7/14/17 08:11, Thomas Weise wrote:
> > > > >> > > > > > >>>
> > > > >> > > > > > >>> next release is 3.8.0
> > > > >> > > > > > >>>>
> > > > >> > > > > > >>>> On Fri, Jul 14, 2017 at 7:55 AM, Vlad Rozov <
> > > > >> > > > > v.rozov@datatorrent.com>
> > > > >> > > > > > >>>> wrote:
> > > > >> > > > > > >>>>
> > > > >> > > > > > >>>> next release - 3.9.0 or 4.0.0?
> > > > >> > > > > > >>>>
> > > > >> > > > > > >>>>> Thank you,
> > > > >> > > > > > >>>>>
> > > > >> > > > > > >>>>> Vlad
> > > > >> > > > > > >>>>>
> > > > >> > > > > > >>>>> On 7/13/17 22:25, Thomas Weise wrote:
> > > > >> > > > > > >>>>>
> > > > >> > > > > > >>>>> It is time to resurrect this thread and get going
> > with
> > > > the
> > > > >> > > work.
> > > > >> > > > > > >>>>>
> > > > >> > > > > > >>>>>> For the next release, I will sign up to do the
> > > package
> > > > >> move
> > > > >> > in
> > > > >> > > > > > Malhar:
> > > > >> > > > > > >>>>>>
> > > > >> > > > > > >>>>>> https://issues.apache.org/
> > > jira/browse/APEXMALHAR-2517
> > > > >> > > > > > >>>>>>
> > > > >> > > > > > >>>>>> In general this will be straightforward; most
> > classes
> > > > in
> > > > >> > > Malhar
> > > > >> > > > > are
> > > > >> > > > > > >>>>>> marked
> > > > >> > > > > > >>>>>> evolving and it is trivial for users to change
> > import
> > > > >> > > > statements.
> > > > >> > > > > > >>>>>> However,
> > > > >> > > > > > >>>>>> I would suggest to discuss if there are selected
> > > > >> operators
> > > > >> > > that
> > > > >> > > > > are
> > > > >> > > > > > >>>>>> worth
> > > > >> > > > > > >>>>>> keeping a backward compatibility stub in the old
> > > > >> location.
> > > > >> > > > > > >>>>>>
> > > > >> > > > > > >>>>>> Here is my plan:
> > > > >> > > > > > >>>>>>
> > > > >> > > > > > >>>>>> 1. Relocate all classes in *lib* and *contrib*
> > within
> > > > one
> > > > >> > PR -
> > > > >> > > > > this
> > > > >> > > > > > is
> > > > >> > > > > > >>>>>> all
> > > > >> > > > > > >>>>>>
> > > > >> > > > > > >>>>>> IDE automated work
> > > > >> > > > > > >>>>>> 2. Add backward compatibility classes, if, any in
> > > > >> separate
> > > > >> > PR
> > > > >> > > > > > >>>>>> 3. Create PR for Megh library to reflect moved
> > > classes
> > > > >> > > > > > >>>>>>
> > > > >> > > > > > >>>>>> Thanks,
> > > > >> > > > > > >>>>>> Thomas
> > > > >> > > > > > >>>>>>
> > > > >> > > > > > >>>>>>
> > > > >> > > > > > >>>>>>
> > > > >> > > > > > >>>>>> On Wed, Mar 1, 2017 at 2:24 PM, Pramod Immaneni <
> > > > >> > > > > > >>>>>> pramod@datatorrent.com
> > > > >> > > > > > >>>>>> wrote:
> > > > >> > > > > > >>>>>>
> > > > >> > > > > > >>>>>> Inline
> > > > >> > > > > > >>>>>>
> > > > >> > > > > > >>>>>> On Mon, Feb 27, 2017 at 11:03 PM, Thomas Weise <
> > > > >> > > thw@apache.org>
> > > > >> > > > > > >>>>>>> wrote:
> > > > >> > > > > > >>>>>>>
> > > > >> > > > > > >>>>>>> -->
> > > > >> > > > > > >>>>>>>
> > > > >> > > > > > >>>>>>> On Mon, Feb 27, 2017 at 1:27 PM, Pramod
> Immaneni <
> > > > >> > > > > > >>>>>>>> pramod@datatorrent.com
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>> wrote:
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>> For malhar, for existing operators, I prefer we
> > do
> > > > >> this as
> > > > >> > > > part
> > > > >> > > > > of
> > > > >> > > > > > >>>>>>>> the
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>> planned refactoring for breaking the monolith
> > > modules
> > > > >> into
> > > > >> > > > baby
> > > > >> > > > > > >>>>>>>>>
> > > > >> > > > > > >>>>>>>>> packages
> > > > >> > > > > > >>>>>>>>>
> > > > >> > > > > > >>>>>>>> and would also prefer deprecating the existing
> > > > >> operators
> > > > >> > in
> > > > >> > > > > place.
> > > > >> > > > > > >>>>>>>> Refactor into smaller modules was discussed for
> > > > >> > > malhar-contrib
> > > > >> > > > > and
> > > > >> > > > > > >>>>>>>> given
> > > > >> > > > > > >>>>>>>> the overall state of that module I think it is
> OK
> > > to
> > > > >> defer
> > > > >> > > > > package
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>> renaming
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>> there. I do however prefer to see the package
> > rename
> > > > >> > > addressed
> > > > >> > > > > for
> > > > >> > > > > > >>>>>>>
> > > > >> > > > > > >>>>>>>> other
> > > > >> > > > > > >>>>>>>> modules, especially for the main library
> module.
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>> Should we consider breaking the library into
> > > smaller
> > > > >> > modules
> > > > >> > > > as
> > > > >> > > > > > >>>>>>>> well,
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>> the
> > > > >> > > > > > >>>>>>> file/block operators for example probably can be
> > in
> > > > >> their
> > > > >> > own
> > > > >> > > > > > module
> > > > >> > > > > > >>>>>>> from
> > > > >> > > > > > >>>>>>> just an organizational perspective.
> > > > >> > > > > > >>>>>>>
> > > > >> > > > > > >>>>>>>
> > > > >> > > > > > >>>>>>>
> > > > >> > > > > > >>>>>>> This
> > > > >> > > > > > >>>>>>>
> > > > >> > > > > > >>>>>>>> will help us achieve two things. First, the
> user
> > > will
> > > > >> see
> > > > >> > > all
> > > > >> > > > > the
> > > > >> > > > > > >>>>>>>>> new
> > > > >> > > > > > >>>>>>>>> changes at once as opposed to dealing with it
> > > twice
> > > > >> (with
> > > > >> > > > > package
> > > > >> > > > > > >>>>>>>>>
> > > > >> > > > > > >>>>>>>>> rename
> > > > >> > > > > > >>>>>>>>>
> > > > >> > > > > > >>>>>>>> and dependency changes) and second it will
> allow
> > > for
> > > > a
> > > > >> > > > smoother
> > > > >> > > > > > >>>>>>>> transition
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>> as the existing code will still work in a
> > > deprecated
> > > > >> > state.
> > > > >> > > It
> > > > >> > > > > > will
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>>> also
> > > > >> > > > > > >>>>>>>>>
> > > > >> > > > > > >>>>>>>> give a more consistent structure to malhar. For
> > new
> > > > >> > > operators,
> > > > >> > > > > we
> > > > >> > > > > > >>>>>>>> can
> > > > >> > > > > > >>>>>>>> go
> > > > >> > > > > > >>>>>>>> with the new package path but we need to ensure
> > > they
> > > > >> will
> > > > >> > > get
> > > > >> > > > > > moved
> > > > >> > > > > > >>>>>>>> into
> > > > >> > > > > > >>>>>>>> the baby packages as well.
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>> I think existing operators should be renamed so
> > > that
> > > > >> git
> > > > >> > > > history
> > > > >> > > > > > >>>>>>>>>
> > > > >> > > > > > >>>>>>>>> remains. A
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>> possible solution for backward compatibility
> could
> > > be
> > > > to
> > > > >> > > > > > subsequently
> > > > >> > > > > > >>>>>>>
> > > > >> > > > > > >>>>>>>> add
> > > > >> > > > > > >>>>>>>> empty subclasses in the previous location (for
> > > > existing
> > > > >> > > > concrete
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>> operators
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>> that we know are actually in use) to simplify
> > > > migration
> > > > >> for
> > > > >> > > > > users.
> > > > >> > > > > > >>>>>>>
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>> Yes we can do that.
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>> For demos, we can modify the paths as the apps
> > are
> > > > >> > typically
> > > > >> > > > > used
> > > > >> > > > > > >>>>>>>
> > > > >> > > > > > >>>>>>> wholesale
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>> and the interface is typically manual
> > interaction.
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>>> For core, if we are adding new api subsystems,
> > > like
> > > > >> the
> > > > >> > > > > launcher
> > > > >> > > > > > >>>>>>>>> api
> > > > >> > > > > > >>>>>>>>> we
> > > > >> > > > > > >>>>>>>>> added recently for example, we can go with new
> > > > package
> > > > >> > path
> > > > >> > > > but
> > > > >> > > > > > if
> > > > >> > > > > > >>>>>>>>> we
> > > > >> > > > > > >>>>>>>>>
> > > > >> > > > > > >>>>>>>>> are
> > > > >> > > > > > >>>>>>>>>
> > > > >> > > > > > >>>>>>>> making incremental additions to existing
> > > > >> functionality, I
> > > > >> > > feel
> > > > >> > > > > it
> > > > >> > > > > > is
> > > > >> > > > > > >>>>>>>> better
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>> to keep it in the same package. I also prefer
> we
> > > keep
> > > > >> the
> > > > >> > > > > package
> > > > >> > > > > > of
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>>> the
> > > > >> > > > > > >>>>>>>>>
> > > > >> > > > > > >>>>>>>> implementation classes consistent with api, for
> > > > >> > > > > understandability
> > > > >> > > > > > >>>>>>>> and
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>> readability of the code. So, for example, we
> > don't
> > > > >> change
> > > > >> > > > > package
> > > > >> > > > > > >>>>>>>>> path
> > > > >> > > > > > >>>>>>>>>
> > > > >> > > > > > >>>>>>>>> of
> > > > >> > > > > > >>>>>>>>>
> > > > >> > > > > > >>>>>>>> LogicalPlan as it is an implementation of DAG.
> It
> > > is
> > > > >> > > > subjective,
> > > > >> > > > > > but
> > > > >> > > > > > >>>>>>>> it
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>> will be good if we can also do the same with
> > > classes
> > > > >> > closely
> > > > >> > > > > > related
> > > > >> > > > > > >>>>>>>>> to
> > > > >> > > > > > >>>>>>>>>
> > > > >> > > > > > >>>>>>>>> the
> > > > >> > > > > > >>>>>>>>>
> > > > >> > > > > > >>>>>>>> implementation classes as well. Maybe we can
> > moving
> > > > >> these
> > > > >> > > on a
> > > > >> > > > > > >>>>>>>> package
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>>> by
> > > > >> > > > > > >>>>>>>>>
> > > > >> > > > > > >>>>>>>> package basis, like everything in
> > > > >> > > com.datatorrent.stram.engine
> > > > >> > > > > > could
> > > > >> > > > > > >>>>>>>> be
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>> moved. For completely internal components like
> > > buffer
> > > > >> > > server,
> > > > >> > > > we
> > > > >> > > > > > can
> > > > >> > > > > > >>>>>>>>>
> > > > >> > > > > > >>>>>>>>> move
> > > > >> > > > > > >>>>>>>>>
> > > > >> > > > > > >>>>>>>> them wholesale. We can consider moving all api
> > and
> > > > >> > classes,
> > > > >> > > > when
> > > > >> > > > > > we
> > > > >> > > > > > >>>>>>>> go
> > > > >> > > > > > >>>>>>>> to
> > > > >> > > > > > >>>>>>>> next major release but would like to see if we
> > can
> > > > >> find a
> > > > >> > > way
> > > > >> > > > to
> > > > >> > > > > > >>>>>>>> support
> > > > >> > > > > > >>>>>>>> existing api for one more major release in
> > > deprecated
> > > > >> > mode.
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>> The point of the major release is to enable
> > > backward
> > > > >> > > > > incompatible
> > > > >> > > > > > >>>>>>>>>
> > > > >> > > > > > >>>>>>>>> changes
> > > > >> > > > > > >>>>>>>> and I don't think it is realistic to support
> the
> > > > >> existing
> > > > >> > > API
> > > > >> > > > > for
> > > > >> > > > > > >>>>>>>> another
> > > > >> > > > > > >>>>>>>> major release. IMO it is also not necessary as
> > most
> > > > >> > existing
> > > > >> > > > > > >>>>>>>> application
> > > > >> > > > > > >>>>>>>> code refers to operators, attributes and the
> > > > >> application
> > > > >> > > > > > interface.
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>> Perhaps
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>> it is possible to keep those around as interface
> > > > >> extensions
> > > > >> > > to
> > > > >> > > > > help
> > > > >> > > > > > >>>>>>>
> > > > >> > > > > > >>>>>>>> migration. Custom operators may need to be
> > migrated
> > > > to
> > > > >> > > reflect
> > > > >> > > > > API
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>> changes,
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>> and I would consider that a reasonable task for
> > > > operator
> > > > >> > > > > developers
> > > > >> > > > > > >>>>>>> as
> > > > >> > > > > > >>>>>>>
> > > > >> > > > > > >>>>>>>> part
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>> of a major upgrade.
> > > > >> > > > > > >>>>>>>
> > > > >> > > > > > >>>>>>>> It would be good if we can keep them as
> > deprecated
> > > > >> > interface
> > > > >> > > > > > >>>>>>>> extensions
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>> for
> > > > >> > > > > > >>>>>>> one release to provide a smoother transition.
> > > > >> > > > > > >>>>>>>
> > > > >> > > > > > >>>>>>>
> > > > >> > > > > > >>>>>>> API and implementation in engine are kept
> separate
> > > > >> > > > intentionally.
> > > > >> > > > > > >>>>>>> They
> > > > >> > > > > > >>>>>>>
> > > > >> > > > > > >>>>>>> reside in different packages today, so I don't
> > see a
> > > > >> > problem
> > > > >> > > > > > renaming
> > > > >> > > > > > >>>>>>>> com.datatorrent.stram.engine as you say, even
> > when
> > > > the
> > > > >> API
> > > > >> > > > > cannot
> > > > >> > > > > > be
> > > > >> > > > > > >>>>>>>> touched right away.
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>> They are different packages but sharing a
> common
> > > > prefix
> > > > >> > with
> > > > >> > > > api
> > > > >> > > > > > >>>>>>>> will
> > > > >> > > > > > >>>>>>>> be
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>> helpful to someone new to codebase in terms of
> > > > >> > readability.
> > > > >> > > > Not
> > > > >> > > > > a
> > > > >> > > > > > >>>>>>> big
> > > > >> > > > > > >>>>>>> deal
> > > > >> > > > > > >>>>>>> and can be changed.
> > > > >> > > > > > >>>>>>>
> > > > >> > > > > > >>>>>>>
> > > > >> > > > > > >>>>>>> Thanks
> > > > >> > > > > > >>>>>>>
> > > > >> > > > > > >>>>>>> On Mon, Feb 27, 2017 at 7:39 AM, Thomas Weise <
> > > > >> > > thw@apache.org>
> > > > >> > > > > > >>>>>>>> wrote:
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>>> Hi,
> > > > >> > > > > > >>>>>>>>>
> > > > >> > > > > > >>>>>>>>> This topic has come up on several PRs and I
> > think
> > > it
> > > > >> > > > warrants a
> > > > >> > > > > > >>>>>>>>>>
> > > > >> > > > > > >>>>>>>>>> broader
> > > > >> > > > > > >>>>>>>>>>
> > > > >> > > > > > >>>>>>>>> discussion.
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>> At the time of incubation, the decision was to
> > > defer
> > > > >> > change
> > > > >> > > of
> > > > >> > > > > > Java
> > > > >> > > > > > >>>>>>>>>
> > > > >> > > > > > >>>>>>>>>> packages from com.datatorrent to
> > org.apache.apex
> > > > till
> > > > >> > next
> > > > >> > > > > major
> > > > >> > > > > > >>>>>>>>>>
> > > > >> > > > > > >>>>>>>>>> release
> > > > >> > > > > > >>>>>>>>>>
> > > > >> > > > > > >>>>>>>>> to
> > > > >> > > > > > >>>>>>>>>
> > > > >> > > > > > >>>>>>>>> ensure backward compatibility for users.
> > > > >> > > > > > >>>>>>>>>
> > > > >> > > > > > >>>>>>>>>> Unfortunately that has lead to some
> confusion,
> > as
> > > > >> > > > contributors
> > > > >> > > > > > >>>>>>>>>>
> > > > >> > > > > > >>>>>>>>>> continue
> > > > >> > > > > > >>>>>>>>>>
> > > > >> > > > > > >>>>>>>>> to
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>> add new code under legacy packages.
> > > > >> > > > > > >>>>>>>>>
> > > > >> > > > > > >>>>>>>>>> It is also a wider issue that examples for
> > using
> > > > Apex
> > > > >> > > > continue
> > > > >> > > > > > to
> > > > >> > > > > > >>>>>>>>>>
> > > > >> > > > > > >>>>>>>>>> refer
> > > > >> > > > > > >>>>>>>>>>
> > > > >> > > > > > >>>>>>>>> to
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>> com.datatorrent packages, nearly one year after
> > > > >> > graduation.
> > > > >> > > > More
> > > > >> > > > > > and
> > > > >> > > > > > >>>>>>>>>
> > > > >> > > > > > >>>>>>>>>> more
> > > > >> > > > > > >>>>>>>>>>
> > > > >> > > > > > >>>>>>>>> user code is being built on top of something
> > that
> > > > >> needs
> > > > >> > to
> > > > >> > > > > > change,
> > > > >> > > > > > >>>>>>>>> the
> > > > >> > > > > > >>>>>>>>>
> > > > >> > > > > > >>>>>>>>> can
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>> is being kicked down the road and users will
> face
> > > > more
> > > > >> > > changes
> > > > >> > > > > > >>>>>>>>> later.
> > > > >> > > > > > >>>>>>>>>
> > > > >> > > > > > >>>>>>>>>> I would like to propose the following:
> > > > >> > > > > > >>>>>>>>>>
> > > > >> > > > > > >>>>>>>>>> 1. All new code has to be submitted under
> > > > >> > org.apache.apex
> > > > >> > > > > > packages
> > > > >> > > > > > >>>>>>>>>>
> > > > >> > > > > > >>>>>>>>>> 2. Not all code is under backward
> compatibility
> > > > >> > > restriction
> > > > >> > > > > and
> > > > >> > > > > > in
> > > > >> > > > > > >>>>>>>>>>
> > > > >> > > > > > >>>>>>>>>> those
> > > > >> > > > > > >>>>>>>>>>
> > > > >> > > > > > >>>>>>>>> cases we can rename the packages right away.
> > > > Examples:
> > > > >> > > buffer
> > > > >> > > > > > >>>>>>>>> server,
> > > > >> > > > > > >>>>>>>>>
> > > > >> > > > > > >>>>>>>>> engine, demos/examples, benchmarks
> > > > >> > > > > > >>>>>>>>>>
> > > > >> > > > > > >>>>>>>>>> 3. Discuss when the core API and operators
> can
> > be
> > > > >> > changed.
> > > > >> > > > For
> > > > >> > > > > > >>>>>>>>>>
> > > > >> > > > > > >>>>>>>>>> operators
> > > > >> > > > > > >>>>>>>>>>
> > > > >> > > > > > >>>>>>>>> we
> > > > >> > > > > > >>>>>>>>>
> > > > >> > > > > > >>>>>>>>> have a bit more freedom to do changes before a
> > > major
> > > > >> > > release
> > > > >> > > > as
> > > > >> > > > > > >>>>>>>>> most
> > > > >> > > > > > >>>>>>>>>
> > > > >> > > > > > >>>>>>>>>> of
> > > > >> > > > > > >>>>>>>>>>
> > > > >> > > > > > >>>>>>>>> them are marked @Evolving and users have the
> > > ability
> > > > >> to
> > > > >> > > > > continue
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>> using
> > > > >> > > > > > >>>>>>>>>
> > > > >> > > > > > >>>>>>>>> prior version of Malhar with newer engine due
> to
> > > > >> engine
> > > > >> > > > > backward
> > > > >> > > > > > >>>>>>>>
> > > > >> > > > > > >>>>>>>> compatibility guarantee.
> > > > >> > > > > > >>>>>>>>>
> > > > >> > > > > > >>>>>>>>>> Thanks,
> > > > >> > > > > > >>>>>>>>>> Thomas
> > > > >> > > > > > >>>>>>>>>>
> > > > >> > > > > > >>>>>>>>>>
> > > > >> > > > > > >>>>>>>>>>
> > > > >> > > > > > >>>>>>>>>>
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: Java packages: legacy -> org.apache.apex
Posted by Pramod Immaneni <pr...@datatorrent.com>.
I think if there is a will to allow older apps to continue to benefit from
improvements to operators, without changes, it can be done along with the
proposed move, especially since we are not taking about all operators.
However, since that is not the case, I am fine with starting a 4.0 release
line as long as we don't seal the future of 3.x now. What I would prefer is
to create a release-3 branch and not a release-3.8 branch now and let the
future determine if 3.8 will be the last 3.x minor release.
Thanks
On Fri, Aug 18, 2017 at 5:04 PM Thomas Weise <th...@apache.org> wrote:
> The proposed change does NOT require changes to existing apps. It follows
> what you see for example in the JDK where older classes are retired without
> making breaking major release changes.
>
> Even though not required technically, it looks like the preference is to
> make this change with a 4.0 release. I will therefore propose to change the
> master branch to 4.0-SNAPSHOT and make this and other changes deemed
> worthwhile.
>
> To me it is more important that to newcomers the codebase does not look
> like an outdated legacy situation and therefore if I have to make a choice
> than pretty much any version number will do for that. So far I have seen
> very little interest and initiative in addressing maintenance work like the
> packages, checkstyle, CI stability, documentation etc. So it would be great
> if people interested would step up and contribute.
>
> Here is the next proposal how to proceed:
>
> - create release-3.8 branch for last 3.8 release right away
> - update master to 4.0-SNAPSHOT as part of the open PR #664
> - identify further cleanup work for master
> - release 4.0
>
> Thanks,
> Thomas
>
>
>
>
>
>
>
>
>
> On Thu, Aug 17, 2017 at 4:48 PM, Amol Kekre <am...@datatorrent.com> wrote:
>
> > This following pull request should be taken up in 4.0.0. See my comments
> in
> > https://github.com/apache/apex-malhar/pull/664
> > <https://www.google.com/url?q=https%3A%2F%2Fgithub.com%
> > 2Fapache%2Fapex-malhar%2Fpull%2F664&sa=D&ust=1503020269444000&usg=
> > AFQjCNHDPZIg2e6I33Jb1XjB5Ir1FkdURQ>
> > https://github.com/apache/apex-malhar/pull/662
> > <https://www.google.com/url?q=https%3A%2F%2Fgithub.com%
> > 2Fapache%2Fapex-malhar%2Fpull%2F662&sa=D&ust=1503020269444000&usg=
> > AFQjCNF7Xde7X55M3qmc8z-D5y3ZwNg7Fg>
> >
> > This merge should not be done without a consensus. This will require code
> > changes to existing apps.
> >
> > Thks,
> > Amol
> >
> >
> >
> > E:amol@datatorrent.com | M: 510-449-2606 | Twitter: @*amolhkekre*
> >
> > www.datatorrent.com
> >
> >
> > On Mon, Aug 14, 2017 at 7:48 PM, Thomas Weise <th...@apache.org> wrote:
> >
> > > Hi,
> > >
> > > I opened following PRs for the package change:
> > >
> > > https://github.com/apache/apex-malhar/pull/662
> > >
> > > Moves all classes with history retained (hence 2 commits). Also
> contains
> > > checkstyle and other mechanical changes.
> > >
> > > https://github.com/apache/apex-malhar/pull/664
> > >
> > > Adds backward compatibility jar.
> > >
> > > Once above PRs are merged the new artifact can be deployed and
> introduced
> > > as dependency in malhar-library.
> > >
> > > Please review.
> > >
> > > Thanks,
> > > Thomas
> > >
> > >
> > >
> > > On Sun, Jul 16, 2017 at 7:04 AM, Thomas Weise <th...@apache.org> wrote:
> > >
> > > > My original list of work items contained the b/w compatibility
> aspect,
> > I
> > > > don't think there should be any confusion of whether it will be
> covered
> > > > here or not.
> > > >
> > > > The proposed shading will provide the old classes and they will be
> > frozen
> > > > as of release 3.7. That's the same as making a copy of the code and
> > never
> > > > again making changes to the original classes. This cannot be
> > accomplished
> > > > by using the older 3.7 release in your project because you cannot
> use 2
> > > > different versions of Malhar in parallel unless you apply shading.
> > > >
> > > > The shaded artifact will only expose the com.datatorrent classes, and
> > > they
> > > > will be self-contained as the rest of the classes that they may
> depend
> > on
> > > > are shaded. The shaded artifact does not evolve, there are not more
> > > changes
> > > > to com.datatorrent classes after they are relocated in master.
> > > >
> > > > Thanks,
> > > > Thomas
> > > >
> > > >
> > > > On Sun, Jul 16, 2017 at 2:00 AM, Pramod Immaneni <
> > pramod@datatorrent.com
> > > >
> > > > wrote:
> > > >
> > > >> I don't think we can limit the topic strictly to relocation without
> > > having
> > > >> a good b/w compatibility story or at least one that goes far enough.
> > > >>
> > > >> The shading idea sounds interesting. Why not let the shaded version
> > move
> > > >> forward with each release till we hit a major release. If it is
> going
> > to
> > > >> remain pegged at 3.7.0, why shade in the first place as the regular
> > > 3.7.0
> > > >> release would do the same job and it would be same as the loss of
> b/w
> > > >> compatibility with newer releases.
> > > >>
> > > >> Thanks
> > > >>
> > > >> On Sat, Jul 15, 2017 at 7:57 AM, Thomas Weise <th...@apache.org>
> wrote:
> > > >>
> > > >> > Discussing what in the future might become stable needs to be a
> > > separate
> > > >> > thread, it will be a much bigger discussion.
> > > >> >
> > > >> > The topic here is to relocate the packages. With a few exceptions
> > > >> > relocation won't affect the semantic versioning. Semantic
> versioning
> > > is
> > > >> > essentially not effective for Malhar because almost everything is
> > > >> @Evolving
> > > >> > (and there are reasons for that.. -> separate topic)
> > > >> >
> > > >> > I don't really like the idea of creating bw compatibility stubs
> for
> > > the
> > > >> > follow up PR. It creates even more clutter in the source tree than
> > > there
> > > >> > already is and so here is an alternative suggestion:
> > > >> >
> > > >> > https://github.com/tweise/apex-malhar/blob/malhar37-
> > > >> > compat/shaded-malhar37/pom.xml
> > > >> >
> > > >> > Create a shaded artifact that provides the old com.datatorrent.*
> > > >> classes as
> > > >> > of release 3.7. Users can include that artifact if they don't want
> > to
> > > >> > change import statements. At the same time they have an incentive
> to
> > > >> switch
> > > >> > to the relocated classes to take advantage of bug fixes and new
> > > >> > functionality.
> > > >> >
> > > >> > I will work on the first PR that does the relocate. In the
> meantime,
> > > we
> > > >> can
> > > >> > finalize what backward compatibility support we want to provide
> and
> > > how.
> > > >> >
> > > >> > Thanks,
> > > >> > Thomas
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> > On Fri, Jul 14, 2017 at 11:33 AM, Pramod Immaneni <
> > > >> pramod@datatorrent.com>
> > > >> > wrote:
> > > >> >
> > > >> > > How about coming up with a list of @Evolving operators that we
> > would
> > > >> like
> > > >> > > to see in the eventual stable list and move those along with the
> > not
> > > >> > > @Evolving ones in org.apache.apex with b/w stubs and leave the
> > rest
> > > as
> > > >> > they
> > > >> > > are. Then have a follow up JIRA for the rest to be moved over to
> > > >> contrib
> > > >> > > and be deprecated.
> > > >> > >
> > > >> > > Thanks
> > > >> > >
> > > >> > > On Fri, Jul 14, 2017 at 10:37 AM, Thomas Weise <
> > > >> thomas.weise@gmail.com>
> > > >> > > wrote:
> > > >> > >
> > > >> > > > We need to keep the discussion here on topic, if other things
> > are
> > > >> piled
> > > >> > > on
> > > >> > > > top then nothing gets done.
> > > >> > > >
> > > >> > > > Most operators today are not stable, they are @Evolving. So
> > > perhaps
> > > >> it
> > > >> > is
> > > >> > > > necessary to have a separate discussion about that, outside of
> > > this
> > > >> > > thread.
> > > >> > > > My guess is that there are only a few operators that we could
> > > >> declare
> > > >> > > > stable. Specifically, under contrib the closest one might have
> > > been
> > > >> > > Kafka,
> > > >> > > > but that is already superseded by the newer versions.
> > > >> > > >
> > > >> > > > Thomas
> > > >> > > >
> > > >> > > >
> > > >> > > > On Fri, Jul 14, 2017 at 10:21 AM, Pramod Immaneni <
> > > >> > > pramod@datatorrent.com>
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > > We had a discussion a while back, agreed to relegate
> > non-stable
> > > >> and
> > > >> > > > > experimental operators to contrib and also added this to
> > > >> contribution
> > > >> > > > > guidelines. We aexecuted on this and cleaned up the repo by
> > > moving
> > > >> > > > > operators deemed non-suitable for production use at that
> time
> > to
> > > >> > > contrib.
> > > >> > > > > So I wouldn't say the operators that are at the top level
> > today
> > > or
> > > >> > the
> > > >> > > > ones
> > > >> > > > > in library are 0.x.x quality. Granted, we may need to do one
> > > more
> > > >> > pass
> > > >> > > as
> > > >> > > > > some of the operators may no longer be considered the right
> > > >> > > > implementations
> > > >> > > > > with the advent of the windowed operator.
> > > >> > > > >
> > > >> > > > > Since contrib used to be the place that housed operators
> that
> > > >> > required
> > > >> > > > > third party libraries, there are some production quality
> > > >> operators in
> > > >> > > > there
> > > >> > > > > that need to be pulled out into top level modules.
> > > >> > > > >
> > > >> > > > > I think we are in agreement that for operators that we
> > consider
> > > >> > stable,
> > > >> > > > we
> > > >> > > > > should provide a b/w stub. I would add that we strongly
> > consider
> > > >> > > creating
> > > >> > > > > the org.apache.apex counterparts of any stable operators
> that
> > > are
> > > >> in
> > > >> > > > > contrib out in top level modules and have the
> com.datatorrent
> > > >> stubs
> > > >> > in
> > > >> > > > > contrib.
> > > >> > > > >
> > > >> > > > > For the operators not considered stable, I would prefer we
> > > either
> > > >> > leave
> > > >> > > > the
> > > >> > > > > current package path as is or if we change the package path
> > then
> > > >> > create
> > > >> > > > the
> > > >> > > > > b/w stub as I am not sure how widely they are in use (so, in
> > > >> essence,
> > > >> > > > > preserve semantic versioning). It would be good if there is
> a
> > > >> > followup
> > > >> > > > JIRA
> > > >> > > > > that takes a look at what other operators can be moved to
> > > contrib
> > > >> > with
> > > >> > > > the
> > > >> > > > > advent of the newer frameworks and understanding.
> > > >> > > > >
> > > >> > > > > Thanks
> > > >> > > > >
> > > >> > > > > On Fri, Jul 14, 2017 at 9:44 AM, Thomas Weise <
> thw@apache.org
> > >
> > > >> > wrote:
> > > >> > > > >
> > > >> > > > > > Most of the operators evolve, as is quite clearly
> indicated
> > by
> > > >> > > > @Evolving
> > > >> > > > > > annotations. So while perhaps 0.x.x would be a more
> > > appropriate
> > > >> > > version
> > > >> > > > > > number, I don't think you can change that.
> > > >> > > > > >
> > > >> > > > > > Thomas
> > > >> > > > > >
> > > >> > > > > > On Fri, Jul 14, 2017 at 9:35 AM, Vlad Rozov <
> > > >> > v.rozov@datatorrent.com
> > > >> > > >
> > > >> > > > > > wrote:
> > > >> > > > > >
> > > >> > > > > > > If entire library is not stable, its version should be
> > 0.x.x
> > > >> and
> > > >> > > > there
> > > >> > > > > > > should not be any semantic versioning enabled or
> implied.
> > It
> > > >> > > evolves.
> > > >> > > > > If
> > > >> > > > > > > some operators are stable as 3.8.x version suggests, the
> > > >> library
> > > >> > > > should
> > > >> > > > > > > follow semantic versioning and it is not OK to make a
> > major
> > > >> > change
> > > >> > > > that
> > > >> > > > > > > breaks semantic versioning across the entire library. It
> > is
> > > >> not a
> > > >> > > > > finding
> > > >> > > > > > > an excuse not to make a change. For me semantic
> versioning
> > > and
> > > >> > > > backward
> > > >> > > > > > > compatibility is more important than a proper package
> > name.
> > > >> > > > > > >
> > > >> > > > > > > Thank you,
> > > >> > > > > > >
> > > >> > > > > > > Vlad
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > > > On 7/14/17 09:11, Thomas Weise wrote:
> > > >> > > > > > >
> > > >> > > > > > >> Semantic versioning makes sense when you have a stable
> > > >> baseline.
> > > >> > > As
> > > >> > > > > long
> > > >> > > > > > >> as
> > > >> > > > > > >> frequent fixes need to be made to address basic issues,
> > it
> > > >> makes
> > > >> > > no
> > > >> > > > > > sense
> > > >> > > > > > >> to declare operators stable, which is why they are
> marked
> > > >> > > evolving.
> > > >> > > > > > >>
> > > >> > > > > > >> Instead of finding reasons to avoid changes that should
> > > have
> > > >> > been
> > > >> > > > > made a
> > > >> > > > > > >> long time ago, why not discuss what a "stable operator"
> > is
> > > >> and
> > > >> > > which
> > > >> > > > > > ones
> > > >> > > > > > >> deserve to be in that category. Those are the ones that
> > > will
> > > >> get
> > > >> > > the
> > > >> > > > > > >> backward compatibility stub.
> > > >> > > > > > >>
> > > >> > > > > > >> Thanks,
> > > >> > > > > > >> Thomas
> > > >> > > > > > >>
> > > >> > > > > > >>
> > > >> > > > > > >> On Fri, Jul 14, 2017 at 8:34 AM, Vlad Rozov <
> > > >> > > > v.rozov@datatorrent.com>
> > > >> > > > > > >> wrote:
> > > >> > > > > > >>
> > > >> > > > > > >> My preference is to agree that the next Malhar release
> is
> > > >> 4.0.0
> > > >> > > and
> > > >> > > > > make
> > > >> > > > > > >>> the proposed changes when 3.8.0 goes out. Otherwise
> why
> > to
> > > >> keep
> > > >> > > > > > semantic
> > > >> > > > > > >>> versioning checks on Malhar in case there is no
> version
> > > >> > > > > compatibility.
> > > >> > > > > > >>>
> > > >> > > > > > >>> Thank you,
> > > >> > > > > > >>>
> > > >> > > > > > >>> Vlad
> > > >> > > > > > >>>
> > > >> > > > > > >>>
> > > >> > > > > > >>> On 7/14/17 08:11, Thomas Weise wrote:
> > > >> > > > > > >>>
> > > >> > > > > > >>> next release is 3.8.0
> > > >> > > > > > >>>>
> > > >> > > > > > >>>> On Fri, Jul 14, 2017 at 7:55 AM, Vlad Rozov <
> > > >> > > > > v.rozov@datatorrent.com>
> > > >> > > > > > >>>> wrote:
> > > >> > > > > > >>>>
> > > >> > > > > > >>>> next release - 3.9.0 or 4.0.0?
> > > >> > > > > > >>>>
> > > >> > > > > > >>>>> Thank you,
> > > >> > > > > > >>>>>
> > > >> > > > > > >>>>> Vlad
> > > >> > > > > > >>>>>
> > > >> > > > > > >>>>> On 7/13/17 22:25, Thomas Weise wrote:
> > > >> > > > > > >>>>>
> > > >> > > > > > >>>>> It is time to resurrect this thread and get going
> with
> > > the
> > > >> > > work.
> > > >> > > > > > >>>>>
> > > >> > > > > > >>>>>> For the next release, I will sign up to do the
> > package
> > > >> move
> > > >> > in
> > > >> > > > > > Malhar:
> > > >> > > > > > >>>>>>
> > > >> > > > > > >>>>>> https://issues.apache.org/
> > jira/browse/APEXMALHAR-2517
> > > >> > > > > > >>>>>>
> > > >> > > > > > >>>>>> In general this will be straightforward; most
> classes
> > > in
> > > >> > > Malhar
> > > >> > > > > are
> > > >> > > > > > >>>>>> marked
> > > >> > > > > > >>>>>> evolving and it is trivial for users to change
> import
> > > >> > > > statements.
> > > >> > > > > > >>>>>> However,
> > > >> > > > > > >>>>>> I would suggest to discuss if there are selected
> > > >> operators
> > > >> > > that
> > > >> > > > > are
> > > >> > > > > > >>>>>> worth
> > > >> > > > > > >>>>>> keeping a backward compatibility stub in the old
> > > >> location.
> > > >> > > > > > >>>>>>
> > > >> > > > > > >>>>>> Here is my plan:
> > > >> > > > > > >>>>>>
> > > >> > > > > > >>>>>> 1. Relocate all classes in *lib* and *contrib*
> within
> > > one
> > > >> > PR -
> > > >> > > > > this
> > > >> > > > > > is
> > > >> > > > > > >>>>>> all
> > > >> > > > > > >>>>>>
> > > >> > > > > > >>>>>> IDE automated work
> > > >> > > > > > >>>>>> 2. Add backward compatibility classes, if, any in
> > > >> separate
> > > >> > PR
> > > >> > > > > > >>>>>> 3. Create PR for Megh library to reflect moved
> > classes
> > > >> > > > > > >>>>>>
> > > >> > > > > > >>>>>> Thanks,
> > > >> > > > > > >>>>>> Thomas
> > > >> > > > > > >>>>>>
> > > >> > > > > > >>>>>>
> > > >> > > > > > >>>>>>
> > > >> > > > > > >>>>>> On Wed, Mar 1, 2017 at 2:24 PM, Pramod Immaneni <
> > > >> > > > > > >>>>>> pramod@datatorrent.com
> > > >> > > > > > >>>>>> wrote:
> > > >> > > > > > >>>>>>
> > > >> > > > > > >>>>>> Inline
> > > >> > > > > > >>>>>>
> > > >> > > > > > >>>>>> On Mon, Feb 27, 2017 at 11:03 PM, Thomas Weise <
> > > >> > > thw@apache.org>
> > > >> > > > > > >>>>>>> wrote:
> > > >> > > > > > >>>>>>>
> > > >> > > > > > >>>>>>> -->
> > > >> > > > > > >>>>>>>
> > > >> > > > > > >>>>>>> On Mon, Feb 27, 2017 at 1:27 PM, Pramod Immaneni <
> > > >> > > > > > >>>>>>>> pramod@datatorrent.com
> > > >> > > > > > >>>>>>>>
> > > >> > > > > > >>>>>>>> wrote:
> > > >> > > > > > >>>>>>>>
> > > >> > > > > > >>>>>>>> For malhar, for existing operators, I prefer we
> do
> > > >> this as
> > > >> > > > part
> > > >> > > > > of
> > > >> > > > > > >>>>>>>> the
> > > >> > > > > > >>>>>>>>
> > > >> > > > > > >>>>>>>> planned refactoring for breaking the monolith
> > modules
> > > >> into
> > > >> > > > baby
> > > >> > > > > > >>>>>>>>>
> > > >> > > > > > >>>>>>>>> packages
> > > >> > > > > > >>>>>>>>>
> > > >> > > > > > >>>>>>>> and would also prefer deprecating the existing
> > > >> operators
> > > >> > in
> > > >> > > > > place.
> > > >> > > > > > >>>>>>>> Refactor into smaller modules was discussed for
> > > >> > > malhar-contrib
> > > >> > > > > and
> > > >> > > > > > >>>>>>>> given
> > > >> > > > > > >>>>>>>> the overall state of that module I think it is OK
> > to
> > > >> defer
> > > >> > > > > package
> > > >> > > > > > >>>>>>>>
> > > >> > > > > > >>>>>>>> renaming
> > > >> > > > > > >>>>>>>>
> > > >> > > > > > >>>>>>> there. I do however prefer to see the package
> rename
> > > >> > > addressed
> > > >> > > > > for
> > > >> > > > > > >>>>>>>
> > > >> > > > > > >>>>>>>> other
> > > >> > > > > > >>>>>>>> modules, especially for the main library module.
> > > >> > > > > > >>>>>>>>
> > > >> > > > > > >>>>>>>> Should we consider breaking the library into
> > smaller
> > > >> > modules
> > > >> > > > as
> > > >> > > > > > >>>>>>>> well,
> > > >> > > > > > >>>>>>>>
> > > >> > > > > > >>>>>>>> the
> > > >> > > > > > >>>>>>> file/block operators for example probably can be
> in
> > > >> their
> > > >> > own
> > > >> > > > > > module
> > > >> > > > > > >>>>>>> from
> > > >> > > > > > >>>>>>> just an organizational perspective.
> > > >> > > > > > >>>>>>>
> > > >> > > > > > >>>>>>>
> > > >> > > > > > >>>>>>>
> > > >> > > > > > >>>>>>> This
> > > >> > > > > > >>>>>>>
> > > >> > > > > > >>>>>>>> will help us achieve two things. First, the user
> > will
> > > >> see
> > > >> > > all
> > > >> > > > > the
> > > >> > > > > > >>>>>>>>> new
> > > >> > > > > > >>>>>>>>> changes at once as opposed to dealing with it
> > twice
> > > >> (with
> > > >> > > > > package
> > > >> > > > > > >>>>>>>>>
> > > >> > > > > > >>>>>>>>> rename
> > > >> > > > > > >>>>>>>>>
> > > >> > > > > > >>>>>>>> and dependency changes) and second it will allow
> > for
> > > a
> > > >> > > > smoother
> > > >> > > > > > >>>>>>>> transition
> > > >> > > > > > >>>>>>>>
> > > >> > > > > > >>>>>>>> as the existing code will still work in a
> > deprecated
> > > >> > state.
> > > >> > > It
> > > >> > > > > > will
> > > >> > > > > > >>>>>>>>
> > > >> > > > > > >>>>>>>>> also
> > > >> > > > > > >>>>>>>>>
> > > >> > > > > > >>>>>>>> give a more consistent structure to malhar. For
> new
> > > >> > > operators,
> > > >> > > > > we
> > > >> > > > > > >>>>>>>> can
> > > >> > > > > > >>>>>>>> go
> > > >> > > > > > >>>>>>>> with the new package path but we need to ensure
> > they
> > > >> will
> > > >> > > get
> > > >> > > > > > moved
> > > >> > > > > > >>>>>>>> into
> > > >> > > > > > >>>>>>>> the baby packages as well.
> > > >> > > > > > >>>>>>>>
> > > >> > > > > > >>>>>>>> I think existing operators should be renamed so
> > that
> > > >> git
> > > >> > > > history
> > > >> > > > > > >>>>>>>>>
> > > >> > > > > > >>>>>>>>> remains. A
> > > >> > > > > > >>>>>>>>
> > > >> > > > > > >>>>>>> possible solution for backward compatibility could
> > be
> > > to
> > > >> > > > > > subsequently
> > > >> > > > > > >>>>>>>
> > > >> > > > > > >>>>>>>> add
> > > >> > > > > > >>>>>>>> empty subclasses in the previous location (for
> > > existing
> > > >> > > > concrete
> > > >> > > > > > >>>>>>>>
> > > >> > > > > > >>>>>>>> operators
> > > >> > > > > > >>>>>>>>
> > > >> > > > > > >>>>>>> that we know are actually in use) to simplify
> > > migration
> > > >> for
> > > >> > > > > users.
> > > >> > > > > > >>>>>>>
> > > >> > > > > > >>>>>>>>
> > > >> > > > > > >>>>>>>> Yes we can do that.
> > > >> > > > > > >>>>>>>>
> > > >> > > > > > >>>>>>>> For demos, we can modify the paths as the apps
> are
> > > >> > typically
> > > >> > > > > used
> > > >> > > > > > >>>>>>>
> > > >> > > > > > >>>>>>> wholesale
> > > >> > > > > > >>>>>>>>
> > > >> > > > > > >>>>>>>> and the interface is typically manual
> interaction.
> > > >> > > > > > >>>>>>>>
> > > >> > > > > > >>>>>>>>> For core, if we are adding new api subsystems,
> > like
> > > >> the
> > > >> > > > > launcher
> > > >> > > > > > >>>>>>>>> api
> > > >> > > > > > >>>>>>>>> we
> > > >> > > > > > >>>>>>>>> added recently for example, we can go with new
> > > package
> > > >> > path
> > > >> > > > but
> > > >> > > > > > if
> > > >> > > > > > >>>>>>>>> we
> > > >> > > > > > >>>>>>>>>
> > > >> > > > > > >>>>>>>>> are
> > > >> > > > > > >>>>>>>>>
> > > >> > > > > > >>>>>>>> making incremental additions to existing
> > > >> functionality, I
> > > >> > > feel
> > > >> > > > > it
> > > >> > > > > > is
> > > >> > > > > > >>>>>>>> better
> > > >> > > > > > >>>>>>>>
> > > >> > > > > > >>>>>>>> to keep it in the same package. I also prefer we
> > keep
> > > >> the
> > > >> > > > > package
> > > >> > > > > > of
> > > >> > > > > > >>>>>>>>
> > > >> > > > > > >>>>>>>>> the
> > > >> > > > > > >>>>>>>>>
> > > >> > > > > > >>>>>>>> implementation classes consistent with api, for
> > > >> > > > > understandability
> > > >> > > > > > >>>>>>>> and
> > > >> > > > > > >>>>>>>>
> > > >> > > > > > >>>>>>>> readability of the code. So, for example, we
> don't
> > > >> change
> > > >> > > > > package
> > > >> > > > > > >>>>>>>>> path
> > > >> > > > > > >>>>>>>>>
> > > >> > > > > > >>>>>>>>> of
> > > >> > > > > > >>>>>>>>>
> > > >> > > > > > >>>>>>>> LogicalPlan as it is an implementation of DAG. It
> > is
> > > >> > > > subjective,
> > > >> > > > > > but
> > > >> > > > > > >>>>>>>> it
> > > >> > > > > > >>>>>>>>
> > > >> > > > > > >>>>>>>> will be good if we can also do the same with
> > classes
> > > >> > closely
> > > >> > > > > > related
> > > >> > > > > > >>>>>>>>> to
> > > >> > > > > > >>>>>>>>>
> > > >> > > > > > >>>>>>>>> the
> > > >> > > > > > >>>>>>>>>
> > > >> > > > > > >>>>>>>> implementation classes as well. Maybe we can
> moving
> > > >> these
> > > >> > > on a
> > > >> > > > > > >>>>>>>> package
> > > >> > > > > > >>>>>>>>
> > > >> > > > > > >>>>>>>>> by
> > > >> > > > > > >>>>>>>>>
> > > >> > > > > > >>>>>>>> package basis, like everything in
> > > >> > > com.datatorrent.stram.engine
> > > >> > > > > > could
> > > >> > > > > > >>>>>>>> be
> > > >> > > > > > >>>>>>>>
> > > >> > > > > > >>>>>>>> moved. For completely internal components like
> > buffer
> > > >> > > server,
> > > >> > > > we
> > > >> > > > > > can
> > > >> > > > > > >>>>>>>>>
> > > >> > > > > > >>>>>>>>> move
> > > >> > > > > > >>>>>>>>>
> > > >> > > > > > >>>>>>>> them wholesale. We can consider moving all api
> and
> > > >> > classes,
> > > >> > > > when
> > > >> > > > > > we
> > > >> > > > > > >>>>>>>> go
> > > >> > > > > > >>>>>>>> to
> > > >> > > > > > >>>>>>>> next major release but would like to see if we
> can
> > > >> find a
> > > >> > > way
> > > >> > > > to
> > > >> > > > > > >>>>>>>> support
> > > >> > > > > > >>>>>>>> existing api for one more major release in
> > deprecated
> > > >> > mode.
> > > >> > > > > > >>>>>>>>
> > > >> > > > > > >>>>>>>> The point of the major release is to enable
> > backward
> > > >> > > > > incompatible
> > > >> > > > > > >>>>>>>>>
> > > >> > > > > > >>>>>>>>> changes
> > > >> > > > > > >>>>>>>> and I don't think it is realistic to support the
> > > >> existing
> > > >> > > API
> > > >> > > > > for
> > > >> > > > > > >>>>>>>> another
> > > >> > > > > > >>>>>>>> major release. IMO it is also not necessary as
> most
> > > >> > existing
> > > >> > > > > > >>>>>>>> application
> > > >> > > > > > >>>>>>>> code refers to operators, attributes and the
> > > >> application
> > > >> > > > > > interface.
> > > >> > > > > > >>>>>>>>
> > > >> > > > > > >>>>>>>> Perhaps
> > > >> > > > > > >>>>>>>>
> > > >> > > > > > >>>>>>> it is possible to keep those around as interface
> > > >> extensions
> > > >> > > to
> > > >> > > > > help
> > > >> > > > > > >>>>>>>
> > > >> > > > > > >>>>>>>> migration. Custom operators may need to be
> migrated
> > > to
> > > >> > > reflect
> > > >> > > > > API
> > > >> > > > > > >>>>>>>>
> > > >> > > > > > >>>>>>>> changes,
> > > >> > > > > > >>>>>>>>
> > > >> > > > > > >>>>>>> and I would consider that a reasonable task for
> > > operator
> > > >> > > > > developers
> > > >> > > > > > >>>>>>> as
> > > >> > > > > > >>>>>>>
> > > >> > > > > > >>>>>>>> part
> > > >> > > > > > >>>>>>>>
> > > >> > > > > > >>>>>>> of a major upgrade.
> > > >> > > > > > >>>>>>>
> > > >> > > > > > >>>>>>>> It would be good if we can keep them as
> deprecated
> > > >> > interface
> > > >> > > > > > >>>>>>>> extensions
> > > >> > > > > > >>>>>>>>
> > > >> > > > > > >>>>>>>> for
> > > >> > > > > > >>>>>>> one release to provide a smoother transition.
> > > >> > > > > > >>>>>>>
> > > >> > > > > > >>>>>>>
> > > >> > > > > > >>>>>>> API and implementation in engine are kept separate
> > > >> > > > intentionally.
> > > >> > > > > > >>>>>>> They
> > > >> > > > > > >>>>>>>
> > > >> > > > > > >>>>>>> reside in different packages today, so I don't
> see a
> > > >> > problem
> > > >> > > > > > renaming
> > > >> > > > > > >>>>>>>> com.datatorrent.stram.engine as you say, even
> when
> > > the
> > > >> API
> > > >> > > > > cannot
> > > >> > > > > > be
> > > >> > > > > > >>>>>>>> touched right away.
> > > >> > > > > > >>>>>>>>
> > > >> > > > > > >>>>>>>> They are different packages but sharing a common
> > > prefix
> > > >> > with
> > > >> > > > api
> > > >> > > > > > >>>>>>>> will
> > > >> > > > > > >>>>>>>> be
> > > >> > > > > > >>>>>>>>
> > > >> > > > > > >>>>>>>> helpful to someone new to codebase in terms of
> > > >> > readability.
> > > >> > > > Not
> > > >> > > > > a
> > > >> > > > > > >>>>>>> big
> > > >> > > > > > >>>>>>> deal
> > > >> > > > > > >>>>>>> and can be changed.
> > > >> > > > > > >>>>>>>
> > > >> > > > > > >>>>>>>
> > > >> > > > > > >>>>>>> Thanks
> > > >> > > > > > >>>>>>>
> > > >> > > > > > >>>>>>> On Mon, Feb 27, 2017 at 7:39 AM, Thomas Weise <
> > > >> > > thw@apache.org>
> > > >> > > > > > >>>>>>>> wrote:
> > > >> > > > > > >>>>>>>>
> > > >> > > > > > >>>>>>>>> Hi,
> > > >> > > > > > >>>>>>>>>
> > > >> > > > > > >>>>>>>>> This topic has come up on several PRs and I
> think
> > it
> > > >> > > > warrants a
> > > >> > > > > > >>>>>>>>>>
> > > >> > > > > > >>>>>>>>>> broader
> > > >> > > > > > >>>>>>>>>>
> > > >> > > > > > >>>>>>>>> discussion.
> > > >> > > > > > >>>>>>>>
> > > >> > > > > > >>>>>>>> At the time of incubation, the decision was to
> > defer
> > > >> > change
> > > >> > > of
> > > >> > > > > > Java
> > > >> > > > > > >>>>>>>>>
> > > >> > > > > > >>>>>>>>>> packages from com.datatorrent to
> org.apache.apex
> > > till
> > > >> > next
> > > >> > > > > major
> > > >> > > > > > >>>>>>>>>>
> > > >> > > > > > >>>>>>>>>> release
> > > >> > > > > > >>>>>>>>>>
> > > >> > > > > > >>>>>>>>> to
> > > >> > > > > > >>>>>>>>>
> > > >> > > > > > >>>>>>>>> ensure backward compatibility for users.
> > > >> > > > > > >>>>>>>>>
> > > >> > > > > > >>>>>>>>>> Unfortunately that has lead to some confusion,
> as
> > > >> > > > contributors
> > > >> > > > > > >>>>>>>>>>
> > > >> > > > > > >>>>>>>>>> continue
> > > >> > > > > > >>>>>>>>>>
> > > >> > > > > > >>>>>>>>> to
> > > >> > > > > > >>>>>>>>
> > > >> > > > > > >>>>>>>> add new code under legacy packages.
> > > >> > > > > > >>>>>>>>>
> > > >> > > > > > >>>>>>>>>> It is also a wider issue that examples for
> using
> > > Apex
> > > >> > > > continue
> > > >> > > > > > to
> > > >> > > > > > >>>>>>>>>>
> > > >> > > > > > >>>>>>>>>> refer
> > > >> > > > > > >>>>>>>>>>
> > > >> > > > > > >>>>>>>>> to
> > > >> > > > > > >>>>>>>>
> > > >> > > > > > >>>>>>>> com.datatorrent packages, nearly one year after
> > > >> > graduation.
> > > >> > > > More
> > > >> > > > > > and
> > > >> > > > > > >>>>>>>>>
> > > >> > > > > > >>>>>>>>>> more
> > > >> > > > > > >>>>>>>>>>
> > > >> > > > > > >>>>>>>>> user code is being built on top of something
> that
> > > >> needs
> > > >> > to
> > > >> > > > > > change,
> > > >> > > > > > >>>>>>>>> the
> > > >> > > > > > >>>>>>>>>
> > > >> > > > > > >>>>>>>>> can
> > > >> > > > > > >>>>>>>>
> > > >> > > > > > >>>>>>>> is being kicked down the road and users will face
> > > more
> > > >> > > changes
> > > >> > > > > > >>>>>>>>> later.
> > > >> > > > > > >>>>>>>>>
> > > >> > > > > > >>>>>>>>>> I would like to propose the following:
> > > >> > > > > > >>>>>>>>>>
> > > >> > > > > > >>>>>>>>>> 1. All new code has to be submitted under
> > > >> > org.apache.apex
> > > >> > > > > > packages
> > > >> > > > > > >>>>>>>>>>
> > > >> > > > > > >>>>>>>>>> 2. Not all code is under backward compatibility
> > > >> > > restriction
> > > >> > > > > and
> > > >> > > > > > in
> > > >> > > > > > >>>>>>>>>>
> > > >> > > > > > >>>>>>>>>> those
> > > >> > > > > > >>>>>>>>>>
> > > >> > > > > > >>>>>>>>> cases we can rename the packages right away.
> > > Examples:
> > > >> > > buffer
> > > >> > > > > > >>>>>>>>> server,
> > > >> > > > > > >>>>>>>>>
> > > >> > > > > > >>>>>>>>> engine, demos/examples, benchmarks
> > > >> > > > > > >>>>>>>>>>
> > > >> > > > > > >>>>>>>>>> 3. Discuss when the core API and operators can
> be
> > > >> > changed.
> > > >> > > > For
> > > >> > > > > > >>>>>>>>>>
> > > >> > > > > > >>>>>>>>>> operators
> > > >> > > > > > >>>>>>>>>>
> > > >> > > > > > >>>>>>>>> we
> > > >> > > > > > >>>>>>>>>
> > > >> > > > > > >>>>>>>>> have a bit more freedom to do changes before a
> > major
> > > >> > > release
> > > >> > > > as
> > > >> > > > > > >>>>>>>>> most
> > > >> > > > > > >>>>>>>>>
> > > >> > > > > > >>>>>>>>>> of
> > > >> > > > > > >>>>>>>>>>
> > > >> > > > > > >>>>>>>>> them are marked @Evolving and users have the
> > ability
> > > >> to
> > > >> > > > > continue
> > > >> > > > > > >>>>>>>>
> > > >> > > > > > >>>>>>>> using
> > > >> > > > > > >>>>>>>>>
> > > >> > > > > > >>>>>>>>> prior version of Malhar with newer engine due to
> > > >> engine
> > > >> > > > > backward
> > > >> > > > > > >>>>>>>>
> > > >> > > > > > >>>>>>>> compatibility guarantee.
> > > >> > > > > > >>>>>>>>>
> > > >> > > > > > >>>>>>>>>> Thanks,
> > > >> > > > > > >>>>>>>>>> Thomas
> > > >> > > > > > >>>>>>>>>>
> > > >> > > > > > >>>>>>>>>>
> > > >> > > > > > >>>>>>>>>>
> > > >> > > > > > >>>>>>>>>>
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
>
Re: Java packages: legacy -> org.apache.apex
Posted by Thomas Weise <th...@apache.org>.
The proposed change does NOT require changes to existing apps. It follows
what you see for example in the JDK where older classes are retired without
making breaking major release changes.
Even though not required technically, it looks like the preference is to
make this change with a 4.0 release. I will therefore propose to change the
master branch to 4.0-SNAPSHOT and make this and other changes deemed
worthwhile.
To me it is more important that to newcomers the codebase does not look
like an outdated legacy situation and therefore if I have to make a choice
than pretty much any version number will do for that. So far I have seen
very little interest and initiative in addressing maintenance work like the
packages, checkstyle, CI stability, documentation etc. So it would be great
if people interested would step up and contribute.
Here is the next proposal how to proceed:
- create release-3.8 branch for last 3.8 release right away
- update master to 4.0-SNAPSHOT as part of the open PR #664
- identify further cleanup work for master
- release 4.0
Thanks,
Thomas
On Thu, Aug 17, 2017 at 4:48 PM, Amol Kekre <am...@datatorrent.com> wrote:
> This following pull request should be taken up in 4.0.0. See my comments in
> https://github.com/apache/apex-malhar/pull/664
> <https://www.google.com/url?q=https%3A%2F%2Fgithub.com%
> 2Fapache%2Fapex-malhar%2Fpull%2F664&sa=D&ust=1503020269444000&usg=
> AFQjCNHDPZIg2e6I33Jb1XjB5Ir1FkdURQ>
> https://github.com/apache/apex-malhar/pull/662
> <https://www.google.com/url?q=https%3A%2F%2Fgithub.com%
> 2Fapache%2Fapex-malhar%2Fpull%2F662&sa=D&ust=1503020269444000&usg=
> AFQjCNF7Xde7X55M3qmc8z-D5y3ZwNg7Fg>
>
> This merge should not be done without a consensus. This will require code
> changes to existing apps.
>
> Thks,
> Amol
>
>
>
> E:amol@datatorrent.com | M: 510-449-2606 | Twitter: @*amolhkekre*
>
> www.datatorrent.com
>
>
> On Mon, Aug 14, 2017 at 7:48 PM, Thomas Weise <th...@apache.org> wrote:
>
> > Hi,
> >
> > I opened following PRs for the package change:
> >
> > https://github.com/apache/apex-malhar/pull/662
> >
> > Moves all classes with history retained (hence 2 commits). Also contains
> > checkstyle and other mechanical changes.
> >
> > https://github.com/apache/apex-malhar/pull/664
> >
> > Adds backward compatibility jar.
> >
> > Once above PRs are merged the new artifact can be deployed and introduced
> > as dependency in malhar-library.
> >
> > Please review.
> >
> > Thanks,
> > Thomas
> >
> >
> >
> > On Sun, Jul 16, 2017 at 7:04 AM, Thomas Weise <th...@apache.org> wrote:
> >
> > > My original list of work items contained the b/w compatibility aspect,
> I
> > > don't think there should be any confusion of whether it will be covered
> > > here or not.
> > >
> > > The proposed shading will provide the old classes and they will be
> frozen
> > > as of release 3.7. That's the same as making a copy of the code and
> never
> > > again making changes to the original classes. This cannot be
> accomplished
> > > by using the older 3.7 release in your project because you cannot use 2
> > > different versions of Malhar in parallel unless you apply shading.
> > >
> > > The shaded artifact will only expose the com.datatorrent classes, and
> > they
> > > will be self-contained as the rest of the classes that they may depend
> on
> > > are shaded. The shaded artifact does not evolve, there are not more
> > changes
> > > to com.datatorrent classes after they are relocated in master.
> > >
> > > Thanks,
> > > Thomas
> > >
> > >
> > > On Sun, Jul 16, 2017 at 2:00 AM, Pramod Immaneni <
> pramod@datatorrent.com
> > >
> > > wrote:
> > >
> > >> I don't think we can limit the topic strictly to relocation without
> > having
> > >> a good b/w compatibility story or at least one that goes far enough.
> > >>
> > >> The shading idea sounds interesting. Why not let the shaded version
> move
> > >> forward with each release till we hit a major release. If it is going
> to
> > >> remain pegged at 3.7.0, why shade in the first place as the regular
> > 3.7.0
> > >> release would do the same job and it would be same as the loss of b/w
> > >> compatibility with newer releases.
> > >>
> > >> Thanks
> > >>
> > >> On Sat, Jul 15, 2017 at 7:57 AM, Thomas Weise <th...@apache.org> wrote:
> > >>
> > >> > Discussing what in the future might become stable needs to be a
> > separate
> > >> > thread, it will be a much bigger discussion.
> > >> >
> > >> > The topic here is to relocate the packages. With a few exceptions
> > >> > relocation won't affect the semantic versioning. Semantic versioning
> > is
> > >> > essentially not effective for Malhar because almost everything is
> > >> @Evolving
> > >> > (and there are reasons for that.. -> separate topic)
> > >> >
> > >> > I don't really like the idea of creating bw compatibility stubs for
> > the
> > >> > follow up PR. It creates even more clutter in the source tree than
> > there
> > >> > already is and so here is an alternative suggestion:
> > >> >
> > >> > https://github.com/tweise/apex-malhar/blob/malhar37-
> > >> > compat/shaded-malhar37/pom.xml
> > >> >
> > >> > Create a shaded artifact that provides the old com.datatorrent.*
> > >> classes as
> > >> > of release 3.7. Users can include that artifact if they don't want
> to
> > >> > change import statements. At the same time they have an incentive to
> > >> switch
> > >> > to the relocated classes to take advantage of bug fixes and new
> > >> > functionality.
> > >> >
> > >> > I will work on the first PR that does the relocate. In the meantime,
> > we
> > >> can
> > >> > finalize what backward compatibility support we want to provide and
> > how.
> > >> >
> > >> > Thanks,
> > >> > Thomas
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > On Fri, Jul 14, 2017 at 11:33 AM, Pramod Immaneni <
> > >> pramod@datatorrent.com>
> > >> > wrote:
> > >> >
> > >> > > How about coming up with a list of @Evolving operators that we
> would
> > >> like
> > >> > > to see in the eventual stable list and move those along with the
> not
> > >> > > @Evolving ones in org.apache.apex with b/w stubs and leave the
> rest
> > as
> > >> > they
> > >> > > are. Then have a follow up JIRA for the rest to be moved over to
> > >> contrib
> > >> > > and be deprecated.
> > >> > >
> > >> > > Thanks
> > >> > >
> > >> > > On Fri, Jul 14, 2017 at 10:37 AM, Thomas Weise <
> > >> thomas.weise@gmail.com>
> > >> > > wrote:
> > >> > >
> > >> > > > We need to keep the discussion here on topic, if other things
> are
> > >> piled
> > >> > > on
> > >> > > > top then nothing gets done.
> > >> > > >
> > >> > > > Most operators today are not stable, they are @Evolving. So
> > perhaps
> > >> it
> > >> > is
> > >> > > > necessary to have a separate discussion about that, outside of
> > this
> > >> > > thread.
> > >> > > > My guess is that there are only a few operators that we could
> > >> declare
> > >> > > > stable. Specifically, under contrib the closest one might have
> > been
> > >> > > Kafka,
> > >> > > > but that is already superseded by the newer versions.
> > >> > > >
> > >> > > > Thomas
> > >> > > >
> > >> > > >
> > >> > > > On Fri, Jul 14, 2017 at 10:21 AM, Pramod Immaneni <
> > >> > > pramod@datatorrent.com>
> > >> > > > wrote:
> > >> > > >
> > >> > > > > We had a discussion a while back, agreed to relegate
> non-stable
> > >> and
> > >> > > > > experimental operators to contrib and also added this to
> > >> contribution
> > >> > > > > guidelines. We aexecuted on this and cleaned up the repo by
> > moving
> > >> > > > > operators deemed non-suitable for production use at that time
> to
> > >> > > contrib.
> > >> > > > > So I wouldn't say the operators that are at the top level
> today
> > or
> > >> > the
> > >> > > > ones
> > >> > > > > in library are 0.x.x quality. Granted, we may need to do one
> > more
> > >> > pass
> > >> > > as
> > >> > > > > some of the operators may no longer be considered the right
> > >> > > > implementations
> > >> > > > > with the advent of the windowed operator.
> > >> > > > >
> > >> > > > > Since contrib used to be the place that housed operators that
> > >> > required
> > >> > > > > third party libraries, there are some production quality
> > >> operators in
> > >> > > > there
> > >> > > > > that need to be pulled out into top level modules.
> > >> > > > >
> > >> > > > > I think we are in agreement that for operators that we
> consider
> > >> > stable,
> > >> > > > we
> > >> > > > > should provide a b/w stub. I would add that we strongly
> consider
> > >> > > creating
> > >> > > > > the org.apache.apex counterparts of any stable operators that
> > are
> > >> in
> > >> > > > > contrib out in top level modules and have the com.datatorrent
> > >> stubs
> > >> > in
> > >> > > > > contrib.
> > >> > > > >
> > >> > > > > For the operators not considered stable, I would prefer we
> > either
> > >> > leave
> > >> > > > the
> > >> > > > > current package path as is or if we change the package path
> then
> > >> > create
> > >> > > > the
> > >> > > > > b/w stub as I am not sure how widely they are in use (so, in
> > >> essence,
> > >> > > > > preserve semantic versioning). It would be good if there is a
> > >> > followup
> > >> > > > JIRA
> > >> > > > > that takes a look at what other operators can be moved to
> > contrib
> > >> > with
> > >> > > > the
> > >> > > > > advent of the newer frameworks and understanding.
> > >> > > > >
> > >> > > > > Thanks
> > >> > > > >
> > >> > > > > On Fri, Jul 14, 2017 at 9:44 AM, Thomas Weise <thw@apache.org
> >
> > >> > wrote:
> > >> > > > >
> > >> > > > > > Most of the operators evolve, as is quite clearly indicated
> by
> > >> > > > @Evolving
> > >> > > > > > annotations. So while perhaps 0.x.x would be a more
> > appropriate
> > >> > > version
> > >> > > > > > number, I don't think you can change that.
> > >> > > > > >
> > >> > > > > > Thomas
> > >> > > > > >
> > >> > > > > > On Fri, Jul 14, 2017 at 9:35 AM, Vlad Rozov <
> > >> > v.rozov@datatorrent.com
> > >> > > >
> > >> > > > > > wrote:
> > >> > > > > >
> > >> > > > > > > If entire library is not stable, its version should be
> 0.x.x
> > >> and
> > >> > > > there
> > >> > > > > > > should not be any semantic versioning enabled or implied.
> It
> > >> > > evolves.
> > >> > > > > If
> > >> > > > > > > some operators are stable as 3.8.x version suggests, the
> > >> library
> > >> > > > should
> > >> > > > > > > follow semantic versioning and it is not OK to make a
> major
> > >> > change
> > >> > > > that
> > >> > > > > > > breaks semantic versioning across the entire library. It
> is
> > >> not a
> > >> > > > > finding
> > >> > > > > > > an excuse not to make a change. For me semantic versioning
> > and
> > >> > > > backward
> > >> > > > > > > compatibility is more important than a proper package
> name.
> > >> > > > > > >
> > >> > > > > > > Thank you,
> > >> > > > > > >
> > >> > > > > > > Vlad
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > > On 7/14/17 09:11, Thomas Weise wrote:
> > >> > > > > > >
> > >> > > > > > >> Semantic versioning makes sense when you have a stable
> > >> baseline.
> > >> > > As
> > >> > > > > long
> > >> > > > > > >> as
> > >> > > > > > >> frequent fixes need to be made to address basic issues,
> it
> > >> makes
> > >> > > no
> > >> > > > > > sense
> > >> > > > > > >> to declare operators stable, which is why they are marked
> > >> > > evolving.
> > >> > > > > > >>
> > >> > > > > > >> Instead of finding reasons to avoid changes that should
> > have
> > >> > been
> > >> > > > > made a
> > >> > > > > > >> long time ago, why not discuss what a "stable operator"
> is
> > >> and
> > >> > > which
> > >> > > > > > ones
> > >> > > > > > >> deserve to be in that category. Those are the ones that
> > will
> > >> get
> > >> > > the
> > >> > > > > > >> backward compatibility stub.
> > >> > > > > > >>
> > >> > > > > > >> Thanks,
> > >> > > > > > >> Thomas
> > >> > > > > > >>
> > >> > > > > > >>
> > >> > > > > > >> On Fri, Jul 14, 2017 at 8:34 AM, Vlad Rozov <
> > >> > > > v.rozov@datatorrent.com>
> > >> > > > > > >> wrote:
> > >> > > > > > >>
> > >> > > > > > >> My preference is to agree that the next Malhar release is
> > >> 4.0.0
> > >> > > and
> > >> > > > > make
> > >> > > > > > >>> the proposed changes when 3.8.0 goes out. Otherwise why
> to
> > >> keep
> > >> > > > > > semantic
> > >> > > > > > >>> versioning checks on Malhar in case there is no version
> > >> > > > > compatibility.
> > >> > > > > > >>>
> > >> > > > > > >>> Thank you,
> > >> > > > > > >>>
> > >> > > > > > >>> Vlad
> > >> > > > > > >>>
> > >> > > > > > >>>
> > >> > > > > > >>> On 7/14/17 08:11, Thomas Weise wrote:
> > >> > > > > > >>>
> > >> > > > > > >>> next release is 3.8.0
> > >> > > > > > >>>>
> > >> > > > > > >>>> On Fri, Jul 14, 2017 at 7:55 AM, Vlad Rozov <
> > >> > > > > v.rozov@datatorrent.com>
> > >> > > > > > >>>> wrote:
> > >> > > > > > >>>>
> > >> > > > > > >>>> next release - 3.9.0 or 4.0.0?
> > >> > > > > > >>>>
> > >> > > > > > >>>>> Thank you,
> > >> > > > > > >>>>>
> > >> > > > > > >>>>> Vlad
> > >> > > > > > >>>>>
> > >> > > > > > >>>>> On 7/13/17 22:25, Thomas Weise wrote:
> > >> > > > > > >>>>>
> > >> > > > > > >>>>> It is time to resurrect this thread and get going with
> > the
> > >> > > work.
> > >> > > > > > >>>>>
> > >> > > > > > >>>>>> For the next release, I will sign up to do the
> package
> > >> move
> > >> > in
> > >> > > > > > Malhar:
> > >> > > > > > >>>>>>
> > >> > > > > > >>>>>> https://issues.apache.org/
> jira/browse/APEXMALHAR-2517
> > >> > > > > > >>>>>>
> > >> > > > > > >>>>>> In general this will be straightforward; most classes
> > in
> > >> > > Malhar
> > >> > > > > are
> > >> > > > > > >>>>>> marked
> > >> > > > > > >>>>>> evolving and it is trivial for users to change import
> > >> > > > statements.
> > >> > > > > > >>>>>> However,
> > >> > > > > > >>>>>> I would suggest to discuss if there are selected
> > >> operators
> > >> > > that
> > >> > > > > are
> > >> > > > > > >>>>>> worth
> > >> > > > > > >>>>>> keeping a backward compatibility stub in the old
> > >> location.
> > >> > > > > > >>>>>>
> > >> > > > > > >>>>>> Here is my plan:
> > >> > > > > > >>>>>>
> > >> > > > > > >>>>>> 1. Relocate all classes in *lib* and *contrib* within
> > one
> > >> > PR -
> > >> > > > > this
> > >> > > > > > is
> > >> > > > > > >>>>>> all
> > >> > > > > > >>>>>>
> > >> > > > > > >>>>>> IDE automated work
> > >> > > > > > >>>>>> 2. Add backward compatibility classes, if, any in
> > >> separate
> > >> > PR
> > >> > > > > > >>>>>> 3. Create PR for Megh library to reflect moved
> classes
> > >> > > > > > >>>>>>
> > >> > > > > > >>>>>> Thanks,
> > >> > > > > > >>>>>> Thomas
> > >> > > > > > >>>>>>
> > >> > > > > > >>>>>>
> > >> > > > > > >>>>>>
> > >> > > > > > >>>>>> On Wed, Mar 1, 2017 at 2:24 PM, Pramod Immaneni <
> > >> > > > > > >>>>>> pramod@datatorrent.com
> > >> > > > > > >>>>>> wrote:
> > >> > > > > > >>>>>>
> > >> > > > > > >>>>>> Inline
> > >> > > > > > >>>>>>
> > >> > > > > > >>>>>> On Mon, Feb 27, 2017 at 11:03 PM, Thomas Weise <
> > >> > > thw@apache.org>
> > >> > > > > > >>>>>>> wrote:
> > >> > > > > > >>>>>>>
> > >> > > > > > >>>>>>> -->
> > >> > > > > > >>>>>>>
> > >> > > > > > >>>>>>> On Mon, Feb 27, 2017 at 1:27 PM, Pramod Immaneni <
> > >> > > > > > >>>>>>>> pramod@datatorrent.com
> > >> > > > > > >>>>>>>>
> > >> > > > > > >>>>>>>> wrote:
> > >> > > > > > >>>>>>>>
> > >> > > > > > >>>>>>>> For malhar, for existing operators, I prefer we do
> > >> this as
> > >> > > > part
> > >> > > > > of
> > >> > > > > > >>>>>>>> the
> > >> > > > > > >>>>>>>>
> > >> > > > > > >>>>>>>> planned refactoring for breaking the monolith
> modules
> > >> into
> > >> > > > baby
> > >> > > > > > >>>>>>>>>
> > >> > > > > > >>>>>>>>> packages
> > >> > > > > > >>>>>>>>>
> > >> > > > > > >>>>>>>> and would also prefer deprecating the existing
> > >> operators
> > >> > in
> > >> > > > > place.
> > >> > > > > > >>>>>>>> Refactor into smaller modules was discussed for
> > >> > > malhar-contrib
> > >> > > > > and
> > >> > > > > > >>>>>>>> given
> > >> > > > > > >>>>>>>> the overall state of that module I think it is OK
> to
> > >> defer
> > >> > > > > package
> > >> > > > > > >>>>>>>>
> > >> > > > > > >>>>>>>> renaming
> > >> > > > > > >>>>>>>>
> > >> > > > > > >>>>>>> there. I do however prefer to see the package rename
> > >> > > addressed
> > >> > > > > for
> > >> > > > > > >>>>>>>
> > >> > > > > > >>>>>>>> other
> > >> > > > > > >>>>>>>> modules, especially for the main library module.
> > >> > > > > > >>>>>>>>
> > >> > > > > > >>>>>>>> Should we consider breaking the library into
> smaller
> > >> > modules
> > >> > > > as
> > >> > > > > > >>>>>>>> well,
> > >> > > > > > >>>>>>>>
> > >> > > > > > >>>>>>>> the
> > >> > > > > > >>>>>>> file/block operators for example probably can be in
> > >> their
> > >> > own
> > >> > > > > > module
> > >> > > > > > >>>>>>> from
> > >> > > > > > >>>>>>> just an organizational perspective.
> > >> > > > > > >>>>>>>
> > >> > > > > > >>>>>>>
> > >> > > > > > >>>>>>>
> > >> > > > > > >>>>>>> This
> > >> > > > > > >>>>>>>
> > >> > > > > > >>>>>>>> will help us achieve two things. First, the user
> will
> > >> see
> > >> > > all
> > >> > > > > the
> > >> > > > > > >>>>>>>>> new
> > >> > > > > > >>>>>>>>> changes at once as opposed to dealing with it
> twice
> > >> (with
> > >> > > > > package
> > >> > > > > > >>>>>>>>>
> > >> > > > > > >>>>>>>>> rename
> > >> > > > > > >>>>>>>>>
> > >> > > > > > >>>>>>>> and dependency changes) and second it will allow
> for
> > a
> > >> > > > smoother
> > >> > > > > > >>>>>>>> transition
> > >> > > > > > >>>>>>>>
> > >> > > > > > >>>>>>>> as the existing code will still work in a
> deprecated
> > >> > state.
> > >> > > It
> > >> > > > > > will
> > >> > > > > > >>>>>>>>
> > >> > > > > > >>>>>>>>> also
> > >> > > > > > >>>>>>>>>
> > >> > > > > > >>>>>>>> give a more consistent structure to malhar. For new
> > >> > > operators,
> > >> > > > > we
> > >> > > > > > >>>>>>>> can
> > >> > > > > > >>>>>>>> go
> > >> > > > > > >>>>>>>> with the new package path but we need to ensure
> they
> > >> will
> > >> > > get
> > >> > > > > > moved
> > >> > > > > > >>>>>>>> into
> > >> > > > > > >>>>>>>> the baby packages as well.
> > >> > > > > > >>>>>>>>
> > >> > > > > > >>>>>>>> I think existing operators should be renamed so
> that
> > >> git
> > >> > > > history
> > >> > > > > > >>>>>>>>>
> > >> > > > > > >>>>>>>>> remains. A
> > >> > > > > > >>>>>>>>
> > >> > > > > > >>>>>>> possible solution for backward compatibility could
> be
> > to
> > >> > > > > > subsequently
> > >> > > > > > >>>>>>>
> > >> > > > > > >>>>>>>> add
> > >> > > > > > >>>>>>>> empty subclasses in the previous location (for
> > existing
> > >> > > > concrete
> > >> > > > > > >>>>>>>>
> > >> > > > > > >>>>>>>> operators
> > >> > > > > > >>>>>>>>
> > >> > > > > > >>>>>>> that we know are actually in use) to simplify
> > migration
> > >> for
> > >> > > > > users.
> > >> > > > > > >>>>>>>
> > >> > > > > > >>>>>>>>
> > >> > > > > > >>>>>>>> Yes we can do that.
> > >> > > > > > >>>>>>>>
> > >> > > > > > >>>>>>>> For demos, we can modify the paths as the apps are
> > >> > typically
> > >> > > > > used
> > >> > > > > > >>>>>>>
> > >> > > > > > >>>>>>> wholesale
> > >> > > > > > >>>>>>>>
> > >> > > > > > >>>>>>>> and the interface is typically manual interaction.
> > >> > > > > > >>>>>>>>
> > >> > > > > > >>>>>>>>> For core, if we are adding new api subsystems,
> like
> > >> the
> > >> > > > > launcher
> > >> > > > > > >>>>>>>>> api
> > >> > > > > > >>>>>>>>> we
> > >> > > > > > >>>>>>>>> added recently for example, we can go with new
> > package
> > >> > path
> > >> > > > but
> > >> > > > > > if
> > >> > > > > > >>>>>>>>> we
> > >> > > > > > >>>>>>>>>
> > >> > > > > > >>>>>>>>> are
> > >> > > > > > >>>>>>>>>
> > >> > > > > > >>>>>>>> making incremental additions to existing
> > >> functionality, I
> > >> > > feel
> > >> > > > > it
> > >> > > > > > is
> > >> > > > > > >>>>>>>> better
> > >> > > > > > >>>>>>>>
> > >> > > > > > >>>>>>>> to keep it in the same package. I also prefer we
> keep
> > >> the
> > >> > > > > package
> > >> > > > > > of
> > >> > > > > > >>>>>>>>
> > >> > > > > > >>>>>>>>> the
> > >> > > > > > >>>>>>>>>
> > >> > > > > > >>>>>>>> implementation classes consistent with api, for
> > >> > > > > understandability
> > >> > > > > > >>>>>>>> and
> > >> > > > > > >>>>>>>>
> > >> > > > > > >>>>>>>> readability of the code. So, for example, we don't
> > >> change
> > >> > > > > package
> > >> > > > > > >>>>>>>>> path
> > >> > > > > > >>>>>>>>>
> > >> > > > > > >>>>>>>>> of
> > >> > > > > > >>>>>>>>>
> > >> > > > > > >>>>>>>> LogicalPlan as it is an implementation of DAG. It
> is
> > >> > > > subjective,
> > >> > > > > > but
> > >> > > > > > >>>>>>>> it
> > >> > > > > > >>>>>>>>
> > >> > > > > > >>>>>>>> will be good if we can also do the same with
> classes
> > >> > closely
> > >> > > > > > related
> > >> > > > > > >>>>>>>>> to
> > >> > > > > > >>>>>>>>>
> > >> > > > > > >>>>>>>>> the
> > >> > > > > > >>>>>>>>>
> > >> > > > > > >>>>>>>> implementation classes as well. Maybe we can moving
> > >> these
> > >> > > on a
> > >> > > > > > >>>>>>>> package
> > >> > > > > > >>>>>>>>
> > >> > > > > > >>>>>>>>> by
> > >> > > > > > >>>>>>>>>
> > >> > > > > > >>>>>>>> package basis, like everything in
> > >> > > com.datatorrent.stram.engine
> > >> > > > > > could
> > >> > > > > > >>>>>>>> be
> > >> > > > > > >>>>>>>>
> > >> > > > > > >>>>>>>> moved. For completely internal components like
> buffer
> > >> > > server,
> > >> > > > we
> > >> > > > > > can
> > >> > > > > > >>>>>>>>>
> > >> > > > > > >>>>>>>>> move
> > >> > > > > > >>>>>>>>>
> > >> > > > > > >>>>>>>> them wholesale. We can consider moving all api and
> > >> > classes,
> > >> > > > when
> > >> > > > > > we
> > >> > > > > > >>>>>>>> go
> > >> > > > > > >>>>>>>> to
> > >> > > > > > >>>>>>>> next major release but would like to see if we can
> > >> find a
> > >> > > way
> > >> > > > to
> > >> > > > > > >>>>>>>> support
> > >> > > > > > >>>>>>>> existing api for one more major release in
> deprecated
> > >> > mode.
> > >> > > > > > >>>>>>>>
> > >> > > > > > >>>>>>>> The point of the major release is to enable
> backward
> > >> > > > > incompatible
> > >> > > > > > >>>>>>>>>
> > >> > > > > > >>>>>>>>> changes
> > >> > > > > > >>>>>>>> and I don't think it is realistic to support the
> > >> existing
> > >> > > API
> > >> > > > > for
> > >> > > > > > >>>>>>>> another
> > >> > > > > > >>>>>>>> major release. IMO it is also not necessary as most
> > >> > existing
> > >> > > > > > >>>>>>>> application
> > >> > > > > > >>>>>>>> code refers to operators, attributes and the
> > >> application
> > >> > > > > > interface.
> > >> > > > > > >>>>>>>>
> > >> > > > > > >>>>>>>> Perhaps
> > >> > > > > > >>>>>>>>
> > >> > > > > > >>>>>>> it is possible to keep those around as interface
> > >> extensions
> > >> > > to
> > >> > > > > help
> > >> > > > > > >>>>>>>
> > >> > > > > > >>>>>>>> migration. Custom operators may need to be migrated
> > to
> > >> > > reflect
> > >> > > > > API
> > >> > > > > > >>>>>>>>
> > >> > > > > > >>>>>>>> changes,
> > >> > > > > > >>>>>>>>
> > >> > > > > > >>>>>>> and I would consider that a reasonable task for
> > operator
> > >> > > > > developers
> > >> > > > > > >>>>>>> as
> > >> > > > > > >>>>>>>
> > >> > > > > > >>>>>>>> part
> > >> > > > > > >>>>>>>>
> > >> > > > > > >>>>>>> of a major upgrade.
> > >> > > > > > >>>>>>>
> > >> > > > > > >>>>>>>> It would be good if we can keep them as deprecated
> > >> > interface
> > >> > > > > > >>>>>>>> extensions
> > >> > > > > > >>>>>>>>
> > >> > > > > > >>>>>>>> for
> > >> > > > > > >>>>>>> one release to provide a smoother transition.
> > >> > > > > > >>>>>>>
> > >> > > > > > >>>>>>>
> > >> > > > > > >>>>>>> API and implementation in engine are kept separate
> > >> > > > intentionally.
> > >> > > > > > >>>>>>> They
> > >> > > > > > >>>>>>>
> > >> > > > > > >>>>>>> reside in different packages today, so I don't see a
> > >> > problem
> > >> > > > > > renaming
> > >> > > > > > >>>>>>>> com.datatorrent.stram.engine as you say, even when
> > the
> > >> API
> > >> > > > > cannot
> > >> > > > > > be
> > >> > > > > > >>>>>>>> touched right away.
> > >> > > > > > >>>>>>>>
> > >> > > > > > >>>>>>>> They are different packages but sharing a common
> > prefix
> > >> > with
> > >> > > > api
> > >> > > > > > >>>>>>>> will
> > >> > > > > > >>>>>>>> be
> > >> > > > > > >>>>>>>>
> > >> > > > > > >>>>>>>> helpful to someone new to codebase in terms of
> > >> > readability.
> > >> > > > Not
> > >> > > > > a
> > >> > > > > > >>>>>>> big
> > >> > > > > > >>>>>>> deal
> > >> > > > > > >>>>>>> and can be changed.
> > >> > > > > > >>>>>>>
> > >> > > > > > >>>>>>>
> > >> > > > > > >>>>>>> Thanks
> > >> > > > > > >>>>>>>
> > >> > > > > > >>>>>>> On Mon, Feb 27, 2017 at 7:39 AM, Thomas Weise <
> > >> > > thw@apache.org>
> > >> > > > > > >>>>>>>> wrote:
> > >> > > > > > >>>>>>>>
> > >> > > > > > >>>>>>>>> Hi,
> > >> > > > > > >>>>>>>>>
> > >> > > > > > >>>>>>>>> This topic has come up on several PRs and I think
> it
> > >> > > > warrants a
> > >> > > > > > >>>>>>>>>>
> > >> > > > > > >>>>>>>>>> broader
> > >> > > > > > >>>>>>>>>>
> > >> > > > > > >>>>>>>>> discussion.
> > >> > > > > > >>>>>>>>
> > >> > > > > > >>>>>>>> At the time of incubation, the decision was to
> defer
> > >> > change
> > >> > > of
> > >> > > > > > Java
> > >> > > > > > >>>>>>>>>
> > >> > > > > > >>>>>>>>>> packages from com.datatorrent to org.apache.apex
> > till
> > >> > next
> > >> > > > > major
> > >> > > > > > >>>>>>>>>>
> > >> > > > > > >>>>>>>>>> release
> > >> > > > > > >>>>>>>>>>
> > >> > > > > > >>>>>>>>> to
> > >> > > > > > >>>>>>>>>
> > >> > > > > > >>>>>>>>> ensure backward compatibility for users.
> > >> > > > > > >>>>>>>>>
> > >> > > > > > >>>>>>>>>> Unfortunately that has lead to some confusion, as
> > >> > > > contributors
> > >> > > > > > >>>>>>>>>>
> > >> > > > > > >>>>>>>>>> continue
> > >> > > > > > >>>>>>>>>>
> > >> > > > > > >>>>>>>>> to
> > >> > > > > > >>>>>>>>
> > >> > > > > > >>>>>>>> add new code under legacy packages.
> > >> > > > > > >>>>>>>>>
> > >> > > > > > >>>>>>>>>> It is also a wider issue that examples for using
> > Apex
> > >> > > > continue
> > >> > > > > > to
> > >> > > > > > >>>>>>>>>>
> > >> > > > > > >>>>>>>>>> refer
> > >> > > > > > >>>>>>>>>>
> > >> > > > > > >>>>>>>>> to
> > >> > > > > > >>>>>>>>
> > >> > > > > > >>>>>>>> com.datatorrent packages, nearly one year after
> > >> > graduation.
> > >> > > > More
> > >> > > > > > and
> > >> > > > > > >>>>>>>>>
> > >> > > > > > >>>>>>>>>> more
> > >> > > > > > >>>>>>>>>>
> > >> > > > > > >>>>>>>>> user code is being built on top of something that
> > >> needs
> > >> > to
> > >> > > > > > change,
> > >> > > > > > >>>>>>>>> the
> > >> > > > > > >>>>>>>>>
> > >> > > > > > >>>>>>>>> can
> > >> > > > > > >>>>>>>>
> > >> > > > > > >>>>>>>> is being kicked down the road and users will face
> > more
> > >> > > changes
> > >> > > > > > >>>>>>>>> later.
> > >> > > > > > >>>>>>>>>
> > >> > > > > > >>>>>>>>>> I would like to propose the following:
> > >> > > > > > >>>>>>>>>>
> > >> > > > > > >>>>>>>>>> 1. All new code has to be submitted under
> > >> > org.apache.apex
> > >> > > > > > packages
> > >> > > > > > >>>>>>>>>>
> > >> > > > > > >>>>>>>>>> 2. Not all code is under backward compatibility
> > >> > > restriction
> > >> > > > > and
> > >> > > > > > in
> > >> > > > > > >>>>>>>>>>
> > >> > > > > > >>>>>>>>>> those
> > >> > > > > > >>>>>>>>>>
> > >> > > > > > >>>>>>>>> cases we can rename the packages right away.
> > Examples:
> > >> > > buffer
> > >> > > > > > >>>>>>>>> server,
> > >> > > > > > >>>>>>>>>
> > >> > > > > > >>>>>>>>> engine, demos/examples, benchmarks
> > >> > > > > > >>>>>>>>>>
> > >> > > > > > >>>>>>>>>> 3. Discuss when the core API and operators can be
> > >> > changed.
> > >> > > > For
> > >> > > > > > >>>>>>>>>>
> > >> > > > > > >>>>>>>>>> operators
> > >> > > > > > >>>>>>>>>>
> > >> > > > > > >>>>>>>>> we
> > >> > > > > > >>>>>>>>>
> > >> > > > > > >>>>>>>>> have a bit more freedom to do changes before a
> major
> > >> > > release
> > >> > > > as
> > >> > > > > > >>>>>>>>> most
> > >> > > > > > >>>>>>>>>
> > >> > > > > > >>>>>>>>>> of
> > >> > > > > > >>>>>>>>>>
> > >> > > > > > >>>>>>>>> them are marked @Evolving and users have the
> ability
> > >> to
> > >> > > > > continue
> > >> > > > > > >>>>>>>>
> > >> > > > > > >>>>>>>> using
> > >> > > > > > >>>>>>>>>
> > >> > > > > > >>>>>>>>> prior version of Malhar with newer engine due to
> > >> engine
> > >> > > > > backward
> > >> > > > > > >>>>>>>>
> > >> > > > > > >>>>>>>> compatibility guarantee.
> > >> > > > > > >>>>>>>>>
> > >> > > > > > >>>>>>>>>> Thanks,
> > >> > > > > > >>>>>>>>>> Thomas
> > >> > > > > > >>>>>>>>>>
> > >> > > > > > >>>>>>>>>>
> > >> > > > > > >>>>>>>>>>
> > >> > > > > > >>>>>>>>>>
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>
Re: Java packages: legacy -> org.apache.apex
Posted by Amol Kekre <am...@datatorrent.com>.
This following pull request should be taken up in 4.0.0. See my comments in
https://github.com/apache/apex-malhar/pull/664
<https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2Fapache%2Fapex-malhar%2Fpull%2F664&sa=D&ust=1503020269444000&usg=AFQjCNHDPZIg2e6I33Jb1XjB5Ir1FkdURQ>
https://github.com/apache/apex-malhar/pull/662
<https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2Fapache%2Fapex-malhar%2Fpull%2F662&sa=D&ust=1503020269444000&usg=AFQjCNF7Xde7X55M3qmc8z-D5y3ZwNg7Fg>
This merge should not be done without a consensus. This will require code
changes to existing apps.
Thks,
Amol
E:amol@datatorrent.com | M: 510-449-2606 | Twitter: @*amolhkekre*
www.datatorrent.com
On Mon, Aug 14, 2017 at 7:48 PM, Thomas Weise <th...@apache.org> wrote:
> Hi,
>
> I opened following PRs for the package change:
>
> https://github.com/apache/apex-malhar/pull/662
>
> Moves all classes with history retained (hence 2 commits). Also contains
> checkstyle and other mechanical changes.
>
> https://github.com/apache/apex-malhar/pull/664
>
> Adds backward compatibility jar.
>
> Once above PRs are merged the new artifact can be deployed and introduced
> as dependency in malhar-library.
>
> Please review.
>
> Thanks,
> Thomas
>
>
>
> On Sun, Jul 16, 2017 at 7:04 AM, Thomas Weise <th...@apache.org> wrote:
>
> > My original list of work items contained the b/w compatibility aspect, I
> > don't think there should be any confusion of whether it will be covered
> > here or not.
> >
> > The proposed shading will provide the old classes and they will be frozen
> > as of release 3.7. That's the same as making a copy of the code and never
> > again making changes to the original classes. This cannot be accomplished
> > by using the older 3.7 release in your project because you cannot use 2
> > different versions of Malhar in parallel unless you apply shading.
> >
> > The shaded artifact will only expose the com.datatorrent classes, and
> they
> > will be self-contained as the rest of the classes that they may depend on
> > are shaded. The shaded artifact does not evolve, there are not more
> changes
> > to com.datatorrent classes after they are relocated in master.
> >
> > Thanks,
> > Thomas
> >
> >
> > On Sun, Jul 16, 2017 at 2:00 AM, Pramod Immaneni <pramod@datatorrent.com
> >
> > wrote:
> >
> >> I don't think we can limit the topic strictly to relocation without
> having
> >> a good b/w compatibility story or at least one that goes far enough.
> >>
> >> The shading idea sounds interesting. Why not let the shaded version move
> >> forward with each release till we hit a major release. If it is going to
> >> remain pegged at 3.7.0, why shade in the first place as the regular
> 3.7.0
> >> release would do the same job and it would be same as the loss of b/w
> >> compatibility with newer releases.
> >>
> >> Thanks
> >>
> >> On Sat, Jul 15, 2017 at 7:57 AM, Thomas Weise <th...@apache.org> wrote:
> >>
> >> > Discussing what in the future might become stable needs to be a
> separate
> >> > thread, it will be a much bigger discussion.
> >> >
> >> > The topic here is to relocate the packages. With a few exceptions
> >> > relocation won't affect the semantic versioning. Semantic versioning
> is
> >> > essentially not effective for Malhar because almost everything is
> >> @Evolving
> >> > (and there are reasons for that.. -> separate topic)
> >> >
> >> > I don't really like the idea of creating bw compatibility stubs for
> the
> >> > follow up PR. It creates even more clutter in the source tree than
> there
> >> > already is and so here is an alternative suggestion:
> >> >
> >> > https://github.com/tweise/apex-malhar/blob/malhar37-
> >> > compat/shaded-malhar37/pom.xml
> >> >
> >> > Create a shaded artifact that provides the old com.datatorrent.*
> >> classes as
> >> > of release 3.7. Users can include that artifact if they don't want to
> >> > change import statements. At the same time they have an incentive to
> >> switch
> >> > to the relocated classes to take advantage of bug fixes and new
> >> > functionality.
> >> >
> >> > I will work on the first PR that does the relocate. In the meantime,
> we
> >> can
> >> > finalize what backward compatibility support we want to provide and
> how.
> >> >
> >> > Thanks,
> >> > Thomas
> >> >
> >> >
> >> >
> >> >
> >> > On Fri, Jul 14, 2017 at 11:33 AM, Pramod Immaneni <
> >> pramod@datatorrent.com>
> >> > wrote:
> >> >
> >> > > How about coming up with a list of @Evolving operators that we would
> >> like
> >> > > to see in the eventual stable list and move those along with the not
> >> > > @Evolving ones in org.apache.apex with b/w stubs and leave the rest
> as
> >> > they
> >> > > are. Then have a follow up JIRA for the rest to be moved over to
> >> contrib
> >> > > and be deprecated.
> >> > >
> >> > > Thanks
> >> > >
> >> > > On Fri, Jul 14, 2017 at 10:37 AM, Thomas Weise <
> >> thomas.weise@gmail.com>
> >> > > wrote:
> >> > >
> >> > > > We need to keep the discussion here on topic, if other things are
> >> piled
> >> > > on
> >> > > > top then nothing gets done.
> >> > > >
> >> > > > Most operators today are not stable, they are @Evolving. So
> perhaps
> >> it
> >> > is
> >> > > > necessary to have a separate discussion about that, outside of
> this
> >> > > thread.
> >> > > > My guess is that there are only a few operators that we could
> >> declare
> >> > > > stable. Specifically, under contrib the closest one might have
> been
> >> > > Kafka,
> >> > > > but that is already superseded by the newer versions.
> >> > > >
> >> > > > Thomas
> >> > > >
> >> > > >
> >> > > > On Fri, Jul 14, 2017 at 10:21 AM, Pramod Immaneni <
> >> > > pramod@datatorrent.com>
> >> > > > wrote:
> >> > > >
> >> > > > > We had a discussion a while back, agreed to relegate non-stable
> >> and
> >> > > > > experimental operators to contrib and also added this to
> >> contribution
> >> > > > > guidelines. We aexecuted on this and cleaned up the repo by
> moving
> >> > > > > operators deemed non-suitable for production use at that time to
> >> > > contrib.
> >> > > > > So I wouldn't say the operators that are at the top level today
> or
> >> > the
> >> > > > ones
> >> > > > > in library are 0.x.x quality. Granted, we may need to do one
> more
> >> > pass
> >> > > as
> >> > > > > some of the operators may no longer be considered the right
> >> > > > implementations
> >> > > > > with the advent of the windowed operator.
> >> > > > >
> >> > > > > Since contrib used to be the place that housed operators that
> >> > required
> >> > > > > third party libraries, there are some production quality
> >> operators in
> >> > > > there
> >> > > > > that need to be pulled out into top level modules.
> >> > > > >
> >> > > > > I think we are in agreement that for operators that we consider
> >> > stable,
> >> > > > we
> >> > > > > should provide a b/w stub. I would add that we strongly consider
> >> > > creating
> >> > > > > the org.apache.apex counterparts of any stable operators that
> are
> >> in
> >> > > > > contrib out in top level modules and have the com.datatorrent
> >> stubs
> >> > in
> >> > > > > contrib.
> >> > > > >
> >> > > > > For the operators not considered stable, I would prefer we
> either
> >> > leave
> >> > > > the
> >> > > > > current package path as is or if we change the package path then
> >> > create
> >> > > > the
> >> > > > > b/w stub as I am not sure how widely they are in use (so, in
> >> essence,
> >> > > > > preserve semantic versioning). It would be good if there is a
> >> > followup
> >> > > > JIRA
> >> > > > > that takes a look at what other operators can be moved to
> contrib
> >> > with
> >> > > > the
> >> > > > > advent of the newer frameworks and understanding.
> >> > > > >
> >> > > > > Thanks
> >> > > > >
> >> > > > > On Fri, Jul 14, 2017 at 9:44 AM, Thomas Weise <th...@apache.org>
> >> > wrote:
> >> > > > >
> >> > > > > > Most of the operators evolve, as is quite clearly indicated by
> >> > > > @Evolving
> >> > > > > > annotations. So while perhaps 0.x.x would be a more
> appropriate
> >> > > version
> >> > > > > > number, I don't think you can change that.
> >> > > > > >
> >> > > > > > Thomas
> >> > > > > >
> >> > > > > > On Fri, Jul 14, 2017 at 9:35 AM, Vlad Rozov <
> >> > v.rozov@datatorrent.com
> >> > > >
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > > > If entire library is not stable, its version should be 0.x.x
> >> and
> >> > > > there
> >> > > > > > > should not be any semantic versioning enabled or implied. It
> >> > > evolves.
> >> > > > > If
> >> > > > > > > some operators are stable as 3.8.x version suggests, the
> >> library
> >> > > > should
> >> > > > > > > follow semantic versioning and it is not OK to make a major
> >> > change
> >> > > > that
> >> > > > > > > breaks semantic versioning across the entire library. It is
> >> not a
> >> > > > > finding
> >> > > > > > > an excuse not to make a change. For me semantic versioning
> and
> >> > > > backward
> >> > > > > > > compatibility is more important than a proper package name.
> >> > > > > > >
> >> > > > > > > Thank you,
> >> > > > > > >
> >> > > > > > > Vlad
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > On 7/14/17 09:11, Thomas Weise wrote:
> >> > > > > > >
> >> > > > > > >> Semantic versioning makes sense when you have a stable
> >> baseline.
> >> > > As
> >> > > > > long
> >> > > > > > >> as
> >> > > > > > >> frequent fixes need to be made to address basic issues, it
> >> makes
> >> > > no
> >> > > > > > sense
> >> > > > > > >> to declare operators stable, which is why they are marked
> >> > > evolving.
> >> > > > > > >>
> >> > > > > > >> Instead of finding reasons to avoid changes that should
> have
> >> > been
> >> > > > > made a
> >> > > > > > >> long time ago, why not discuss what a "stable operator" is
> >> and
> >> > > which
> >> > > > > > ones
> >> > > > > > >> deserve to be in that category. Those are the ones that
> will
> >> get
> >> > > the
> >> > > > > > >> backward compatibility stub.
> >> > > > > > >>
> >> > > > > > >> Thanks,
> >> > > > > > >> Thomas
> >> > > > > > >>
> >> > > > > > >>
> >> > > > > > >> On Fri, Jul 14, 2017 at 8:34 AM, Vlad Rozov <
> >> > > > v.rozov@datatorrent.com>
> >> > > > > > >> wrote:
> >> > > > > > >>
> >> > > > > > >> My preference is to agree that the next Malhar release is
> >> 4.0.0
> >> > > and
> >> > > > > make
> >> > > > > > >>> the proposed changes when 3.8.0 goes out. Otherwise why to
> >> keep
> >> > > > > > semantic
> >> > > > > > >>> versioning checks on Malhar in case there is no version
> >> > > > > compatibility.
> >> > > > > > >>>
> >> > > > > > >>> Thank you,
> >> > > > > > >>>
> >> > > > > > >>> Vlad
> >> > > > > > >>>
> >> > > > > > >>>
> >> > > > > > >>> On 7/14/17 08:11, Thomas Weise wrote:
> >> > > > > > >>>
> >> > > > > > >>> next release is 3.8.0
> >> > > > > > >>>>
> >> > > > > > >>>> On Fri, Jul 14, 2017 at 7:55 AM, Vlad Rozov <
> >> > > > > v.rozov@datatorrent.com>
> >> > > > > > >>>> wrote:
> >> > > > > > >>>>
> >> > > > > > >>>> next release - 3.9.0 or 4.0.0?
> >> > > > > > >>>>
> >> > > > > > >>>>> Thank you,
> >> > > > > > >>>>>
> >> > > > > > >>>>> Vlad
> >> > > > > > >>>>>
> >> > > > > > >>>>> On 7/13/17 22:25, Thomas Weise wrote:
> >> > > > > > >>>>>
> >> > > > > > >>>>> It is time to resurrect this thread and get going with
> the
> >> > > work.
> >> > > > > > >>>>>
> >> > > > > > >>>>>> For the next release, I will sign up to do the package
> >> move
> >> > in
> >> > > > > > Malhar:
> >> > > > > > >>>>>>
> >> > > > > > >>>>>> https://issues.apache.org/jira/browse/APEXMALHAR-2517
> >> > > > > > >>>>>>
> >> > > > > > >>>>>> In general this will be straightforward; most classes
> in
> >> > > Malhar
> >> > > > > are
> >> > > > > > >>>>>> marked
> >> > > > > > >>>>>> evolving and it is trivial for users to change import
> >> > > > statements.
> >> > > > > > >>>>>> However,
> >> > > > > > >>>>>> I would suggest to discuss if there are selected
> >> operators
> >> > > that
> >> > > > > are
> >> > > > > > >>>>>> worth
> >> > > > > > >>>>>> keeping a backward compatibility stub in the old
> >> location.
> >> > > > > > >>>>>>
> >> > > > > > >>>>>> Here is my plan:
> >> > > > > > >>>>>>
> >> > > > > > >>>>>> 1. Relocate all classes in *lib* and *contrib* within
> one
> >> > PR -
> >> > > > > this
> >> > > > > > is
> >> > > > > > >>>>>> all
> >> > > > > > >>>>>>
> >> > > > > > >>>>>> IDE automated work
> >> > > > > > >>>>>> 2. Add backward compatibility classes, if, any in
> >> separate
> >> > PR
> >> > > > > > >>>>>> 3. Create PR for Megh library to reflect moved classes
> >> > > > > > >>>>>>
> >> > > > > > >>>>>> Thanks,
> >> > > > > > >>>>>> Thomas
> >> > > > > > >>>>>>
> >> > > > > > >>>>>>
> >> > > > > > >>>>>>
> >> > > > > > >>>>>> On Wed, Mar 1, 2017 at 2:24 PM, Pramod Immaneni <
> >> > > > > > >>>>>> pramod@datatorrent.com
> >> > > > > > >>>>>> wrote:
> >> > > > > > >>>>>>
> >> > > > > > >>>>>> Inline
> >> > > > > > >>>>>>
> >> > > > > > >>>>>> On Mon, Feb 27, 2017 at 11:03 PM, Thomas Weise <
> >> > > thw@apache.org>
> >> > > > > > >>>>>>> wrote:
> >> > > > > > >>>>>>>
> >> > > > > > >>>>>>> -->
> >> > > > > > >>>>>>>
> >> > > > > > >>>>>>> On Mon, Feb 27, 2017 at 1:27 PM, Pramod Immaneni <
> >> > > > > > >>>>>>>> pramod@datatorrent.com
> >> > > > > > >>>>>>>>
> >> > > > > > >>>>>>>> wrote:
> >> > > > > > >>>>>>>>
> >> > > > > > >>>>>>>> For malhar, for existing operators, I prefer we do
> >> this as
> >> > > > part
> >> > > > > of
> >> > > > > > >>>>>>>> the
> >> > > > > > >>>>>>>>
> >> > > > > > >>>>>>>> planned refactoring for breaking the monolith modules
> >> into
> >> > > > baby
> >> > > > > > >>>>>>>>>
> >> > > > > > >>>>>>>>> packages
> >> > > > > > >>>>>>>>>
> >> > > > > > >>>>>>>> and would also prefer deprecating the existing
> >> operators
> >> > in
> >> > > > > place.
> >> > > > > > >>>>>>>> Refactor into smaller modules was discussed for
> >> > > malhar-contrib
> >> > > > > and
> >> > > > > > >>>>>>>> given
> >> > > > > > >>>>>>>> the overall state of that module I think it is OK to
> >> defer
> >> > > > > package
> >> > > > > > >>>>>>>>
> >> > > > > > >>>>>>>> renaming
> >> > > > > > >>>>>>>>
> >> > > > > > >>>>>>> there. I do however prefer to see the package rename
> >> > > addressed
> >> > > > > for
> >> > > > > > >>>>>>>
> >> > > > > > >>>>>>>> other
> >> > > > > > >>>>>>>> modules, especially for the main library module.
> >> > > > > > >>>>>>>>
> >> > > > > > >>>>>>>> Should we consider breaking the library into smaller
> >> > modules
> >> > > > as
> >> > > > > > >>>>>>>> well,
> >> > > > > > >>>>>>>>
> >> > > > > > >>>>>>>> the
> >> > > > > > >>>>>>> file/block operators for example probably can be in
> >> their
> >> > own
> >> > > > > > module
> >> > > > > > >>>>>>> from
> >> > > > > > >>>>>>> just an organizational perspective.
> >> > > > > > >>>>>>>
> >> > > > > > >>>>>>>
> >> > > > > > >>>>>>>
> >> > > > > > >>>>>>> This
> >> > > > > > >>>>>>>
> >> > > > > > >>>>>>>> will help us achieve two things. First, the user will
> >> see
> >> > > all
> >> > > > > the
> >> > > > > > >>>>>>>>> new
> >> > > > > > >>>>>>>>> changes at once as opposed to dealing with it twice
> >> (with
> >> > > > > package
> >> > > > > > >>>>>>>>>
> >> > > > > > >>>>>>>>> rename
> >> > > > > > >>>>>>>>>
> >> > > > > > >>>>>>>> and dependency changes) and second it will allow for
> a
> >> > > > smoother
> >> > > > > > >>>>>>>> transition
> >> > > > > > >>>>>>>>
> >> > > > > > >>>>>>>> as the existing code will still work in a deprecated
> >> > state.
> >> > > It
> >> > > > > > will
> >> > > > > > >>>>>>>>
> >> > > > > > >>>>>>>>> also
> >> > > > > > >>>>>>>>>
> >> > > > > > >>>>>>>> give a more consistent structure to malhar. For new
> >> > > operators,
> >> > > > > we
> >> > > > > > >>>>>>>> can
> >> > > > > > >>>>>>>> go
> >> > > > > > >>>>>>>> with the new package path but we need to ensure they
> >> will
> >> > > get
> >> > > > > > moved
> >> > > > > > >>>>>>>> into
> >> > > > > > >>>>>>>> the baby packages as well.
> >> > > > > > >>>>>>>>
> >> > > > > > >>>>>>>> I think existing operators should be renamed so that
> >> git
> >> > > > history
> >> > > > > > >>>>>>>>>
> >> > > > > > >>>>>>>>> remains. A
> >> > > > > > >>>>>>>>
> >> > > > > > >>>>>>> possible solution for backward compatibility could be
> to
> >> > > > > > subsequently
> >> > > > > > >>>>>>>
> >> > > > > > >>>>>>>> add
> >> > > > > > >>>>>>>> empty subclasses in the previous location (for
> existing
> >> > > > concrete
> >> > > > > > >>>>>>>>
> >> > > > > > >>>>>>>> operators
> >> > > > > > >>>>>>>>
> >> > > > > > >>>>>>> that we know are actually in use) to simplify
> migration
> >> for
> >> > > > > users.
> >> > > > > > >>>>>>>
> >> > > > > > >>>>>>>>
> >> > > > > > >>>>>>>> Yes we can do that.
> >> > > > > > >>>>>>>>
> >> > > > > > >>>>>>>> For demos, we can modify the paths as the apps are
> >> > typically
> >> > > > > used
> >> > > > > > >>>>>>>
> >> > > > > > >>>>>>> wholesale
> >> > > > > > >>>>>>>>
> >> > > > > > >>>>>>>> and the interface is typically manual interaction.
> >> > > > > > >>>>>>>>
> >> > > > > > >>>>>>>>> For core, if we are adding new api subsystems, like
> >> the
> >> > > > > launcher
> >> > > > > > >>>>>>>>> api
> >> > > > > > >>>>>>>>> we
> >> > > > > > >>>>>>>>> added recently for example, we can go with new
> package
> >> > path
> >> > > > but
> >> > > > > > if
> >> > > > > > >>>>>>>>> we
> >> > > > > > >>>>>>>>>
> >> > > > > > >>>>>>>>> are
> >> > > > > > >>>>>>>>>
> >> > > > > > >>>>>>>> making incremental additions to existing
> >> functionality, I
> >> > > feel
> >> > > > > it
> >> > > > > > is
> >> > > > > > >>>>>>>> better
> >> > > > > > >>>>>>>>
> >> > > > > > >>>>>>>> to keep it in the same package. I also prefer we keep
> >> the
> >> > > > > package
> >> > > > > > of
> >> > > > > > >>>>>>>>
> >> > > > > > >>>>>>>>> the
> >> > > > > > >>>>>>>>>
> >> > > > > > >>>>>>>> implementation classes consistent with api, for
> >> > > > > understandability
> >> > > > > > >>>>>>>> and
> >> > > > > > >>>>>>>>
> >> > > > > > >>>>>>>> readability of the code. So, for example, we don't
> >> change
> >> > > > > package
> >> > > > > > >>>>>>>>> path
> >> > > > > > >>>>>>>>>
> >> > > > > > >>>>>>>>> of
> >> > > > > > >>>>>>>>>
> >> > > > > > >>>>>>>> LogicalPlan as it is an implementation of DAG. It is
> >> > > > subjective,
> >> > > > > > but
> >> > > > > > >>>>>>>> it
> >> > > > > > >>>>>>>>
> >> > > > > > >>>>>>>> will be good if we can also do the same with classes
> >> > closely
> >> > > > > > related
> >> > > > > > >>>>>>>>> to
> >> > > > > > >>>>>>>>>
> >> > > > > > >>>>>>>>> the
> >> > > > > > >>>>>>>>>
> >> > > > > > >>>>>>>> implementation classes as well. Maybe we can moving
> >> these
> >> > > on a
> >> > > > > > >>>>>>>> package
> >> > > > > > >>>>>>>>
> >> > > > > > >>>>>>>>> by
> >> > > > > > >>>>>>>>>
> >> > > > > > >>>>>>>> package basis, like everything in
> >> > > com.datatorrent.stram.engine
> >> > > > > > could
> >> > > > > > >>>>>>>> be
> >> > > > > > >>>>>>>>
> >> > > > > > >>>>>>>> moved. For completely internal components like buffer
> >> > > server,
> >> > > > we
> >> > > > > > can
> >> > > > > > >>>>>>>>>
> >> > > > > > >>>>>>>>> move
> >> > > > > > >>>>>>>>>
> >> > > > > > >>>>>>>> them wholesale. We can consider moving all api and
> >> > classes,
> >> > > > when
> >> > > > > > we
> >> > > > > > >>>>>>>> go
> >> > > > > > >>>>>>>> to
> >> > > > > > >>>>>>>> next major release but would like to see if we can
> >> find a
> >> > > way
> >> > > > to
> >> > > > > > >>>>>>>> support
> >> > > > > > >>>>>>>> existing api for one more major release in deprecated
> >> > mode.
> >> > > > > > >>>>>>>>
> >> > > > > > >>>>>>>> The point of the major release is to enable backward
> >> > > > > incompatible
> >> > > > > > >>>>>>>>>
> >> > > > > > >>>>>>>>> changes
> >> > > > > > >>>>>>>> and I don't think it is realistic to support the
> >> existing
> >> > > API
> >> > > > > for
> >> > > > > > >>>>>>>> another
> >> > > > > > >>>>>>>> major release. IMO it is also not necessary as most
> >> > existing
> >> > > > > > >>>>>>>> application
> >> > > > > > >>>>>>>> code refers to operators, attributes and the
> >> application
> >> > > > > > interface.
> >> > > > > > >>>>>>>>
> >> > > > > > >>>>>>>> Perhaps
> >> > > > > > >>>>>>>>
> >> > > > > > >>>>>>> it is possible to keep those around as interface
> >> extensions
> >> > > to
> >> > > > > help
> >> > > > > > >>>>>>>
> >> > > > > > >>>>>>>> migration. Custom operators may need to be migrated
> to
> >> > > reflect
> >> > > > > API
> >> > > > > > >>>>>>>>
> >> > > > > > >>>>>>>> changes,
> >> > > > > > >>>>>>>>
> >> > > > > > >>>>>>> and I would consider that a reasonable task for
> operator
> >> > > > > developers
> >> > > > > > >>>>>>> as
> >> > > > > > >>>>>>>
> >> > > > > > >>>>>>>> part
> >> > > > > > >>>>>>>>
> >> > > > > > >>>>>>> of a major upgrade.
> >> > > > > > >>>>>>>
> >> > > > > > >>>>>>>> It would be good if we can keep them as deprecated
> >> > interface
> >> > > > > > >>>>>>>> extensions
> >> > > > > > >>>>>>>>
> >> > > > > > >>>>>>>> for
> >> > > > > > >>>>>>> one release to provide a smoother transition.
> >> > > > > > >>>>>>>
> >> > > > > > >>>>>>>
> >> > > > > > >>>>>>> API and implementation in engine are kept separate
> >> > > > intentionally.
> >> > > > > > >>>>>>> They
> >> > > > > > >>>>>>>
> >> > > > > > >>>>>>> reside in different packages today, so I don't see a
> >> > problem
> >> > > > > > renaming
> >> > > > > > >>>>>>>> com.datatorrent.stram.engine as you say, even when
> the
> >> API
> >> > > > > cannot
> >> > > > > > be
> >> > > > > > >>>>>>>> touched right away.
> >> > > > > > >>>>>>>>
> >> > > > > > >>>>>>>> They are different packages but sharing a common
> prefix
> >> > with
> >> > > > api
> >> > > > > > >>>>>>>> will
> >> > > > > > >>>>>>>> be
> >> > > > > > >>>>>>>>
> >> > > > > > >>>>>>>> helpful to someone new to codebase in terms of
> >> > readability.
> >> > > > Not
> >> > > > > a
> >> > > > > > >>>>>>> big
> >> > > > > > >>>>>>> deal
> >> > > > > > >>>>>>> and can be changed.
> >> > > > > > >>>>>>>
> >> > > > > > >>>>>>>
> >> > > > > > >>>>>>> Thanks
> >> > > > > > >>>>>>>
> >> > > > > > >>>>>>> On Mon, Feb 27, 2017 at 7:39 AM, Thomas Weise <
> >> > > thw@apache.org>
> >> > > > > > >>>>>>>> wrote:
> >> > > > > > >>>>>>>>
> >> > > > > > >>>>>>>>> Hi,
> >> > > > > > >>>>>>>>>
> >> > > > > > >>>>>>>>> This topic has come up on several PRs and I think it
> >> > > > warrants a
> >> > > > > > >>>>>>>>>>
> >> > > > > > >>>>>>>>>> broader
> >> > > > > > >>>>>>>>>>
> >> > > > > > >>>>>>>>> discussion.
> >> > > > > > >>>>>>>>
> >> > > > > > >>>>>>>> At the time of incubation, the decision was to defer
> >> > change
> >> > > of
> >> > > > > > Java
> >> > > > > > >>>>>>>>>
> >> > > > > > >>>>>>>>>> packages from com.datatorrent to org.apache.apex
> till
> >> > next
> >> > > > > major
> >> > > > > > >>>>>>>>>>
> >> > > > > > >>>>>>>>>> release
> >> > > > > > >>>>>>>>>>
> >> > > > > > >>>>>>>>> to
> >> > > > > > >>>>>>>>>
> >> > > > > > >>>>>>>>> ensure backward compatibility for users.
> >> > > > > > >>>>>>>>>
> >> > > > > > >>>>>>>>>> Unfortunately that has lead to some confusion, as
> >> > > > contributors
> >> > > > > > >>>>>>>>>>
> >> > > > > > >>>>>>>>>> continue
> >> > > > > > >>>>>>>>>>
> >> > > > > > >>>>>>>>> to
> >> > > > > > >>>>>>>>
> >> > > > > > >>>>>>>> add new code under legacy packages.
> >> > > > > > >>>>>>>>>
> >> > > > > > >>>>>>>>>> It is also a wider issue that examples for using
> Apex
> >> > > > continue
> >> > > > > > to
> >> > > > > > >>>>>>>>>>
> >> > > > > > >>>>>>>>>> refer
> >> > > > > > >>>>>>>>>>
> >> > > > > > >>>>>>>>> to
> >> > > > > > >>>>>>>>
> >> > > > > > >>>>>>>> com.datatorrent packages, nearly one year after
> >> > graduation.
> >> > > > More
> >> > > > > > and
> >> > > > > > >>>>>>>>>
> >> > > > > > >>>>>>>>>> more
> >> > > > > > >>>>>>>>>>
> >> > > > > > >>>>>>>>> user code is being built on top of something that
> >> needs
> >> > to
> >> > > > > > change,
> >> > > > > > >>>>>>>>> the
> >> > > > > > >>>>>>>>>
> >> > > > > > >>>>>>>>> can
> >> > > > > > >>>>>>>>
> >> > > > > > >>>>>>>> is being kicked down the road and users will face
> more
> >> > > changes
> >> > > > > > >>>>>>>>> later.
> >> > > > > > >>>>>>>>>
> >> > > > > > >>>>>>>>>> I would like to propose the following:
> >> > > > > > >>>>>>>>>>
> >> > > > > > >>>>>>>>>> 1. All new code has to be submitted under
> >> > org.apache.apex
> >> > > > > > packages
> >> > > > > > >>>>>>>>>>
> >> > > > > > >>>>>>>>>> 2. Not all code is under backward compatibility
> >> > > restriction
> >> > > > > and
> >> > > > > > in
> >> > > > > > >>>>>>>>>>
> >> > > > > > >>>>>>>>>> those
> >> > > > > > >>>>>>>>>>
> >> > > > > > >>>>>>>>> cases we can rename the packages right away.
> Examples:
> >> > > buffer
> >> > > > > > >>>>>>>>> server,
> >> > > > > > >>>>>>>>>
> >> > > > > > >>>>>>>>> engine, demos/examples, benchmarks
> >> > > > > > >>>>>>>>>>
> >> > > > > > >>>>>>>>>> 3. Discuss when the core API and operators can be
> >> > changed.
> >> > > > For
> >> > > > > > >>>>>>>>>>
> >> > > > > > >>>>>>>>>> operators
> >> > > > > > >>>>>>>>>>
> >> > > > > > >>>>>>>>> we
> >> > > > > > >>>>>>>>>
> >> > > > > > >>>>>>>>> have a bit more freedom to do changes before a major
> >> > > release
> >> > > > as
> >> > > > > > >>>>>>>>> most
> >> > > > > > >>>>>>>>>
> >> > > > > > >>>>>>>>>> of
> >> > > > > > >>>>>>>>>>
> >> > > > > > >>>>>>>>> them are marked @Evolving and users have the ability
> >> to
> >> > > > > continue
> >> > > > > > >>>>>>>>
> >> > > > > > >>>>>>>> using
> >> > > > > > >>>>>>>>>
> >> > > > > > >>>>>>>>> prior version of Malhar with newer engine due to
> >> engine
> >> > > > > backward
> >> > > > > > >>>>>>>>
> >> > > > > > >>>>>>>> compatibility guarantee.
> >> > > > > > >>>>>>>>>
> >> > > > > > >>>>>>>>>> Thanks,
> >> > > > > > >>>>>>>>>> Thomas
> >> > > > > > >>>>>>>>>>
> >> > > > > > >>>>>>>>>>
> >> > > > > > >>>>>>>>>>
> >> > > > > > >>>>>>>>>>
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>
Re: Java packages: legacy -> org.apache.apex
Posted by Thomas Weise <th...@apache.org>.
Hi,
I opened following PRs for the package change:
https://github.com/apache/apex-malhar/pull/662
Moves all classes with history retained (hence 2 commits). Also contains
checkstyle and other mechanical changes.
https://github.com/apache/apex-malhar/pull/664
Adds backward compatibility jar.
Once above PRs are merged the new artifact can be deployed and introduced
as dependency in malhar-library.
Please review.
Thanks,
Thomas
On Sun, Jul 16, 2017 at 7:04 AM, Thomas Weise <th...@apache.org> wrote:
> My original list of work items contained the b/w compatibility aspect, I
> don't think there should be any confusion of whether it will be covered
> here or not.
>
> The proposed shading will provide the old classes and they will be frozen
> as of release 3.7. That's the same as making a copy of the code and never
> again making changes to the original classes. This cannot be accomplished
> by using the older 3.7 release in your project because you cannot use 2
> different versions of Malhar in parallel unless you apply shading.
>
> The shaded artifact will only expose the com.datatorrent classes, and they
> will be self-contained as the rest of the classes that they may depend on
> are shaded. The shaded artifact does not evolve, there are not more changes
> to com.datatorrent classes after they are relocated in master.
>
> Thanks,
> Thomas
>
>
> On Sun, Jul 16, 2017 at 2:00 AM, Pramod Immaneni <pr...@datatorrent.com>
> wrote:
>
>> I don't think we can limit the topic strictly to relocation without having
>> a good b/w compatibility story or at least one that goes far enough.
>>
>> The shading idea sounds interesting. Why not let the shaded version move
>> forward with each release till we hit a major release. If it is going to
>> remain pegged at 3.7.0, why shade in the first place as the regular 3.7.0
>> release would do the same job and it would be same as the loss of b/w
>> compatibility with newer releases.
>>
>> Thanks
>>
>> On Sat, Jul 15, 2017 at 7:57 AM, Thomas Weise <th...@apache.org> wrote:
>>
>> > Discussing what in the future might become stable needs to be a separate
>> > thread, it will be a much bigger discussion.
>> >
>> > The topic here is to relocate the packages. With a few exceptions
>> > relocation won't affect the semantic versioning. Semantic versioning is
>> > essentially not effective for Malhar because almost everything is
>> @Evolving
>> > (and there are reasons for that.. -> separate topic)
>> >
>> > I don't really like the idea of creating bw compatibility stubs for the
>> > follow up PR. It creates even more clutter in the source tree than there
>> > already is and so here is an alternative suggestion:
>> >
>> > https://github.com/tweise/apex-malhar/blob/malhar37-
>> > compat/shaded-malhar37/pom.xml
>> >
>> > Create a shaded artifact that provides the old com.datatorrent.*
>> classes as
>> > of release 3.7. Users can include that artifact if they don't want to
>> > change import statements. At the same time they have an incentive to
>> switch
>> > to the relocated classes to take advantage of bug fixes and new
>> > functionality.
>> >
>> > I will work on the first PR that does the relocate. In the meantime, we
>> can
>> > finalize what backward compatibility support we want to provide and how.
>> >
>> > Thanks,
>> > Thomas
>> >
>> >
>> >
>> >
>> > On Fri, Jul 14, 2017 at 11:33 AM, Pramod Immaneni <
>> pramod@datatorrent.com>
>> > wrote:
>> >
>> > > How about coming up with a list of @Evolving operators that we would
>> like
>> > > to see in the eventual stable list and move those along with the not
>> > > @Evolving ones in org.apache.apex with b/w stubs and leave the rest as
>> > they
>> > > are. Then have a follow up JIRA for the rest to be moved over to
>> contrib
>> > > and be deprecated.
>> > >
>> > > Thanks
>> > >
>> > > On Fri, Jul 14, 2017 at 10:37 AM, Thomas Weise <
>> thomas.weise@gmail.com>
>> > > wrote:
>> > >
>> > > > We need to keep the discussion here on topic, if other things are
>> piled
>> > > on
>> > > > top then nothing gets done.
>> > > >
>> > > > Most operators today are not stable, they are @Evolving. So perhaps
>> it
>> > is
>> > > > necessary to have a separate discussion about that, outside of this
>> > > thread.
>> > > > My guess is that there are only a few operators that we could
>> declare
>> > > > stable. Specifically, under contrib the closest one might have been
>> > > Kafka,
>> > > > but that is already superseded by the newer versions.
>> > > >
>> > > > Thomas
>> > > >
>> > > >
>> > > > On Fri, Jul 14, 2017 at 10:21 AM, Pramod Immaneni <
>> > > pramod@datatorrent.com>
>> > > > wrote:
>> > > >
>> > > > > We had a discussion a while back, agreed to relegate non-stable
>> and
>> > > > > experimental operators to contrib and also added this to
>> contribution
>> > > > > guidelines. We aexecuted on this and cleaned up the repo by moving
>> > > > > operators deemed non-suitable for production use at that time to
>> > > contrib.
>> > > > > So I wouldn't say the operators that are at the top level today or
>> > the
>> > > > ones
>> > > > > in library are 0.x.x quality. Granted, we may need to do one more
>> > pass
>> > > as
>> > > > > some of the operators may no longer be considered the right
>> > > > implementations
>> > > > > with the advent of the windowed operator.
>> > > > >
>> > > > > Since contrib used to be the place that housed operators that
>> > required
>> > > > > third party libraries, there are some production quality
>> operators in
>> > > > there
>> > > > > that need to be pulled out into top level modules.
>> > > > >
>> > > > > I think we are in agreement that for operators that we consider
>> > stable,
>> > > > we
>> > > > > should provide a b/w stub. I would add that we strongly consider
>> > > creating
>> > > > > the org.apache.apex counterparts of any stable operators that are
>> in
>> > > > > contrib out in top level modules and have the com.datatorrent
>> stubs
>> > in
>> > > > > contrib.
>> > > > >
>> > > > > For the operators not considered stable, I would prefer we either
>> > leave
>> > > > the
>> > > > > current package path as is or if we change the package path then
>> > create
>> > > > the
>> > > > > b/w stub as I am not sure how widely they are in use (so, in
>> essence,
>> > > > > preserve semantic versioning). It would be good if there is a
>> > followup
>> > > > JIRA
>> > > > > that takes a look at what other operators can be moved to contrib
>> > with
>> > > > the
>> > > > > advent of the newer frameworks and understanding.
>> > > > >
>> > > > > Thanks
>> > > > >
>> > > > > On Fri, Jul 14, 2017 at 9:44 AM, Thomas Weise <th...@apache.org>
>> > wrote:
>> > > > >
>> > > > > > Most of the operators evolve, as is quite clearly indicated by
>> > > > @Evolving
>> > > > > > annotations. So while perhaps 0.x.x would be a more appropriate
>> > > version
>> > > > > > number, I don't think you can change that.
>> > > > > >
>> > > > > > Thomas
>> > > > > >
>> > > > > > On Fri, Jul 14, 2017 at 9:35 AM, Vlad Rozov <
>> > v.rozov@datatorrent.com
>> > > >
>> > > > > > wrote:
>> > > > > >
>> > > > > > > If entire library is not stable, its version should be 0.x.x
>> and
>> > > > there
>> > > > > > > should not be any semantic versioning enabled or implied. It
>> > > evolves.
>> > > > > If
>> > > > > > > some operators are stable as 3.8.x version suggests, the
>> library
>> > > > should
>> > > > > > > follow semantic versioning and it is not OK to make a major
>> > change
>> > > > that
>> > > > > > > breaks semantic versioning across the entire library. It is
>> not a
>> > > > > finding
>> > > > > > > an excuse not to make a change. For me semantic versioning and
>> > > > backward
>> > > > > > > compatibility is more important than a proper package name.
>> > > > > > >
>> > > > > > > Thank you,
>> > > > > > >
>> > > > > > > Vlad
>> > > > > > >
>> > > > > > >
>> > > > > > > On 7/14/17 09:11, Thomas Weise wrote:
>> > > > > > >
>> > > > > > >> Semantic versioning makes sense when you have a stable
>> baseline.
>> > > As
>> > > > > long
>> > > > > > >> as
>> > > > > > >> frequent fixes need to be made to address basic issues, it
>> makes
>> > > no
>> > > > > > sense
>> > > > > > >> to declare operators stable, which is why they are marked
>> > > evolving.
>> > > > > > >>
>> > > > > > >> Instead of finding reasons to avoid changes that should have
>> > been
>> > > > > made a
>> > > > > > >> long time ago, why not discuss what a "stable operator" is
>> and
>> > > which
>> > > > > > ones
>> > > > > > >> deserve to be in that category. Those are the ones that will
>> get
>> > > the
>> > > > > > >> backward compatibility stub.
>> > > > > > >>
>> > > > > > >> Thanks,
>> > > > > > >> Thomas
>> > > > > > >>
>> > > > > > >>
>> > > > > > >> On Fri, Jul 14, 2017 at 8:34 AM, Vlad Rozov <
>> > > > v.rozov@datatorrent.com>
>> > > > > > >> wrote:
>> > > > > > >>
>> > > > > > >> My preference is to agree that the next Malhar release is
>> 4.0.0
>> > > and
>> > > > > make
>> > > > > > >>> the proposed changes when 3.8.0 goes out. Otherwise why to
>> keep
>> > > > > > semantic
>> > > > > > >>> versioning checks on Malhar in case there is no version
>> > > > > compatibility.
>> > > > > > >>>
>> > > > > > >>> Thank you,
>> > > > > > >>>
>> > > > > > >>> Vlad
>> > > > > > >>>
>> > > > > > >>>
>> > > > > > >>> On 7/14/17 08:11, Thomas Weise wrote:
>> > > > > > >>>
>> > > > > > >>> next release is 3.8.0
>> > > > > > >>>>
>> > > > > > >>>> On Fri, Jul 14, 2017 at 7:55 AM, Vlad Rozov <
>> > > > > v.rozov@datatorrent.com>
>> > > > > > >>>> wrote:
>> > > > > > >>>>
>> > > > > > >>>> next release - 3.9.0 or 4.0.0?
>> > > > > > >>>>
>> > > > > > >>>>> Thank you,
>> > > > > > >>>>>
>> > > > > > >>>>> Vlad
>> > > > > > >>>>>
>> > > > > > >>>>> On 7/13/17 22:25, Thomas Weise wrote:
>> > > > > > >>>>>
>> > > > > > >>>>> It is time to resurrect this thread and get going with the
>> > > work.
>> > > > > > >>>>>
>> > > > > > >>>>>> For the next release, I will sign up to do the package
>> move
>> > in
>> > > > > > Malhar:
>> > > > > > >>>>>>
>> > > > > > >>>>>> https://issues.apache.org/jira/browse/APEXMALHAR-2517
>> > > > > > >>>>>>
>> > > > > > >>>>>> In general this will be straightforward; most classes in
>> > > Malhar
>> > > > > are
>> > > > > > >>>>>> marked
>> > > > > > >>>>>> evolving and it is trivial for users to change import
>> > > > statements.
>> > > > > > >>>>>> However,
>> > > > > > >>>>>> I would suggest to discuss if there are selected
>> operators
>> > > that
>> > > > > are
>> > > > > > >>>>>> worth
>> > > > > > >>>>>> keeping a backward compatibility stub in the old
>> location.
>> > > > > > >>>>>>
>> > > > > > >>>>>> Here is my plan:
>> > > > > > >>>>>>
>> > > > > > >>>>>> 1. Relocate all classes in *lib* and *contrib* within one
>> > PR -
>> > > > > this
>> > > > > > is
>> > > > > > >>>>>> all
>> > > > > > >>>>>>
>> > > > > > >>>>>> IDE automated work
>> > > > > > >>>>>> 2. Add backward compatibility classes, if, any in
>> separate
>> > PR
>> > > > > > >>>>>> 3. Create PR for Megh library to reflect moved classes
>> > > > > > >>>>>>
>> > > > > > >>>>>> Thanks,
>> > > > > > >>>>>> Thomas
>> > > > > > >>>>>>
>> > > > > > >>>>>>
>> > > > > > >>>>>>
>> > > > > > >>>>>> On Wed, Mar 1, 2017 at 2:24 PM, Pramod Immaneni <
>> > > > > > >>>>>> pramod@datatorrent.com
>> > > > > > >>>>>> wrote:
>> > > > > > >>>>>>
>> > > > > > >>>>>> Inline
>> > > > > > >>>>>>
>> > > > > > >>>>>> On Mon, Feb 27, 2017 at 11:03 PM, Thomas Weise <
>> > > thw@apache.org>
>> > > > > > >>>>>>> wrote:
>> > > > > > >>>>>>>
>> > > > > > >>>>>>> -->
>> > > > > > >>>>>>>
>> > > > > > >>>>>>> On Mon, Feb 27, 2017 at 1:27 PM, Pramod Immaneni <
>> > > > > > >>>>>>>> pramod@datatorrent.com
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>> wrote:
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>> For malhar, for existing operators, I prefer we do
>> this as
>> > > > part
>> > > > > of
>> > > > > > >>>>>>>> the
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>> planned refactoring for breaking the monolith modules
>> into
>> > > > baby
>> > > > > > >>>>>>>>>
>> > > > > > >>>>>>>>> packages
>> > > > > > >>>>>>>>>
>> > > > > > >>>>>>>> and would also prefer deprecating the existing
>> operators
>> > in
>> > > > > place.
>> > > > > > >>>>>>>> Refactor into smaller modules was discussed for
>> > > malhar-contrib
>> > > > > and
>> > > > > > >>>>>>>> given
>> > > > > > >>>>>>>> the overall state of that module I think it is OK to
>> defer
>> > > > > package
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>> renaming
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>> there. I do however prefer to see the package rename
>> > > addressed
>> > > > > for
>> > > > > > >>>>>>>
>> > > > > > >>>>>>>> other
>> > > > > > >>>>>>>> modules, especially for the main library module.
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>> Should we consider breaking the library into smaller
>> > modules
>> > > > as
>> > > > > > >>>>>>>> well,
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>> the
>> > > > > > >>>>>>> file/block operators for example probably can be in
>> their
>> > own
>> > > > > > module
>> > > > > > >>>>>>> from
>> > > > > > >>>>>>> just an organizational perspective.
>> > > > > > >>>>>>>
>> > > > > > >>>>>>>
>> > > > > > >>>>>>>
>> > > > > > >>>>>>> This
>> > > > > > >>>>>>>
>> > > > > > >>>>>>>> will help us achieve two things. First, the user will
>> see
>> > > all
>> > > > > the
>> > > > > > >>>>>>>>> new
>> > > > > > >>>>>>>>> changes at once as opposed to dealing with it twice
>> (with
>> > > > > package
>> > > > > > >>>>>>>>>
>> > > > > > >>>>>>>>> rename
>> > > > > > >>>>>>>>>
>> > > > > > >>>>>>>> and dependency changes) and second it will allow for a
>> > > > smoother
>> > > > > > >>>>>>>> transition
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>> as the existing code will still work in a deprecated
>> > state.
>> > > It
>> > > > > > will
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>>> also
>> > > > > > >>>>>>>>>
>> > > > > > >>>>>>>> give a more consistent structure to malhar. For new
>> > > operators,
>> > > > > we
>> > > > > > >>>>>>>> can
>> > > > > > >>>>>>>> go
>> > > > > > >>>>>>>> with the new package path but we need to ensure they
>> will
>> > > get
>> > > > > > moved
>> > > > > > >>>>>>>> into
>> > > > > > >>>>>>>> the baby packages as well.
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>> I think existing operators should be renamed so that
>> git
>> > > > history
>> > > > > > >>>>>>>>>
>> > > > > > >>>>>>>>> remains. A
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>> possible solution for backward compatibility could be to
>> > > > > > subsequently
>> > > > > > >>>>>>>
>> > > > > > >>>>>>>> add
>> > > > > > >>>>>>>> empty subclasses in the previous location (for existing
>> > > > concrete
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>> operators
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>> that we know are actually in use) to simplify migration
>> for
>> > > > > users.
>> > > > > > >>>>>>>
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>> Yes we can do that.
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>> For demos, we can modify the paths as the apps are
>> > typically
>> > > > > used
>> > > > > > >>>>>>>
>> > > > > > >>>>>>> wholesale
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>> and the interface is typically manual interaction.
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>>> For core, if we are adding new api subsystems, like
>> the
>> > > > > launcher
>> > > > > > >>>>>>>>> api
>> > > > > > >>>>>>>>> we
>> > > > > > >>>>>>>>> added recently for example, we can go with new package
>> > path
>> > > > but
>> > > > > > if
>> > > > > > >>>>>>>>> we
>> > > > > > >>>>>>>>>
>> > > > > > >>>>>>>>> are
>> > > > > > >>>>>>>>>
>> > > > > > >>>>>>>> making incremental additions to existing
>> functionality, I
>> > > feel
>> > > > > it
>> > > > > > is
>> > > > > > >>>>>>>> better
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>> to keep it in the same package. I also prefer we keep
>> the
>> > > > > package
>> > > > > > of
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>>> the
>> > > > > > >>>>>>>>>
>> > > > > > >>>>>>>> implementation classes consistent with api, for
>> > > > > understandability
>> > > > > > >>>>>>>> and
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>> readability of the code. So, for example, we don't
>> change
>> > > > > package
>> > > > > > >>>>>>>>> path
>> > > > > > >>>>>>>>>
>> > > > > > >>>>>>>>> of
>> > > > > > >>>>>>>>>
>> > > > > > >>>>>>>> LogicalPlan as it is an implementation of DAG. It is
>> > > > subjective,
>> > > > > > but
>> > > > > > >>>>>>>> it
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>> will be good if we can also do the same with classes
>> > closely
>> > > > > > related
>> > > > > > >>>>>>>>> to
>> > > > > > >>>>>>>>>
>> > > > > > >>>>>>>>> the
>> > > > > > >>>>>>>>>
>> > > > > > >>>>>>>> implementation classes as well. Maybe we can moving
>> these
>> > > on a
>> > > > > > >>>>>>>> package
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>>> by
>> > > > > > >>>>>>>>>
>> > > > > > >>>>>>>> package basis, like everything in
>> > > com.datatorrent.stram.engine
>> > > > > > could
>> > > > > > >>>>>>>> be
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>> moved. For completely internal components like buffer
>> > > server,
>> > > > we
>> > > > > > can
>> > > > > > >>>>>>>>>
>> > > > > > >>>>>>>>> move
>> > > > > > >>>>>>>>>
>> > > > > > >>>>>>>> them wholesale. We can consider moving all api and
>> > classes,
>> > > > when
>> > > > > > we
>> > > > > > >>>>>>>> go
>> > > > > > >>>>>>>> to
>> > > > > > >>>>>>>> next major release but would like to see if we can
>> find a
>> > > way
>> > > > to
>> > > > > > >>>>>>>> support
>> > > > > > >>>>>>>> existing api for one more major release in deprecated
>> > mode.
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>> The point of the major release is to enable backward
>> > > > > incompatible
>> > > > > > >>>>>>>>>
>> > > > > > >>>>>>>>> changes
>> > > > > > >>>>>>>> and I don't think it is realistic to support the
>> existing
>> > > API
>> > > > > for
>> > > > > > >>>>>>>> another
>> > > > > > >>>>>>>> major release. IMO it is also not necessary as most
>> > existing
>> > > > > > >>>>>>>> application
>> > > > > > >>>>>>>> code refers to operators, attributes and the
>> application
>> > > > > > interface.
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>> Perhaps
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>> it is possible to keep those around as interface
>> extensions
>> > > to
>> > > > > help
>> > > > > > >>>>>>>
>> > > > > > >>>>>>>> migration. Custom operators may need to be migrated to
>> > > reflect
>> > > > > API
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>> changes,
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>> and I would consider that a reasonable task for operator
>> > > > > developers
>> > > > > > >>>>>>> as
>> > > > > > >>>>>>>
>> > > > > > >>>>>>>> part
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>> of a major upgrade.
>> > > > > > >>>>>>>
>> > > > > > >>>>>>>> It would be good if we can keep them as deprecated
>> > interface
>> > > > > > >>>>>>>> extensions
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>> for
>> > > > > > >>>>>>> one release to provide a smoother transition.
>> > > > > > >>>>>>>
>> > > > > > >>>>>>>
>> > > > > > >>>>>>> API and implementation in engine are kept separate
>> > > > intentionally.
>> > > > > > >>>>>>> They
>> > > > > > >>>>>>>
>> > > > > > >>>>>>> reside in different packages today, so I don't see a
>> > problem
>> > > > > > renaming
>> > > > > > >>>>>>>> com.datatorrent.stram.engine as you say, even when the
>> API
>> > > > > cannot
>> > > > > > be
>> > > > > > >>>>>>>> touched right away.
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>> They are different packages but sharing a common prefix
>> > with
>> > > > api
>> > > > > > >>>>>>>> will
>> > > > > > >>>>>>>> be
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>> helpful to someone new to codebase in terms of
>> > readability.
>> > > > Not
>> > > > > a
>> > > > > > >>>>>>> big
>> > > > > > >>>>>>> deal
>> > > > > > >>>>>>> and can be changed.
>> > > > > > >>>>>>>
>> > > > > > >>>>>>>
>> > > > > > >>>>>>> Thanks
>> > > > > > >>>>>>>
>> > > > > > >>>>>>> On Mon, Feb 27, 2017 at 7:39 AM, Thomas Weise <
>> > > thw@apache.org>
>> > > > > > >>>>>>>> wrote:
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>>> Hi,
>> > > > > > >>>>>>>>>
>> > > > > > >>>>>>>>> This topic has come up on several PRs and I think it
>> > > > warrants a
>> > > > > > >>>>>>>>>>
>> > > > > > >>>>>>>>>> broader
>> > > > > > >>>>>>>>>>
>> > > > > > >>>>>>>>> discussion.
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>> At the time of incubation, the decision was to defer
>> > change
>> > > of
>> > > > > > Java
>> > > > > > >>>>>>>>>
>> > > > > > >>>>>>>>>> packages from com.datatorrent to org.apache.apex till
>> > next
>> > > > > major
>> > > > > > >>>>>>>>>>
>> > > > > > >>>>>>>>>> release
>> > > > > > >>>>>>>>>>
>> > > > > > >>>>>>>>> to
>> > > > > > >>>>>>>>>
>> > > > > > >>>>>>>>> ensure backward compatibility for users.
>> > > > > > >>>>>>>>>
>> > > > > > >>>>>>>>>> Unfortunately that has lead to some confusion, as
>> > > > contributors
>> > > > > > >>>>>>>>>>
>> > > > > > >>>>>>>>>> continue
>> > > > > > >>>>>>>>>>
>> > > > > > >>>>>>>>> to
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>> add new code under legacy packages.
>> > > > > > >>>>>>>>>
>> > > > > > >>>>>>>>>> It is also a wider issue that examples for using Apex
>> > > > continue
>> > > > > > to
>> > > > > > >>>>>>>>>>
>> > > > > > >>>>>>>>>> refer
>> > > > > > >>>>>>>>>>
>> > > > > > >>>>>>>>> to
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>> com.datatorrent packages, nearly one year after
>> > graduation.
>> > > > More
>> > > > > > and
>> > > > > > >>>>>>>>>
>> > > > > > >>>>>>>>>> more
>> > > > > > >>>>>>>>>>
>> > > > > > >>>>>>>>> user code is being built on top of something that
>> needs
>> > to
>> > > > > > change,
>> > > > > > >>>>>>>>> the
>> > > > > > >>>>>>>>>
>> > > > > > >>>>>>>>> can
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>> is being kicked down the road and users will face more
>> > > changes
>> > > > > > >>>>>>>>> later.
>> > > > > > >>>>>>>>>
>> > > > > > >>>>>>>>>> I would like to propose the following:
>> > > > > > >>>>>>>>>>
>> > > > > > >>>>>>>>>> 1. All new code has to be submitted under
>> > org.apache.apex
>> > > > > > packages
>> > > > > > >>>>>>>>>>
>> > > > > > >>>>>>>>>> 2. Not all code is under backward compatibility
>> > > restriction
>> > > > > and
>> > > > > > in
>> > > > > > >>>>>>>>>>
>> > > > > > >>>>>>>>>> those
>> > > > > > >>>>>>>>>>
>> > > > > > >>>>>>>>> cases we can rename the packages right away. Examples:
>> > > buffer
>> > > > > > >>>>>>>>> server,
>> > > > > > >>>>>>>>>
>> > > > > > >>>>>>>>> engine, demos/examples, benchmarks
>> > > > > > >>>>>>>>>>
>> > > > > > >>>>>>>>>> 3. Discuss when the core API and operators can be
>> > changed.
>> > > > For
>> > > > > > >>>>>>>>>>
>> > > > > > >>>>>>>>>> operators
>> > > > > > >>>>>>>>>>
>> > > > > > >>>>>>>>> we
>> > > > > > >>>>>>>>>
>> > > > > > >>>>>>>>> have a bit more freedom to do changes before a major
>> > > release
>> > > > as
>> > > > > > >>>>>>>>> most
>> > > > > > >>>>>>>>>
>> > > > > > >>>>>>>>>> of
>> > > > > > >>>>>>>>>>
>> > > > > > >>>>>>>>> them are marked @Evolving and users have the ability
>> to
>> > > > > continue
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>> using
>> > > > > > >>>>>>>>>
>> > > > > > >>>>>>>>> prior version of Malhar with newer engine due to
>> engine
>> > > > > backward
>> > > > > > >>>>>>>>
>> > > > > > >>>>>>>> compatibility guarantee.
>> > > > > > >>>>>>>>>
>> > > > > > >>>>>>>>>> Thanks,
>> > > > > > >>>>>>>>>> Thomas
>> > > > > > >>>>>>>>>>
>> > > > > > >>>>>>>>>>
>> > > > > > >>>>>>>>>>
>> > > > > > >>>>>>>>>>
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
>
Re: Java packages: legacy -> org.apache.apex
Posted by Thomas Weise <th...@apache.org>.
My original list of work items contained the b/w compatibility aspect, I
don't think there should be any confusion of whether it will be covered
here or not.
The proposed shading will provide the old classes and they will be frozen
as of release 3.7. That's the same as making a copy of the code and never
again making changes to the original classes. This cannot be accomplished
by using the older 3.7 release in your project because you cannot use 2
different versions of Malhar in parallel unless you apply shading.
The shaded artifact will only expose the com.datatorrent classes, and they
will be self-contained as the rest of the classes that they may depend on
are shaded. The shaded artifact does not evolve, there are not more changes
to com.datatorrent classes after they are relocated in master.
Thanks,
Thomas
On Sun, Jul 16, 2017 at 2:00 AM, Pramod Immaneni <pr...@datatorrent.com>
wrote:
> I don't think we can limit the topic strictly to relocation without having
> a good b/w compatibility story or at least one that goes far enough.
>
> The shading idea sounds interesting. Why not let the shaded version move
> forward with each release till we hit a major release. If it is going to
> remain pegged at 3.7.0, why shade in the first place as the regular 3.7.0
> release would do the same job and it would be same as the loss of b/w
> compatibility with newer releases.
>
> Thanks
>
> On Sat, Jul 15, 2017 at 7:57 AM, Thomas Weise <th...@apache.org> wrote:
>
> > Discussing what in the future might become stable needs to be a separate
> > thread, it will be a much bigger discussion.
> >
> > The topic here is to relocate the packages. With a few exceptions
> > relocation won't affect the semantic versioning. Semantic versioning is
> > essentially not effective for Malhar because almost everything is
> @Evolving
> > (and there are reasons for that.. -> separate topic)
> >
> > I don't really like the idea of creating bw compatibility stubs for the
> > follow up PR. It creates even more clutter in the source tree than there
> > already is and so here is an alternative suggestion:
> >
> > https://github.com/tweise/apex-malhar/blob/malhar37-
> > compat/shaded-malhar37/pom.xml
> >
> > Create a shaded artifact that provides the old com.datatorrent.* classes
> as
> > of release 3.7. Users can include that artifact if they don't want to
> > change import statements. At the same time they have an incentive to
> switch
> > to the relocated classes to take advantage of bug fixes and new
> > functionality.
> >
> > I will work on the first PR that does the relocate. In the meantime, we
> can
> > finalize what backward compatibility support we want to provide and how.
> >
> > Thanks,
> > Thomas
> >
> >
> >
> >
> > On Fri, Jul 14, 2017 at 11:33 AM, Pramod Immaneni <
> pramod@datatorrent.com>
> > wrote:
> >
> > > How about coming up with a list of @Evolving operators that we would
> like
> > > to see in the eventual stable list and move those along with the not
> > > @Evolving ones in org.apache.apex with b/w stubs and leave the rest as
> > they
> > > are. Then have a follow up JIRA for the rest to be moved over to
> contrib
> > > and be deprecated.
> > >
> > > Thanks
> > >
> > > On Fri, Jul 14, 2017 at 10:37 AM, Thomas Weise <thomas.weise@gmail.com
> >
> > > wrote:
> > >
> > > > We need to keep the discussion here on topic, if other things are
> piled
> > > on
> > > > top then nothing gets done.
> > > >
> > > > Most operators today are not stable, they are @Evolving. So perhaps
> it
> > is
> > > > necessary to have a separate discussion about that, outside of this
> > > thread.
> > > > My guess is that there are only a few operators that we could declare
> > > > stable. Specifically, under contrib the closest one might have been
> > > Kafka,
> > > > but that is already superseded by the newer versions.
> > > >
> > > > Thomas
> > > >
> > > >
> > > > On Fri, Jul 14, 2017 at 10:21 AM, Pramod Immaneni <
> > > pramod@datatorrent.com>
> > > > wrote:
> > > >
> > > > > We had a discussion a while back, agreed to relegate non-stable and
> > > > > experimental operators to contrib and also added this to
> contribution
> > > > > guidelines. We aexecuted on this and cleaned up the repo by moving
> > > > > operators deemed non-suitable for production use at that time to
> > > contrib.
> > > > > So I wouldn't say the operators that are at the top level today or
> > the
> > > > ones
> > > > > in library are 0.x.x quality. Granted, we may need to do one more
> > pass
> > > as
> > > > > some of the operators may no longer be considered the right
> > > > implementations
> > > > > with the advent of the windowed operator.
> > > > >
> > > > > Since contrib used to be the place that housed operators that
> > required
> > > > > third party libraries, there are some production quality operators
> in
> > > > there
> > > > > that need to be pulled out into top level modules.
> > > > >
> > > > > I think we are in agreement that for operators that we consider
> > stable,
> > > > we
> > > > > should provide a b/w stub. I would add that we strongly consider
> > > creating
> > > > > the org.apache.apex counterparts of any stable operators that are
> in
> > > > > contrib out in top level modules and have the com.datatorrent stubs
> > in
> > > > > contrib.
> > > > >
> > > > > For the operators not considered stable, I would prefer we either
> > leave
> > > > the
> > > > > current package path as is or if we change the package path then
> > create
> > > > the
> > > > > b/w stub as I am not sure how widely they are in use (so, in
> essence,
> > > > > preserve semantic versioning). It would be good if there is a
> > followup
> > > > JIRA
> > > > > that takes a look at what other operators can be moved to contrib
> > with
> > > > the
> > > > > advent of the newer frameworks and understanding.
> > > > >
> > > > > Thanks
> > > > >
> > > > > On Fri, Jul 14, 2017 at 9:44 AM, Thomas Weise <th...@apache.org>
> > wrote:
> > > > >
> > > > > > Most of the operators evolve, as is quite clearly indicated by
> > > > @Evolving
> > > > > > annotations. So while perhaps 0.x.x would be a more appropriate
> > > version
> > > > > > number, I don't think you can change that.
> > > > > >
> > > > > > Thomas
> > > > > >
> > > > > > On Fri, Jul 14, 2017 at 9:35 AM, Vlad Rozov <
> > v.rozov@datatorrent.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > If entire library is not stable, its version should be 0.x.x
> and
> > > > there
> > > > > > > should not be any semantic versioning enabled or implied. It
> > > evolves.
> > > > > If
> > > > > > > some operators are stable as 3.8.x version suggests, the
> library
> > > > should
> > > > > > > follow semantic versioning and it is not OK to make a major
> > change
> > > > that
> > > > > > > breaks semantic versioning across the entire library. It is
> not a
> > > > > finding
> > > > > > > an excuse not to make a change. For me semantic versioning and
> > > > backward
> > > > > > > compatibility is more important than a proper package name.
> > > > > > >
> > > > > > > Thank you,
> > > > > > >
> > > > > > > Vlad
> > > > > > >
> > > > > > >
> > > > > > > On 7/14/17 09:11, Thomas Weise wrote:
> > > > > > >
> > > > > > >> Semantic versioning makes sense when you have a stable
> baseline.
> > > As
> > > > > long
> > > > > > >> as
> > > > > > >> frequent fixes need to be made to address basic issues, it
> makes
> > > no
> > > > > > sense
> > > > > > >> to declare operators stable, which is why they are marked
> > > evolving.
> > > > > > >>
> > > > > > >> Instead of finding reasons to avoid changes that should have
> > been
> > > > > made a
> > > > > > >> long time ago, why not discuss what a "stable operator" is and
> > > which
> > > > > > ones
> > > > > > >> deserve to be in that category. Those are the ones that will
> get
> > > the
> > > > > > >> backward compatibility stub.
> > > > > > >>
> > > > > > >> Thanks,
> > > > > > >> Thomas
> > > > > > >>
> > > > > > >>
> > > > > > >> On Fri, Jul 14, 2017 at 8:34 AM, Vlad Rozov <
> > > > v.rozov@datatorrent.com>
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >> My preference is to agree that the next Malhar release is
> 4.0.0
> > > and
> > > > > make
> > > > > > >>> the proposed changes when 3.8.0 goes out. Otherwise why to
> keep
> > > > > > semantic
> > > > > > >>> versioning checks on Malhar in case there is no version
> > > > > compatibility.
> > > > > > >>>
> > > > > > >>> Thank you,
> > > > > > >>>
> > > > > > >>> Vlad
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> On 7/14/17 08:11, Thomas Weise wrote:
> > > > > > >>>
> > > > > > >>> next release is 3.8.0
> > > > > > >>>>
> > > > > > >>>> On Fri, Jul 14, 2017 at 7:55 AM, Vlad Rozov <
> > > > > v.rozov@datatorrent.com>
> > > > > > >>>> wrote:
> > > > > > >>>>
> > > > > > >>>> next release - 3.9.0 or 4.0.0?
> > > > > > >>>>
> > > > > > >>>>> Thank you,
> > > > > > >>>>>
> > > > > > >>>>> Vlad
> > > > > > >>>>>
> > > > > > >>>>> On 7/13/17 22:25, Thomas Weise wrote:
> > > > > > >>>>>
> > > > > > >>>>> It is time to resurrect this thread and get going with the
> > > work.
> > > > > > >>>>>
> > > > > > >>>>>> For the next release, I will sign up to do the package
> move
> > in
> > > > > > Malhar:
> > > > > > >>>>>>
> > > > > > >>>>>> https://issues.apache.org/jira/browse/APEXMALHAR-2517
> > > > > > >>>>>>
> > > > > > >>>>>> In general this will be straightforward; most classes in
> > > Malhar
> > > > > are
> > > > > > >>>>>> marked
> > > > > > >>>>>> evolving and it is trivial for users to change import
> > > > statements.
> > > > > > >>>>>> However,
> > > > > > >>>>>> I would suggest to discuss if there are selected operators
> > > that
> > > > > are
> > > > > > >>>>>> worth
> > > > > > >>>>>> keeping a backward compatibility stub in the old location.
> > > > > > >>>>>>
> > > > > > >>>>>> Here is my plan:
> > > > > > >>>>>>
> > > > > > >>>>>> 1. Relocate all classes in *lib* and *contrib* within one
> > PR -
> > > > > this
> > > > > > is
> > > > > > >>>>>> all
> > > > > > >>>>>>
> > > > > > >>>>>> IDE automated work
> > > > > > >>>>>> 2. Add backward compatibility classes, if, any in separate
> > PR
> > > > > > >>>>>> 3. Create PR for Megh library to reflect moved classes
> > > > > > >>>>>>
> > > > > > >>>>>> Thanks,
> > > > > > >>>>>> Thomas
> > > > > > >>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>> On Wed, Mar 1, 2017 at 2:24 PM, Pramod Immaneni <
> > > > > > >>>>>> pramod@datatorrent.com
> > > > > > >>>>>> wrote:
> > > > > > >>>>>>
> > > > > > >>>>>> Inline
> > > > > > >>>>>>
> > > > > > >>>>>> On Mon, Feb 27, 2017 at 11:03 PM, Thomas Weise <
> > > thw@apache.org>
> > > > > > >>>>>>> wrote:
> > > > > > >>>>>>>
> > > > > > >>>>>>> -->
> > > > > > >>>>>>>
> > > > > > >>>>>>> On Mon, Feb 27, 2017 at 1:27 PM, Pramod Immaneni <
> > > > > > >>>>>>>> pramod@datatorrent.com
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> wrote:
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> For malhar, for existing operators, I prefer we do this
> as
> > > > part
> > > > > of
> > > > > > >>>>>>>> the
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> planned refactoring for breaking the monolith modules
> into
> > > > baby
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> packages
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>> and would also prefer deprecating the existing operators
> > in
> > > > > place.
> > > > > > >>>>>>>> Refactor into smaller modules was discussed for
> > > malhar-contrib
> > > > > and
> > > > > > >>>>>>>> given
> > > > > > >>>>>>>> the overall state of that module I think it is OK to
> defer
> > > > > package
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> renaming
> > > > > > >>>>>>>>
> > > > > > >>>>>>> there. I do however prefer to see the package rename
> > > addressed
> > > > > for
> > > > > > >>>>>>>
> > > > > > >>>>>>>> other
> > > > > > >>>>>>>> modules, especially for the main library module.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> Should we consider breaking the library into smaller
> > modules
> > > > as
> > > > > > >>>>>>>> well,
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> the
> > > > > > >>>>>>> file/block operators for example probably can be in their
> > own
> > > > > > module
> > > > > > >>>>>>> from
> > > > > > >>>>>>> just an organizational perspective.
> > > > > > >>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>> This
> > > > > > >>>>>>>
> > > > > > >>>>>>>> will help us achieve two things. First, the user will
> see
> > > all
> > > > > the
> > > > > > >>>>>>>>> new
> > > > > > >>>>>>>>> changes at once as opposed to dealing with it twice
> (with
> > > > > package
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> rename
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>> and dependency changes) and second it will allow for a
> > > > smoother
> > > > > > >>>>>>>> transition
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> as the existing code will still work in a deprecated
> > state.
> > > It
> > > > > > will
> > > > > > >>>>>>>>
> > > > > > >>>>>>>>> also
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>> give a more consistent structure to malhar. For new
> > > operators,
> > > > > we
> > > > > > >>>>>>>> can
> > > > > > >>>>>>>> go
> > > > > > >>>>>>>> with the new package path but we need to ensure they
> will
> > > get
> > > > > > moved
> > > > > > >>>>>>>> into
> > > > > > >>>>>>>> the baby packages as well.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> I think existing operators should be renamed so that git
> > > > history
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> remains. A
> > > > > > >>>>>>>>
> > > > > > >>>>>>> possible solution for backward compatibility could be to
> > > > > > subsequently
> > > > > > >>>>>>>
> > > > > > >>>>>>>> add
> > > > > > >>>>>>>> empty subclasses in the previous location (for existing
> > > > concrete
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> operators
> > > > > > >>>>>>>>
> > > > > > >>>>>>> that we know are actually in use) to simplify migration
> for
> > > > > users.
> > > > > > >>>>>>>
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> Yes we can do that.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> For demos, we can modify the paths as the apps are
> > typically
> > > > > used
> > > > > > >>>>>>>
> > > > > > >>>>>>> wholesale
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> and the interface is typically manual interaction.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>>> For core, if we are adding new api subsystems, like the
> > > > > launcher
> > > > > > >>>>>>>>> api
> > > > > > >>>>>>>>> we
> > > > > > >>>>>>>>> added recently for example, we can go with new package
> > path
> > > > but
> > > > > > if
> > > > > > >>>>>>>>> we
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> are
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>> making incremental additions to existing functionality,
> I
> > > feel
> > > > > it
> > > > > > is
> > > > > > >>>>>>>> better
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> to keep it in the same package. I also prefer we keep
> the
> > > > > package
> > > > > > of
> > > > > > >>>>>>>>
> > > > > > >>>>>>>>> the
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>> implementation classes consistent with api, for
> > > > > understandability
> > > > > > >>>>>>>> and
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> readability of the code. So, for example, we don't
> change
> > > > > package
> > > > > > >>>>>>>>> path
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> of
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>> LogicalPlan as it is an implementation of DAG. It is
> > > > subjective,
> > > > > > but
> > > > > > >>>>>>>> it
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> will be good if we can also do the same with classes
> > closely
> > > > > > related
> > > > > > >>>>>>>>> to
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> the
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>> implementation classes as well. Maybe we can moving
> these
> > > on a
> > > > > > >>>>>>>> package
> > > > > > >>>>>>>>
> > > > > > >>>>>>>>> by
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>> package basis, like everything in
> > > com.datatorrent.stram.engine
> > > > > > could
> > > > > > >>>>>>>> be
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> moved. For completely internal components like buffer
> > > server,
> > > > we
> > > > > > can
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> move
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>> them wholesale. We can consider moving all api and
> > classes,
> > > > when
> > > > > > we
> > > > > > >>>>>>>> go
> > > > > > >>>>>>>> to
> > > > > > >>>>>>>> next major release but would like to see if we can find
> a
> > > way
> > > > to
> > > > > > >>>>>>>> support
> > > > > > >>>>>>>> existing api for one more major release in deprecated
> > mode.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> The point of the major release is to enable backward
> > > > > incompatible
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> changes
> > > > > > >>>>>>>> and I don't think it is realistic to support the
> existing
> > > API
> > > > > for
> > > > > > >>>>>>>> another
> > > > > > >>>>>>>> major release. IMO it is also not necessary as most
> > existing
> > > > > > >>>>>>>> application
> > > > > > >>>>>>>> code refers to operators, attributes and the application
> > > > > > interface.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> Perhaps
> > > > > > >>>>>>>>
> > > > > > >>>>>>> it is possible to keep those around as interface
> extensions
> > > to
> > > > > help
> > > > > > >>>>>>>
> > > > > > >>>>>>>> migration. Custom operators may need to be migrated to
> > > reflect
> > > > > API
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> changes,
> > > > > > >>>>>>>>
> > > > > > >>>>>>> and I would consider that a reasonable task for operator
> > > > > developers
> > > > > > >>>>>>> as
> > > > > > >>>>>>>
> > > > > > >>>>>>>> part
> > > > > > >>>>>>>>
> > > > > > >>>>>>> of a major upgrade.
> > > > > > >>>>>>>
> > > > > > >>>>>>>> It would be good if we can keep them as deprecated
> > interface
> > > > > > >>>>>>>> extensions
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> for
> > > > > > >>>>>>> one release to provide a smoother transition.
> > > > > > >>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>> API and implementation in engine are kept separate
> > > > intentionally.
> > > > > > >>>>>>> They
> > > > > > >>>>>>>
> > > > > > >>>>>>> reside in different packages today, so I don't see a
> > problem
> > > > > > renaming
> > > > > > >>>>>>>> com.datatorrent.stram.engine as you say, even when the
> API
> > > > > cannot
> > > > > > be
> > > > > > >>>>>>>> touched right away.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> They are different packages but sharing a common prefix
> > with
> > > > api
> > > > > > >>>>>>>> will
> > > > > > >>>>>>>> be
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> helpful to someone new to codebase in terms of
> > readability.
> > > > Not
> > > > > a
> > > > > > >>>>>>> big
> > > > > > >>>>>>> deal
> > > > > > >>>>>>> and can be changed.
> > > > > > >>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>> Thanks
> > > > > > >>>>>>>
> > > > > > >>>>>>> On Mon, Feb 27, 2017 at 7:39 AM, Thomas Weise <
> > > thw@apache.org>
> > > > > > >>>>>>>> wrote:
> > > > > > >>>>>>>>
> > > > > > >>>>>>>>> Hi,
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> This topic has come up on several PRs and I think it
> > > > warrants a
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> broader
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>> discussion.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> At the time of incubation, the decision was to defer
> > change
> > > of
> > > > > > Java
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>>> packages from com.datatorrent to org.apache.apex till
> > next
> > > > > major
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> release
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>> to
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> ensure backward compatibility for users.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>>> Unfortunately that has lead to some confusion, as
> > > > contributors
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> continue
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>> to
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> add new code under legacy packages.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>>> It is also a wider issue that examples for using Apex
> > > > continue
> > > > > > to
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> refer
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>> to
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> com.datatorrent packages, nearly one year after
> > graduation.
> > > > More
> > > > > > and
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>>> more
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>> user code is being built on top of something that needs
> > to
> > > > > > change,
> > > > > > >>>>>>>>> the
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> can
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> is being kicked down the road and users will face more
> > > changes
> > > > > > >>>>>>>>> later.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>>> I would like to propose the following:
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> 1. All new code has to be submitted under
> > org.apache.apex
> > > > > > packages
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> 2. Not all code is under backward compatibility
> > > restriction
> > > > > and
> > > > > > in
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> those
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>> cases we can rename the packages right away. Examples:
> > > buffer
> > > > > > >>>>>>>>> server,
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> engine, demos/examples, benchmarks
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> 3. Discuss when the core API and operators can be
> > changed.
> > > > For
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>> operators
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>> we
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> have a bit more freedom to do changes before a major
> > > release
> > > > as
> > > > > > >>>>>>>>> most
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>>> of
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>> them are marked @Evolving and users have the ability to
> > > > > continue
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> using
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>> prior version of Malhar with newer engine due to engine
> > > > > backward
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> compatibility guarantee.
> > > > > > >>>>>>>>>
> > > > > > >>>>>>>>>> Thanks,
> > > > > > >>>>>>>>>> Thomas
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>>
> > > > > > >>>>>>>>>>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Java packages: legacy -> org.apache.apex
Posted by Pramod Immaneni <pr...@datatorrent.com>.
I don't think we can limit the topic strictly to relocation without having
a good b/w compatibility story or at least one that goes far enough.
The shading idea sounds interesting. Why not let the shaded version move
forward with each release till we hit a major release. If it is going to
remain pegged at 3.7.0, why shade in the first place as the regular 3.7.0
release would do the same job and it would be same as the loss of b/w
compatibility with newer releases.
Thanks
On Sat, Jul 15, 2017 at 7:57 AM, Thomas Weise <th...@apache.org> wrote:
> Discussing what in the future might become stable needs to be a separate
> thread, it will be a much bigger discussion.
>
> The topic here is to relocate the packages. With a few exceptions
> relocation won't affect the semantic versioning. Semantic versioning is
> essentially not effective for Malhar because almost everything is @Evolving
> (and there are reasons for that.. -> separate topic)
>
> I don't really like the idea of creating bw compatibility stubs for the
> follow up PR. It creates even more clutter in the source tree than there
> already is and so here is an alternative suggestion:
>
> https://github.com/tweise/apex-malhar/blob/malhar37-
> compat/shaded-malhar37/pom.xml
>
> Create a shaded artifact that provides the old com.datatorrent.* classes as
> of release 3.7. Users can include that artifact if they don't want to
> change import statements. At the same time they have an incentive to switch
> to the relocated classes to take advantage of bug fixes and new
> functionality.
>
> I will work on the first PR that does the relocate. In the meantime, we can
> finalize what backward compatibility support we want to provide and how.
>
> Thanks,
> Thomas
>
>
>
>
> On Fri, Jul 14, 2017 at 11:33 AM, Pramod Immaneni <pr...@datatorrent.com>
> wrote:
>
> > How about coming up with a list of @Evolving operators that we would like
> > to see in the eventual stable list and move those along with the not
> > @Evolving ones in org.apache.apex with b/w stubs and leave the rest as
> they
> > are. Then have a follow up JIRA for the rest to be moved over to contrib
> > and be deprecated.
> >
> > Thanks
> >
> > On Fri, Jul 14, 2017 at 10:37 AM, Thomas Weise <th...@gmail.com>
> > wrote:
> >
> > > We need to keep the discussion here on topic, if other things are piled
> > on
> > > top then nothing gets done.
> > >
> > > Most operators today are not stable, they are @Evolving. So perhaps it
> is
> > > necessary to have a separate discussion about that, outside of this
> > thread.
> > > My guess is that there are only a few operators that we could declare
> > > stable. Specifically, under contrib the closest one might have been
> > Kafka,
> > > but that is already superseded by the newer versions.
> > >
> > > Thomas
> > >
> > >
> > > On Fri, Jul 14, 2017 at 10:21 AM, Pramod Immaneni <
> > pramod@datatorrent.com>
> > > wrote:
> > >
> > > > We had a discussion a while back, agreed to relegate non-stable and
> > > > experimental operators to contrib and also added this to contribution
> > > > guidelines. We aexecuted on this and cleaned up the repo by moving
> > > > operators deemed non-suitable for production use at that time to
> > contrib.
> > > > So I wouldn't say the operators that are at the top level today or
> the
> > > ones
> > > > in library are 0.x.x quality. Granted, we may need to do one more
> pass
> > as
> > > > some of the operators may no longer be considered the right
> > > implementations
> > > > with the advent of the windowed operator.
> > > >
> > > > Since contrib used to be the place that housed operators that
> required
> > > > third party libraries, there are some production quality operators in
> > > there
> > > > that need to be pulled out into top level modules.
> > > >
> > > > I think we are in agreement that for operators that we consider
> stable,
> > > we
> > > > should provide a b/w stub. I would add that we strongly consider
> > creating
> > > > the org.apache.apex counterparts of any stable operators that are in
> > > > contrib out in top level modules and have the com.datatorrent stubs
> in
> > > > contrib.
> > > >
> > > > For the operators not considered stable, I would prefer we either
> leave
> > > the
> > > > current package path as is or if we change the package path then
> create
> > > the
> > > > b/w stub as I am not sure how widely they are in use (so, in essence,
> > > > preserve semantic versioning). It would be good if there is a
> followup
> > > JIRA
> > > > that takes a look at what other operators can be moved to contrib
> with
> > > the
> > > > advent of the newer frameworks and understanding.
> > > >
> > > > Thanks
> > > >
> > > > On Fri, Jul 14, 2017 at 9:44 AM, Thomas Weise <th...@apache.org>
> wrote:
> > > >
> > > > > Most of the operators evolve, as is quite clearly indicated by
> > > @Evolving
> > > > > annotations. So while perhaps 0.x.x would be a more appropriate
> > version
> > > > > number, I don't think you can change that.
> > > > >
> > > > > Thomas
> > > > >
> > > > > On Fri, Jul 14, 2017 at 9:35 AM, Vlad Rozov <
> v.rozov@datatorrent.com
> > >
> > > > > wrote:
> > > > >
> > > > > > If entire library is not stable, its version should be 0.x.x and
> > > there
> > > > > > should not be any semantic versioning enabled or implied. It
> > evolves.
> > > > If
> > > > > > some operators are stable as 3.8.x version suggests, the library
> > > should
> > > > > > follow semantic versioning and it is not OK to make a major
> change
> > > that
> > > > > > breaks semantic versioning across the entire library. It is not a
> > > > finding
> > > > > > an excuse not to make a change. For me semantic versioning and
> > > backward
> > > > > > compatibility is more important than a proper package name.
> > > > > >
> > > > > > Thank you,
> > > > > >
> > > > > > Vlad
> > > > > >
> > > > > >
> > > > > > On 7/14/17 09:11, Thomas Weise wrote:
> > > > > >
> > > > > >> Semantic versioning makes sense when you have a stable baseline.
> > As
> > > > long
> > > > > >> as
> > > > > >> frequent fixes need to be made to address basic issues, it makes
> > no
> > > > > sense
> > > > > >> to declare operators stable, which is why they are marked
> > evolving.
> > > > > >>
> > > > > >> Instead of finding reasons to avoid changes that should have
> been
> > > > made a
> > > > > >> long time ago, why not discuss what a "stable operator" is and
> > which
> > > > > ones
> > > > > >> deserve to be in that category. Those are the ones that will get
> > the
> > > > > >> backward compatibility stub.
> > > > > >>
> > > > > >> Thanks,
> > > > > >> Thomas
> > > > > >>
> > > > > >>
> > > > > >> On Fri, Jul 14, 2017 at 8:34 AM, Vlad Rozov <
> > > v.rozov@datatorrent.com>
> > > > > >> wrote:
> > > > > >>
> > > > > >> My preference is to agree that the next Malhar release is 4.0.0
> > and
> > > > make
> > > > > >>> the proposed changes when 3.8.0 goes out. Otherwise why to keep
> > > > > semantic
> > > > > >>> versioning checks on Malhar in case there is no version
> > > > compatibility.
> > > > > >>>
> > > > > >>> Thank you,
> > > > > >>>
> > > > > >>> Vlad
> > > > > >>>
> > > > > >>>
> > > > > >>> On 7/14/17 08:11, Thomas Weise wrote:
> > > > > >>>
> > > > > >>> next release is 3.8.0
> > > > > >>>>
> > > > > >>>> On Fri, Jul 14, 2017 at 7:55 AM, Vlad Rozov <
> > > > v.rozov@datatorrent.com>
> > > > > >>>> wrote:
> > > > > >>>>
> > > > > >>>> next release - 3.9.0 or 4.0.0?
> > > > > >>>>
> > > > > >>>>> Thank you,
> > > > > >>>>>
> > > > > >>>>> Vlad
> > > > > >>>>>
> > > > > >>>>> On 7/13/17 22:25, Thomas Weise wrote:
> > > > > >>>>>
> > > > > >>>>> It is time to resurrect this thread and get going with the
> > work.
> > > > > >>>>>
> > > > > >>>>>> For the next release, I will sign up to do the package move
> in
> > > > > Malhar:
> > > > > >>>>>>
> > > > > >>>>>> https://issues.apache.org/jira/browse/APEXMALHAR-2517
> > > > > >>>>>>
> > > > > >>>>>> In general this will be straightforward; most classes in
> > Malhar
> > > > are
> > > > > >>>>>> marked
> > > > > >>>>>> evolving and it is trivial for users to change import
> > > statements.
> > > > > >>>>>> However,
> > > > > >>>>>> I would suggest to discuss if there are selected operators
> > that
> > > > are
> > > > > >>>>>> worth
> > > > > >>>>>> keeping a backward compatibility stub in the old location.
> > > > > >>>>>>
> > > > > >>>>>> Here is my plan:
> > > > > >>>>>>
> > > > > >>>>>> 1. Relocate all classes in *lib* and *contrib* within one
> PR -
> > > > this
> > > > > is
> > > > > >>>>>> all
> > > > > >>>>>>
> > > > > >>>>>> IDE automated work
> > > > > >>>>>> 2. Add backward compatibility classes, if, any in separate
> PR
> > > > > >>>>>> 3. Create PR for Megh library to reflect moved classes
> > > > > >>>>>>
> > > > > >>>>>> Thanks,
> > > > > >>>>>> Thomas
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>>> On Wed, Mar 1, 2017 at 2:24 PM, Pramod Immaneni <
> > > > > >>>>>> pramod@datatorrent.com
> > > > > >>>>>> wrote:
> > > > > >>>>>>
> > > > > >>>>>> Inline
> > > > > >>>>>>
> > > > > >>>>>> On Mon, Feb 27, 2017 at 11:03 PM, Thomas Weise <
> > thw@apache.org>
> > > > > >>>>>>> wrote:
> > > > > >>>>>>>
> > > > > >>>>>>> -->
> > > > > >>>>>>>
> > > > > >>>>>>> On Mon, Feb 27, 2017 at 1:27 PM, Pramod Immaneni <
> > > > > >>>>>>>> pramod@datatorrent.com
> > > > > >>>>>>>>
> > > > > >>>>>>>> wrote:
> > > > > >>>>>>>>
> > > > > >>>>>>>> For malhar, for existing operators, I prefer we do this as
> > > part
> > > > of
> > > > > >>>>>>>> the
> > > > > >>>>>>>>
> > > > > >>>>>>>> planned refactoring for breaking the monolith modules into
> > > baby
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> packages
> > > > > >>>>>>>>>
> > > > > >>>>>>>> and would also prefer deprecating the existing operators
> in
> > > > place.
> > > > > >>>>>>>> Refactor into smaller modules was discussed for
> > malhar-contrib
> > > > and
> > > > > >>>>>>>> given
> > > > > >>>>>>>> the overall state of that module I think it is OK to defer
> > > > package
> > > > > >>>>>>>>
> > > > > >>>>>>>> renaming
> > > > > >>>>>>>>
> > > > > >>>>>>> there. I do however prefer to see the package rename
> > addressed
> > > > for
> > > > > >>>>>>>
> > > > > >>>>>>>> other
> > > > > >>>>>>>> modules, especially for the main library module.
> > > > > >>>>>>>>
> > > > > >>>>>>>> Should we consider breaking the library into smaller
> modules
> > > as
> > > > > >>>>>>>> well,
> > > > > >>>>>>>>
> > > > > >>>>>>>> the
> > > > > >>>>>>> file/block operators for example probably can be in their
> own
> > > > > module
> > > > > >>>>>>> from
> > > > > >>>>>>> just an organizational perspective.
> > > > > >>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>> This
> > > > > >>>>>>>
> > > > > >>>>>>>> will help us achieve two things. First, the user will see
> > all
> > > > the
> > > > > >>>>>>>>> new
> > > > > >>>>>>>>> changes at once as opposed to dealing with it twice (with
> > > > package
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> rename
> > > > > >>>>>>>>>
> > > > > >>>>>>>> and dependency changes) and second it will allow for a
> > > smoother
> > > > > >>>>>>>> transition
> > > > > >>>>>>>>
> > > > > >>>>>>>> as the existing code will still work in a deprecated
> state.
> > It
> > > > > will
> > > > > >>>>>>>>
> > > > > >>>>>>>>> also
> > > > > >>>>>>>>>
> > > > > >>>>>>>> give a more consistent structure to malhar. For new
> > operators,
> > > > we
> > > > > >>>>>>>> can
> > > > > >>>>>>>> go
> > > > > >>>>>>>> with the new package path but we need to ensure they will
> > get
> > > > > moved
> > > > > >>>>>>>> into
> > > > > >>>>>>>> the baby packages as well.
> > > > > >>>>>>>>
> > > > > >>>>>>>> I think existing operators should be renamed so that git
> > > history
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> remains. A
> > > > > >>>>>>>>
> > > > > >>>>>>> possible solution for backward compatibility could be to
> > > > > subsequently
> > > > > >>>>>>>
> > > > > >>>>>>>> add
> > > > > >>>>>>>> empty subclasses in the previous location (for existing
> > > concrete
> > > > > >>>>>>>>
> > > > > >>>>>>>> operators
> > > > > >>>>>>>>
> > > > > >>>>>>> that we know are actually in use) to simplify migration for
> > > > users.
> > > > > >>>>>>>
> > > > > >>>>>>>>
> > > > > >>>>>>>> Yes we can do that.
> > > > > >>>>>>>>
> > > > > >>>>>>>> For demos, we can modify the paths as the apps are
> typically
> > > > used
> > > > > >>>>>>>
> > > > > >>>>>>> wholesale
> > > > > >>>>>>>>
> > > > > >>>>>>>> and the interface is typically manual interaction.
> > > > > >>>>>>>>
> > > > > >>>>>>>>> For core, if we are adding new api subsystems, like the
> > > > launcher
> > > > > >>>>>>>>> api
> > > > > >>>>>>>>> we
> > > > > >>>>>>>>> added recently for example, we can go with new package
> path
> > > but
> > > > > if
> > > > > >>>>>>>>> we
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> are
> > > > > >>>>>>>>>
> > > > > >>>>>>>> making incremental additions to existing functionality, I
> > feel
> > > > it
> > > > > is
> > > > > >>>>>>>> better
> > > > > >>>>>>>>
> > > > > >>>>>>>> to keep it in the same package. I also prefer we keep the
> > > > package
> > > > > of
> > > > > >>>>>>>>
> > > > > >>>>>>>>> the
> > > > > >>>>>>>>>
> > > > > >>>>>>>> implementation classes consistent with api, for
> > > > understandability
> > > > > >>>>>>>> and
> > > > > >>>>>>>>
> > > > > >>>>>>>> readability of the code. So, for example, we don't change
> > > > package
> > > > > >>>>>>>>> path
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> of
> > > > > >>>>>>>>>
> > > > > >>>>>>>> LogicalPlan as it is an implementation of DAG. It is
> > > subjective,
> > > > > but
> > > > > >>>>>>>> it
> > > > > >>>>>>>>
> > > > > >>>>>>>> will be good if we can also do the same with classes
> closely
> > > > > related
> > > > > >>>>>>>>> to
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> the
> > > > > >>>>>>>>>
> > > > > >>>>>>>> implementation classes as well. Maybe we can moving these
> > on a
> > > > > >>>>>>>> package
> > > > > >>>>>>>>
> > > > > >>>>>>>>> by
> > > > > >>>>>>>>>
> > > > > >>>>>>>> package basis, like everything in
> > com.datatorrent.stram.engine
> > > > > could
> > > > > >>>>>>>> be
> > > > > >>>>>>>>
> > > > > >>>>>>>> moved. For completely internal components like buffer
> > server,
> > > we
> > > > > can
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> move
> > > > > >>>>>>>>>
> > > > > >>>>>>>> them wholesale. We can consider moving all api and
> classes,
> > > when
> > > > > we
> > > > > >>>>>>>> go
> > > > > >>>>>>>> to
> > > > > >>>>>>>> next major release but would like to see if we can find a
> > way
> > > to
> > > > > >>>>>>>> support
> > > > > >>>>>>>> existing api for one more major release in deprecated
> mode.
> > > > > >>>>>>>>
> > > > > >>>>>>>> The point of the major release is to enable backward
> > > > incompatible
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> changes
> > > > > >>>>>>>> and I don't think it is realistic to support the existing
> > API
> > > > for
> > > > > >>>>>>>> another
> > > > > >>>>>>>> major release. IMO it is also not necessary as most
> existing
> > > > > >>>>>>>> application
> > > > > >>>>>>>> code refers to operators, attributes and the application
> > > > > interface.
> > > > > >>>>>>>>
> > > > > >>>>>>>> Perhaps
> > > > > >>>>>>>>
> > > > > >>>>>>> it is possible to keep those around as interface extensions
> > to
> > > > help
> > > > > >>>>>>>
> > > > > >>>>>>>> migration. Custom operators may need to be migrated to
> > reflect
> > > > API
> > > > > >>>>>>>>
> > > > > >>>>>>>> changes,
> > > > > >>>>>>>>
> > > > > >>>>>>> and I would consider that a reasonable task for operator
> > > > developers
> > > > > >>>>>>> as
> > > > > >>>>>>>
> > > > > >>>>>>>> part
> > > > > >>>>>>>>
> > > > > >>>>>>> of a major upgrade.
> > > > > >>>>>>>
> > > > > >>>>>>>> It would be good if we can keep them as deprecated
> interface
> > > > > >>>>>>>> extensions
> > > > > >>>>>>>>
> > > > > >>>>>>>> for
> > > > > >>>>>>> one release to provide a smoother transition.
> > > > > >>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>> API and implementation in engine are kept separate
> > > intentionally.
> > > > > >>>>>>> They
> > > > > >>>>>>>
> > > > > >>>>>>> reside in different packages today, so I don't see a
> problem
> > > > > renaming
> > > > > >>>>>>>> com.datatorrent.stram.engine as you say, even when the API
> > > > cannot
> > > > > be
> > > > > >>>>>>>> touched right away.
> > > > > >>>>>>>>
> > > > > >>>>>>>> They are different packages but sharing a common prefix
> with
> > > api
> > > > > >>>>>>>> will
> > > > > >>>>>>>> be
> > > > > >>>>>>>>
> > > > > >>>>>>>> helpful to someone new to codebase in terms of
> readability.
> > > Not
> > > > a
> > > > > >>>>>>> big
> > > > > >>>>>>> deal
> > > > > >>>>>>> and can be changed.
> > > > > >>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>> Thanks
> > > > > >>>>>>>
> > > > > >>>>>>> On Mon, Feb 27, 2017 at 7:39 AM, Thomas Weise <
> > thw@apache.org>
> > > > > >>>>>>>> wrote:
> > > > > >>>>>>>>
> > > > > >>>>>>>>> Hi,
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> This topic has come up on several PRs and I think it
> > > warrants a
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> broader
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>> discussion.
> > > > > >>>>>>>>
> > > > > >>>>>>>> At the time of incubation, the decision was to defer
> change
> > of
> > > > > Java
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>> packages from com.datatorrent to org.apache.apex till
> next
> > > > major
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> release
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>> to
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> ensure backward compatibility for users.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>> Unfortunately that has lead to some confusion, as
> > > contributors
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> continue
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>> to
> > > > > >>>>>>>>
> > > > > >>>>>>>> add new code under legacy packages.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>> It is also a wider issue that examples for using Apex
> > > continue
> > > > > to
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> refer
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>> to
> > > > > >>>>>>>>
> > > > > >>>>>>>> com.datatorrent packages, nearly one year after
> graduation.
> > > More
> > > > > and
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>> more
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>> user code is being built on top of something that needs
> to
> > > > > change,
> > > > > >>>>>>>>> the
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> can
> > > > > >>>>>>>>
> > > > > >>>>>>>> is being kicked down the road and users will face more
> > changes
> > > > > >>>>>>>>> later.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>> I would like to propose the following:
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> 1. All new code has to be submitted under
> org.apache.apex
> > > > > packages
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> 2. Not all code is under backward compatibility
> > restriction
> > > > and
> > > > > in
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> those
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>> cases we can rename the packages right away. Examples:
> > buffer
> > > > > >>>>>>>>> server,
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> engine, demos/examples, benchmarks
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> 3. Discuss when the core API and operators can be
> changed.
> > > For
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> operators
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>> we
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> have a bit more freedom to do changes before a major
> > release
> > > as
> > > > > >>>>>>>>> most
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>> of
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>> them are marked @Evolving and users have the ability to
> > > > continue
> > > > > >>>>>>>>
> > > > > >>>>>>>> using
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> prior version of Malhar with newer engine due to engine
> > > > backward
> > > > > >>>>>>>>
> > > > > >>>>>>>> compatibility guarantee.
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>> Thanks,
> > > > > >>>>>>>>>> Thomas
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Java packages: legacy -> org.apache.apex
Posted by Thomas Weise <th...@apache.org>.
Discussing what in the future might become stable needs to be a separate
thread, it will be a much bigger discussion.
The topic here is to relocate the packages. With a few exceptions
relocation won't affect the semantic versioning. Semantic versioning is
essentially not effective for Malhar because almost everything is @Evolving
(and there are reasons for that.. -> separate topic)
I don't really like the idea of creating bw compatibility stubs for the
follow up PR. It creates even more clutter in the source tree than there
already is and so here is an alternative suggestion:
https://github.com/tweise/apex-malhar/blob/malhar37-compat/shaded-malhar37/pom.xml
Create a shaded artifact that provides the old com.datatorrent.* classes as
of release 3.7. Users can include that artifact if they don't want to
change import statements. At the same time they have an incentive to switch
to the relocated classes to take advantage of bug fixes and new
functionality.
I will work on the first PR that does the relocate. In the meantime, we can
finalize what backward compatibility support we want to provide and how.
Thanks,
Thomas
On Fri, Jul 14, 2017 at 11:33 AM, Pramod Immaneni <pr...@datatorrent.com>
wrote:
> How about coming up with a list of @Evolving operators that we would like
> to see in the eventual stable list and move those along with the not
> @Evolving ones in org.apache.apex with b/w stubs and leave the rest as they
> are. Then have a follow up JIRA for the rest to be moved over to contrib
> and be deprecated.
>
> Thanks
>
> On Fri, Jul 14, 2017 at 10:37 AM, Thomas Weise <th...@gmail.com>
> wrote:
>
> > We need to keep the discussion here on topic, if other things are piled
> on
> > top then nothing gets done.
> >
> > Most operators today are not stable, they are @Evolving. So perhaps it is
> > necessary to have a separate discussion about that, outside of this
> thread.
> > My guess is that there are only a few operators that we could declare
> > stable. Specifically, under contrib the closest one might have been
> Kafka,
> > but that is already superseded by the newer versions.
> >
> > Thomas
> >
> >
> > On Fri, Jul 14, 2017 at 10:21 AM, Pramod Immaneni <
> pramod@datatorrent.com>
> > wrote:
> >
> > > We had a discussion a while back, agreed to relegate non-stable and
> > > experimental operators to contrib and also added this to contribution
> > > guidelines. We aexecuted on this and cleaned up the repo by moving
> > > operators deemed non-suitable for production use at that time to
> contrib.
> > > So I wouldn't say the operators that are at the top level today or the
> > ones
> > > in library are 0.x.x quality. Granted, we may need to do one more pass
> as
> > > some of the operators may no longer be considered the right
> > implementations
> > > with the advent of the windowed operator.
> > >
> > > Since contrib used to be the place that housed operators that required
> > > third party libraries, there are some production quality operators in
> > there
> > > that need to be pulled out into top level modules.
> > >
> > > I think we are in agreement that for operators that we consider stable,
> > we
> > > should provide a b/w stub. I would add that we strongly consider
> creating
> > > the org.apache.apex counterparts of any stable operators that are in
> > > contrib out in top level modules and have the com.datatorrent stubs in
> > > contrib.
> > >
> > > For the operators not considered stable, I would prefer we either leave
> > the
> > > current package path as is or if we change the package path then create
> > the
> > > b/w stub as I am not sure how widely they are in use (so, in essence,
> > > preserve semantic versioning). It would be good if there is a followup
> > JIRA
> > > that takes a look at what other operators can be moved to contrib with
> > the
> > > advent of the newer frameworks and understanding.
> > >
> > > Thanks
> > >
> > > On Fri, Jul 14, 2017 at 9:44 AM, Thomas Weise <th...@apache.org> wrote:
> > >
> > > > Most of the operators evolve, as is quite clearly indicated by
> > @Evolving
> > > > annotations. So while perhaps 0.x.x would be a more appropriate
> version
> > > > number, I don't think you can change that.
> > > >
> > > > Thomas
> > > >
> > > > On Fri, Jul 14, 2017 at 9:35 AM, Vlad Rozov <v.rozov@datatorrent.com
> >
> > > > wrote:
> > > >
> > > > > If entire library is not stable, its version should be 0.x.x and
> > there
> > > > > should not be any semantic versioning enabled or implied. It
> evolves.
> > > If
> > > > > some operators are stable as 3.8.x version suggests, the library
> > should
> > > > > follow semantic versioning and it is not OK to make a major change
> > that
> > > > > breaks semantic versioning across the entire library. It is not a
> > > finding
> > > > > an excuse not to make a change. For me semantic versioning and
> > backward
> > > > > compatibility is more important than a proper package name.
> > > > >
> > > > > Thank you,
> > > > >
> > > > > Vlad
> > > > >
> > > > >
> > > > > On 7/14/17 09:11, Thomas Weise wrote:
> > > > >
> > > > >> Semantic versioning makes sense when you have a stable baseline.
> As
> > > long
> > > > >> as
> > > > >> frequent fixes need to be made to address basic issues, it makes
> no
> > > > sense
> > > > >> to declare operators stable, which is why they are marked
> evolving.
> > > > >>
> > > > >> Instead of finding reasons to avoid changes that should have been
> > > made a
> > > > >> long time ago, why not discuss what a "stable operator" is and
> which
> > > > ones
> > > > >> deserve to be in that category. Those are the ones that will get
> the
> > > > >> backward compatibility stub.
> > > > >>
> > > > >> Thanks,
> > > > >> Thomas
> > > > >>
> > > > >>
> > > > >> On Fri, Jul 14, 2017 at 8:34 AM, Vlad Rozov <
> > v.rozov@datatorrent.com>
> > > > >> wrote:
> > > > >>
> > > > >> My preference is to agree that the next Malhar release is 4.0.0
> and
> > > make
> > > > >>> the proposed changes when 3.8.0 goes out. Otherwise why to keep
> > > > semantic
> > > > >>> versioning checks on Malhar in case there is no version
> > > compatibility.
> > > > >>>
> > > > >>> Thank you,
> > > > >>>
> > > > >>> Vlad
> > > > >>>
> > > > >>>
> > > > >>> On 7/14/17 08:11, Thomas Weise wrote:
> > > > >>>
> > > > >>> next release is 3.8.0
> > > > >>>>
> > > > >>>> On Fri, Jul 14, 2017 at 7:55 AM, Vlad Rozov <
> > > v.rozov@datatorrent.com>
> > > > >>>> wrote:
> > > > >>>>
> > > > >>>> next release - 3.9.0 or 4.0.0?
> > > > >>>>
> > > > >>>>> Thank you,
> > > > >>>>>
> > > > >>>>> Vlad
> > > > >>>>>
> > > > >>>>> On 7/13/17 22:25, Thomas Weise wrote:
> > > > >>>>>
> > > > >>>>> It is time to resurrect this thread and get going with the
> work.
> > > > >>>>>
> > > > >>>>>> For the next release, I will sign up to do the package move in
> > > > Malhar:
> > > > >>>>>>
> > > > >>>>>> https://issues.apache.org/jira/browse/APEXMALHAR-2517
> > > > >>>>>>
> > > > >>>>>> In general this will be straightforward; most classes in
> Malhar
> > > are
> > > > >>>>>> marked
> > > > >>>>>> evolving and it is trivial for users to change import
> > statements.
> > > > >>>>>> However,
> > > > >>>>>> I would suggest to discuss if there are selected operators
> that
> > > are
> > > > >>>>>> worth
> > > > >>>>>> keeping a backward compatibility stub in the old location.
> > > > >>>>>>
> > > > >>>>>> Here is my plan:
> > > > >>>>>>
> > > > >>>>>> 1. Relocate all classes in *lib* and *contrib* within one PR -
> > > this
> > > > is
> > > > >>>>>> all
> > > > >>>>>>
> > > > >>>>>> IDE automated work
> > > > >>>>>> 2. Add backward compatibility classes, if, any in separate PR
> > > > >>>>>> 3. Create PR for Megh library to reflect moved classes
> > > > >>>>>>
> > > > >>>>>> Thanks,
> > > > >>>>>> Thomas
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>> On Wed, Mar 1, 2017 at 2:24 PM, Pramod Immaneni <
> > > > >>>>>> pramod@datatorrent.com
> > > > >>>>>> wrote:
> > > > >>>>>>
> > > > >>>>>> Inline
> > > > >>>>>>
> > > > >>>>>> On Mon, Feb 27, 2017 at 11:03 PM, Thomas Weise <
> thw@apache.org>
> > > > >>>>>>> wrote:
> > > > >>>>>>>
> > > > >>>>>>> -->
> > > > >>>>>>>
> > > > >>>>>>> On Mon, Feb 27, 2017 at 1:27 PM, Pramod Immaneni <
> > > > >>>>>>>> pramod@datatorrent.com
> > > > >>>>>>>>
> > > > >>>>>>>> wrote:
> > > > >>>>>>>>
> > > > >>>>>>>> For malhar, for existing operators, I prefer we do this as
> > part
> > > of
> > > > >>>>>>>> the
> > > > >>>>>>>>
> > > > >>>>>>>> planned refactoring for breaking the monolith modules into
> > baby
> > > > >>>>>>>>>
> > > > >>>>>>>>> packages
> > > > >>>>>>>>>
> > > > >>>>>>>> and would also prefer deprecating the existing operators in
> > > place.
> > > > >>>>>>>> Refactor into smaller modules was discussed for
> malhar-contrib
> > > and
> > > > >>>>>>>> given
> > > > >>>>>>>> the overall state of that module I think it is OK to defer
> > > package
> > > > >>>>>>>>
> > > > >>>>>>>> renaming
> > > > >>>>>>>>
> > > > >>>>>>> there. I do however prefer to see the package rename
> addressed
> > > for
> > > > >>>>>>>
> > > > >>>>>>>> other
> > > > >>>>>>>> modules, especially for the main library module.
> > > > >>>>>>>>
> > > > >>>>>>>> Should we consider breaking the library into smaller modules
> > as
> > > > >>>>>>>> well,
> > > > >>>>>>>>
> > > > >>>>>>>> the
> > > > >>>>>>> file/block operators for example probably can be in their own
> > > > module
> > > > >>>>>>> from
> > > > >>>>>>> just an organizational perspective.
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>> This
> > > > >>>>>>>
> > > > >>>>>>>> will help us achieve two things. First, the user will see
> all
> > > the
> > > > >>>>>>>>> new
> > > > >>>>>>>>> changes at once as opposed to dealing with it twice (with
> > > package
> > > > >>>>>>>>>
> > > > >>>>>>>>> rename
> > > > >>>>>>>>>
> > > > >>>>>>>> and dependency changes) and second it will allow for a
> > smoother
> > > > >>>>>>>> transition
> > > > >>>>>>>>
> > > > >>>>>>>> as the existing code will still work in a deprecated state.
> It
> > > > will
> > > > >>>>>>>>
> > > > >>>>>>>>> also
> > > > >>>>>>>>>
> > > > >>>>>>>> give a more consistent structure to malhar. For new
> operators,
> > > we
> > > > >>>>>>>> can
> > > > >>>>>>>> go
> > > > >>>>>>>> with the new package path but we need to ensure they will
> get
> > > > moved
> > > > >>>>>>>> into
> > > > >>>>>>>> the baby packages as well.
> > > > >>>>>>>>
> > > > >>>>>>>> I think existing operators should be renamed so that git
> > history
> > > > >>>>>>>>>
> > > > >>>>>>>>> remains. A
> > > > >>>>>>>>
> > > > >>>>>>> possible solution for backward compatibility could be to
> > > > subsequently
> > > > >>>>>>>
> > > > >>>>>>>> add
> > > > >>>>>>>> empty subclasses in the previous location (for existing
> > concrete
> > > > >>>>>>>>
> > > > >>>>>>>> operators
> > > > >>>>>>>>
> > > > >>>>>>> that we know are actually in use) to simplify migration for
> > > users.
> > > > >>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>> Yes we can do that.
> > > > >>>>>>>>
> > > > >>>>>>>> For demos, we can modify the paths as the apps are typically
> > > used
> > > > >>>>>>>
> > > > >>>>>>> wholesale
> > > > >>>>>>>>
> > > > >>>>>>>> and the interface is typically manual interaction.
> > > > >>>>>>>>
> > > > >>>>>>>>> For core, if we are adding new api subsystems, like the
> > > launcher
> > > > >>>>>>>>> api
> > > > >>>>>>>>> we
> > > > >>>>>>>>> added recently for example, we can go with new package path
> > but
> > > > if
> > > > >>>>>>>>> we
> > > > >>>>>>>>>
> > > > >>>>>>>>> are
> > > > >>>>>>>>>
> > > > >>>>>>>> making incremental additions to existing functionality, I
> feel
> > > it
> > > > is
> > > > >>>>>>>> better
> > > > >>>>>>>>
> > > > >>>>>>>> to keep it in the same package. I also prefer we keep the
> > > package
> > > > of
> > > > >>>>>>>>
> > > > >>>>>>>>> the
> > > > >>>>>>>>>
> > > > >>>>>>>> implementation classes consistent with api, for
> > > understandability
> > > > >>>>>>>> and
> > > > >>>>>>>>
> > > > >>>>>>>> readability of the code. So, for example, we don't change
> > > package
> > > > >>>>>>>>> path
> > > > >>>>>>>>>
> > > > >>>>>>>>> of
> > > > >>>>>>>>>
> > > > >>>>>>>> LogicalPlan as it is an implementation of DAG. It is
> > subjective,
> > > > but
> > > > >>>>>>>> it
> > > > >>>>>>>>
> > > > >>>>>>>> will be good if we can also do the same with classes closely
> > > > related
> > > > >>>>>>>>> to
> > > > >>>>>>>>>
> > > > >>>>>>>>> the
> > > > >>>>>>>>>
> > > > >>>>>>>> implementation classes as well. Maybe we can moving these
> on a
> > > > >>>>>>>> package
> > > > >>>>>>>>
> > > > >>>>>>>>> by
> > > > >>>>>>>>>
> > > > >>>>>>>> package basis, like everything in
> com.datatorrent.stram.engine
> > > > could
> > > > >>>>>>>> be
> > > > >>>>>>>>
> > > > >>>>>>>> moved. For completely internal components like buffer
> server,
> > we
> > > > can
> > > > >>>>>>>>>
> > > > >>>>>>>>> move
> > > > >>>>>>>>>
> > > > >>>>>>>> them wholesale. We can consider moving all api and classes,
> > when
> > > > we
> > > > >>>>>>>> go
> > > > >>>>>>>> to
> > > > >>>>>>>> next major release but would like to see if we can find a
> way
> > to
> > > > >>>>>>>> support
> > > > >>>>>>>> existing api for one more major release in deprecated mode.
> > > > >>>>>>>>
> > > > >>>>>>>> The point of the major release is to enable backward
> > > incompatible
> > > > >>>>>>>>>
> > > > >>>>>>>>> changes
> > > > >>>>>>>> and I don't think it is realistic to support the existing
> API
> > > for
> > > > >>>>>>>> another
> > > > >>>>>>>> major release. IMO it is also not necessary as most existing
> > > > >>>>>>>> application
> > > > >>>>>>>> code refers to operators, attributes and the application
> > > > interface.
> > > > >>>>>>>>
> > > > >>>>>>>> Perhaps
> > > > >>>>>>>>
> > > > >>>>>>> it is possible to keep those around as interface extensions
> to
> > > help
> > > > >>>>>>>
> > > > >>>>>>>> migration. Custom operators may need to be migrated to
> reflect
> > > API
> > > > >>>>>>>>
> > > > >>>>>>>> changes,
> > > > >>>>>>>>
> > > > >>>>>>> and I would consider that a reasonable task for operator
> > > developers
> > > > >>>>>>> as
> > > > >>>>>>>
> > > > >>>>>>>> part
> > > > >>>>>>>>
> > > > >>>>>>> of a major upgrade.
> > > > >>>>>>>
> > > > >>>>>>>> It would be good if we can keep them as deprecated interface
> > > > >>>>>>>> extensions
> > > > >>>>>>>>
> > > > >>>>>>>> for
> > > > >>>>>>> one release to provide a smoother transition.
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>> API and implementation in engine are kept separate
> > intentionally.
> > > > >>>>>>> They
> > > > >>>>>>>
> > > > >>>>>>> reside in different packages today, so I don't see a problem
> > > > renaming
> > > > >>>>>>>> com.datatorrent.stram.engine as you say, even when the API
> > > cannot
> > > > be
> > > > >>>>>>>> touched right away.
> > > > >>>>>>>>
> > > > >>>>>>>> They are different packages but sharing a common prefix with
> > api
> > > > >>>>>>>> will
> > > > >>>>>>>> be
> > > > >>>>>>>>
> > > > >>>>>>>> helpful to someone new to codebase in terms of readability.
> > Not
> > > a
> > > > >>>>>>> big
> > > > >>>>>>> deal
> > > > >>>>>>> and can be changed.
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>> Thanks
> > > > >>>>>>>
> > > > >>>>>>> On Mon, Feb 27, 2017 at 7:39 AM, Thomas Weise <
> thw@apache.org>
> > > > >>>>>>>> wrote:
> > > > >>>>>>>>
> > > > >>>>>>>>> Hi,
> > > > >>>>>>>>>
> > > > >>>>>>>>> This topic has come up on several PRs and I think it
> > warrants a
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> broader
> > > > >>>>>>>>>>
> > > > >>>>>>>>> discussion.
> > > > >>>>>>>>
> > > > >>>>>>>> At the time of incubation, the decision was to defer change
> of
> > > > Java
> > > > >>>>>>>>>
> > > > >>>>>>>>>> packages from com.datatorrent to org.apache.apex till next
> > > major
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> release
> > > > >>>>>>>>>>
> > > > >>>>>>>>> to
> > > > >>>>>>>>>
> > > > >>>>>>>>> ensure backward compatibility for users.
> > > > >>>>>>>>>
> > > > >>>>>>>>>> Unfortunately that has lead to some confusion, as
> > contributors
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> continue
> > > > >>>>>>>>>>
> > > > >>>>>>>>> to
> > > > >>>>>>>>
> > > > >>>>>>>> add new code under legacy packages.
> > > > >>>>>>>>>
> > > > >>>>>>>>>> It is also a wider issue that examples for using Apex
> > continue
> > > > to
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> refer
> > > > >>>>>>>>>>
> > > > >>>>>>>>> to
> > > > >>>>>>>>
> > > > >>>>>>>> com.datatorrent packages, nearly one year after graduation.
> > More
> > > > and
> > > > >>>>>>>>>
> > > > >>>>>>>>>> more
> > > > >>>>>>>>>>
> > > > >>>>>>>>> user code is being built on top of something that needs to
> > > > change,
> > > > >>>>>>>>> the
> > > > >>>>>>>>>
> > > > >>>>>>>>> can
> > > > >>>>>>>>
> > > > >>>>>>>> is being kicked down the road and users will face more
> changes
> > > > >>>>>>>>> later.
> > > > >>>>>>>>>
> > > > >>>>>>>>>> I would like to propose the following:
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> 1. All new code has to be submitted under org.apache.apex
> > > > packages
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> 2. Not all code is under backward compatibility
> restriction
> > > and
> > > > in
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> those
> > > > >>>>>>>>>>
> > > > >>>>>>>>> cases we can rename the packages right away. Examples:
> buffer
> > > > >>>>>>>>> server,
> > > > >>>>>>>>>
> > > > >>>>>>>>> engine, demos/examples, benchmarks
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> 3. Discuss when the core API and operators can be changed.
> > For
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> operators
> > > > >>>>>>>>>>
> > > > >>>>>>>>> we
> > > > >>>>>>>>>
> > > > >>>>>>>>> have a bit more freedom to do changes before a major
> release
> > as
> > > > >>>>>>>>> most
> > > > >>>>>>>>>
> > > > >>>>>>>>>> of
> > > > >>>>>>>>>>
> > > > >>>>>>>>> them are marked @Evolving and users have the ability to
> > > continue
> > > > >>>>>>>>
> > > > >>>>>>>> using
> > > > >>>>>>>>>
> > > > >>>>>>>>> prior version of Malhar with newer engine due to engine
> > > backward
> > > > >>>>>>>>
> > > > >>>>>>>> compatibility guarantee.
> > > > >>>>>>>>>
> > > > >>>>>>>>>> Thanks,
> > > > >>>>>>>>>> Thomas
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >
> > > >
> > >
> >
>
Re: Java packages: legacy -> org.apache.apex
Posted by Pramod Immaneni <pr...@datatorrent.com>.
How about coming up with a list of @Evolving operators that we would like
to see in the eventual stable list and move those along with the not
@Evolving ones in org.apache.apex with b/w stubs and leave the rest as they
are. Then have a follow up JIRA for the rest to be moved over to contrib
and be deprecated.
Thanks
On Fri, Jul 14, 2017 at 10:37 AM, Thomas Weise <th...@gmail.com>
wrote:
> We need to keep the discussion here on topic, if other things are piled on
> top then nothing gets done.
>
> Most operators today are not stable, they are @Evolving. So perhaps it is
> necessary to have a separate discussion about that, outside of this thread.
> My guess is that there are only a few operators that we could declare
> stable. Specifically, under contrib the closest one might have been Kafka,
> but that is already superseded by the newer versions.
>
> Thomas
>
>
> On Fri, Jul 14, 2017 at 10:21 AM, Pramod Immaneni <pr...@datatorrent.com>
> wrote:
>
> > We had a discussion a while back, agreed to relegate non-stable and
> > experimental operators to contrib and also added this to contribution
> > guidelines. We aexecuted on this and cleaned up the repo by moving
> > operators deemed non-suitable for production use at that time to contrib.
> > So I wouldn't say the operators that are at the top level today or the
> ones
> > in library are 0.x.x quality. Granted, we may need to do one more pass as
> > some of the operators may no longer be considered the right
> implementations
> > with the advent of the windowed operator.
> >
> > Since contrib used to be the place that housed operators that required
> > third party libraries, there are some production quality operators in
> there
> > that need to be pulled out into top level modules.
> >
> > I think we are in agreement that for operators that we consider stable,
> we
> > should provide a b/w stub. I would add that we strongly consider creating
> > the org.apache.apex counterparts of any stable operators that are in
> > contrib out in top level modules and have the com.datatorrent stubs in
> > contrib.
> >
> > For the operators not considered stable, I would prefer we either leave
> the
> > current package path as is or if we change the package path then create
> the
> > b/w stub as I am not sure how widely they are in use (so, in essence,
> > preserve semantic versioning). It would be good if there is a followup
> JIRA
> > that takes a look at what other operators can be moved to contrib with
> the
> > advent of the newer frameworks and understanding.
> >
> > Thanks
> >
> > On Fri, Jul 14, 2017 at 9:44 AM, Thomas Weise <th...@apache.org> wrote:
> >
> > > Most of the operators evolve, as is quite clearly indicated by
> @Evolving
> > > annotations. So while perhaps 0.x.x would be a more appropriate version
> > > number, I don't think you can change that.
> > >
> > > Thomas
> > >
> > > On Fri, Jul 14, 2017 at 9:35 AM, Vlad Rozov <v....@datatorrent.com>
> > > wrote:
> > >
> > > > If entire library is not stable, its version should be 0.x.x and
> there
> > > > should not be any semantic versioning enabled or implied. It evolves.
> > If
> > > > some operators are stable as 3.8.x version suggests, the library
> should
> > > > follow semantic versioning and it is not OK to make a major change
> that
> > > > breaks semantic versioning across the entire library. It is not a
> > finding
> > > > an excuse not to make a change. For me semantic versioning and
> backward
> > > > compatibility is more important than a proper package name.
> > > >
> > > > Thank you,
> > > >
> > > > Vlad
> > > >
> > > >
> > > > On 7/14/17 09:11, Thomas Weise wrote:
> > > >
> > > >> Semantic versioning makes sense when you have a stable baseline. As
> > long
> > > >> as
> > > >> frequent fixes need to be made to address basic issues, it makes no
> > > sense
> > > >> to declare operators stable, which is why they are marked evolving.
> > > >>
> > > >> Instead of finding reasons to avoid changes that should have been
> > made a
> > > >> long time ago, why not discuss what a "stable operator" is and which
> > > ones
> > > >> deserve to be in that category. Those are the ones that will get the
> > > >> backward compatibility stub.
> > > >>
> > > >> Thanks,
> > > >> Thomas
> > > >>
> > > >>
> > > >> On Fri, Jul 14, 2017 at 8:34 AM, Vlad Rozov <
> v.rozov@datatorrent.com>
> > > >> wrote:
> > > >>
> > > >> My preference is to agree that the next Malhar release is 4.0.0 and
> > make
> > > >>> the proposed changes when 3.8.0 goes out. Otherwise why to keep
> > > semantic
> > > >>> versioning checks on Malhar in case there is no version
> > compatibility.
> > > >>>
> > > >>> Thank you,
> > > >>>
> > > >>> Vlad
> > > >>>
> > > >>>
> > > >>> On 7/14/17 08:11, Thomas Weise wrote:
> > > >>>
> > > >>> next release is 3.8.0
> > > >>>>
> > > >>>> On Fri, Jul 14, 2017 at 7:55 AM, Vlad Rozov <
> > v.rozov@datatorrent.com>
> > > >>>> wrote:
> > > >>>>
> > > >>>> next release - 3.9.0 or 4.0.0?
> > > >>>>
> > > >>>>> Thank you,
> > > >>>>>
> > > >>>>> Vlad
> > > >>>>>
> > > >>>>> On 7/13/17 22:25, Thomas Weise wrote:
> > > >>>>>
> > > >>>>> It is time to resurrect this thread and get going with the work.
> > > >>>>>
> > > >>>>>> For the next release, I will sign up to do the package move in
> > > Malhar:
> > > >>>>>>
> > > >>>>>> https://issues.apache.org/jira/browse/APEXMALHAR-2517
> > > >>>>>>
> > > >>>>>> In general this will be straightforward; most classes in Malhar
> > are
> > > >>>>>> marked
> > > >>>>>> evolving and it is trivial for users to change import
> statements.
> > > >>>>>> However,
> > > >>>>>> I would suggest to discuss if there are selected operators that
> > are
> > > >>>>>> worth
> > > >>>>>> keeping a backward compatibility stub in the old location.
> > > >>>>>>
> > > >>>>>> Here is my plan:
> > > >>>>>>
> > > >>>>>> 1. Relocate all classes in *lib* and *contrib* within one PR -
> > this
> > > is
> > > >>>>>> all
> > > >>>>>>
> > > >>>>>> IDE automated work
> > > >>>>>> 2. Add backward compatibility classes, if, any in separate PR
> > > >>>>>> 3. Create PR for Megh library to reflect moved classes
> > > >>>>>>
> > > >>>>>> Thanks,
> > > >>>>>> Thomas
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> On Wed, Mar 1, 2017 at 2:24 PM, Pramod Immaneni <
> > > >>>>>> pramod@datatorrent.com
> > > >>>>>> wrote:
> > > >>>>>>
> > > >>>>>> Inline
> > > >>>>>>
> > > >>>>>> On Mon, Feb 27, 2017 at 11:03 PM, Thomas Weise <th...@apache.org>
> > > >>>>>>> wrote:
> > > >>>>>>>
> > > >>>>>>> -->
> > > >>>>>>>
> > > >>>>>>> On Mon, Feb 27, 2017 at 1:27 PM, Pramod Immaneni <
> > > >>>>>>>> pramod@datatorrent.com
> > > >>>>>>>>
> > > >>>>>>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>> For malhar, for existing operators, I prefer we do this as
> part
> > of
> > > >>>>>>>> the
> > > >>>>>>>>
> > > >>>>>>>> planned refactoring for breaking the monolith modules into
> baby
> > > >>>>>>>>>
> > > >>>>>>>>> packages
> > > >>>>>>>>>
> > > >>>>>>>> and would also prefer deprecating the existing operators in
> > place.
> > > >>>>>>>> Refactor into smaller modules was discussed for malhar-contrib
> > and
> > > >>>>>>>> given
> > > >>>>>>>> the overall state of that module I think it is OK to defer
> > package
> > > >>>>>>>>
> > > >>>>>>>> renaming
> > > >>>>>>>>
> > > >>>>>>> there. I do however prefer to see the package rename addressed
> > for
> > > >>>>>>>
> > > >>>>>>>> other
> > > >>>>>>>> modules, especially for the main library module.
> > > >>>>>>>>
> > > >>>>>>>> Should we consider breaking the library into smaller modules
> as
> > > >>>>>>>> well,
> > > >>>>>>>>
> > > >>>>>>>> the
> > > >>>>>>> file/block operators for example probably can be in their own
> > > module
> > > >>>>>>> from
> > > >>>>>>> just an organizational perspective.
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> This
> > > >>>>>>>
> > > >>>>>>>> will help us achieve two things. First, the user will see all
> > the
> > > >>>>>>>>> new
> > > >>>>>>>>> changes at once as opposed to dealing with it twice (with
> > package
> > > >>>>>>>>>
> > > >>>>>>>>> rename
> > > >>>>>>>>>
> > > >>>>>>>> and dependency changes) and second it will allow for a
> smoother
> > > >>>>>>>> transition
> > > >>>>>>>>
> > > >>>>>>>> as the existing code will still work in a deprecated state. It
> > > will
> > > >>>>>>>>
> > > >>>>>>>>> also
> > > >>>>>>>>>
> > > >>>>>>>> give a more consistent structure to malhar. For new operators,
> > we
> > > >>>>>>>> can
> > > >>>>>>>> go
> > > >>>>>>>> with the new package path but we need to ensure they will get
> > > moved
> > > >>>>>>>> into
> > > >>>>>>>> the baby packages as well.
> > > >>>>>>>>
> > > >>>>>>>> I think existing operators should be renamed so that git
> history
> > > >>>>>>>>>
> > > >>>>>>>>> remains. A
> > > >>>>>>>>
> > > >>>>>>> possible solution for backward compatibility could be to
> > > subsequently
> > > >>>>>>>
> > > >>>>>>>> add
> > > >>>>>>>> empty subclasses in the previous location (for existing
> concrete
> > > >>>>>>>>
> > > >>>>>>>> operators
> > > >>>>>>>>
> > > >>>>>>> that we know are actually in use) to simplify migration for
> > users.
> > > >>>>>>>
> > > >>>>>>>>
> > > >>>>>>>> Yes we can do that.
> > > >>>>>>>>
> > > >>>>>>>> For demos, we can modify the paths as the apps are typically
> > used
> > > >>>>>>>
> > > >>>>>>> wholesale
> > > >>>>>>>>
> > > >>>>>>>> and the interface is typically manual interaction.
> > > >>>>>>>>
> > > >>>>>>>>> For core, if we are adding new api subsystems, like the
> > launcher
> > > >>>>>>>>> api
> > > >>>>>>>>> we
> > > >>>>>>>>> added recently for example, we can go with new package path
> but
> > > if
> > > >>>>>>>>> we
> > > >>>>>>>>>
> > > >>>>>>>>> are
> > > >>>>>>>>>
> > > >>>>>>>> making incremental additions to existing functionality, I feel
> > it
> > > is
> > > >>>>>>>> better
> > > >>>>>>>>
> > > >>>>>>>> to keep it in the same package. I also prefer we keep the
> > package
> > > of
> > > >>>>>>>>
> > > >>>>>>>>> the
> > > >>>>>>>>>
> > > >>>>>>>> implementation classes consistent with api, for
> > understandability
> > > >>>>>>>> and
> > > >>>>>>>>
> > > >>>>>>>> readability of the code. So, for example, we don't change
> > package
> > > >>>>>>>>> path
> > > >>>>>>>>>
> > > >>>>>>>>> of
> > > >>>>>>>>>
> > > >>>>>>>> LogicalPlan as it is an implementation of DAG. It is
> subjective,
> > > but
> > > >>>>>>>> it
> > > >>>>>>>>
> > > >>>>>>>> will be good if we can also do the same with classes closely
> > > related
> > > >>>>>>>>> to
> > > >>>>>>>>>
> > > >>>>>>>>> the
> > > >>>>>>>>>
> > > >>>>>>>> implementation classes as well. Maybe we can moving these on a
> > > >>>>>>>> package
> > > >>>>>>>>
> > > >>>>>>>>> by
> > > >>>>>>>>>
> > > >>>>>>>> package basis, like everything in com.datatorrent.stram.engine
> > > could
> > > >>>>>>>> be
> > > >>>>>>>>
> > > >>>>>>>> moved. For completely internal components like buffer server,
> we
> > > can
> > > >>>>>>>>>
> > > >>>>>>>>> move
> > > >>>>>>>>>
> > > >>>>>>>> them wholesale. We can consider moving all api and classes,
> when
> > > we
> > > >>>>>>>> go
> > > >>>>>>>> to
> > > >>>>>>>> next major release but would like to see if we can find a way
> to
> > > >>>>>>>> support
> > > >>>>>>>> existing api for one more major release in deprecated mode.
> > > >>>>>>>>
> > > >>>>>>>> The point of the major release is to enable backward
> > incompatible
> > > >>>>>>>>>
> > > >>>>>>>>> changes
> > > >>>>>>>> and I don't think it is realistic to support the existing API
> > for
> > > >>>>>>>> another
> > > >>>>>>>> major release. IMO it is also not necessary as most existing
> > > >>>>>>>> application
> > > >>>>>>>> code refers to operators, attributes and the application
> > > interface.
> > > >>>>>>>>
> > > >>>>>>>> Perhaps
> > > >>>>>>>>
> > > >>>>>>> it is possible to keep those around as interface extensions to
> > help
> > > >>>>>>>
> > > >>>>>>>> migration. Custom operators may need to be migrated to reflect
> > API
> > > >>>>>>>>
> > > >>>>>>>> changes,
> > > >>>>>>>>
> > > >>>>>>> and I would consider that a reasonable task for operator
> > developers
> > > >>>>>>> as
> > > >>>>>>>
> > > >>>>>>>> part
> > > >>>>>>>>
> > > >>>>>>> of a major upgrade.
> > > >>>>>>>
> > > >>>>>>>> It would be good if we can keep them as deprecated interface
> > > >>>>>>>> extensions
> > > >>>>>>>>
> > > >>>>>>>> for
> > > >>>>>>> one release to provide a smoother transition.
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> API and implementation in engine are kept separate
> intentionally.
> > > >>>>>>> They
> > > >>>>>>>
> > > >>>>>>> reside in different packages today, so I don't see a problem
> > > renaming
> > > >>>>>>>> com.datatorrent.stram.engine as you say, even when the API
> > cannot
> > > be
> > > >>>>>>>> touched right away.
> > > >>>>>>>>
> > > >>>>>>>> They are different packages but sharing a common prefix with
> api
> > > >>>>>>>> will
> > > >>>>>>>> be
> > > >>>>>>>>
> > > >>>>>>>> helpful to someone new to codebase in terms of readability.
> Not
> > a
> > > >>>>>>> big
> > > >>>>>>> deal
> > > >>>>>>> and can be changed.
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> Thanks
> > > >>>>>>>
> > > >>>>>>> On Mon, Feb 27, 2017 at 7:39 AM, Thomas Weise <th...@apache.org>
> > > >>>>>>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>>> Hi,
> > > >>>>>>>>>
> > > >>>>>>>>> This topic has come up on several PRs and I think it
> warrants a
> > > >>>>>>>>>>
> > > >>>>>>>>>> broader
> > > >>>>>>>>>>
> > > >>>>>>>>> discussion.
> > > >>>>>>>>
> > > >>>>>>>> At the time of incubation, the decision was to defer change of
> > > Java
> > > >>>>>>>>>
> > > >>>>>>>>>> packages from com.datatorrent to org.apache.apex till next
> > major
> > > >>>>>>>>>>
> > > >>>>>>>>>> release
> > > >>>>>>>>>>
> > > >>>>>>>>> to
> > > >>>>>>>>>
> > > >>>>>>>>> ensure backward compatibility for users.
> > > >>>>>>>>>
> > > >>>>>>>>>> Unfortunately that has lead to some confusion, as
> contributors
> > > >>>>>>>>>>
> > > >>>>>>>>>> continue
> > > >>>>>>>>>>
> > > >>>>>>>>> to
> > > >>>>>>>>
> > > >>>>>>>> add new code under legacy packages.
> > > >>>>>>>>>
> > > >>>>>>>>>> It is also a wider issue that examples for using Apex
> continue
> > > to
> > > >>>>>>>>>>
> > > >>>>>>>>>> refer
> > > >>>>>>>>>>
> > > >>>>>>>>> to
> > > >>>>>>>>
> > > >>>>>>>> com.datatorrent packages, nearly one year after graduation.
> More
> > > and
> > > >>>>>>>>>
> > > >>>>>>>>>> more
> > > >>>>>>>>>>
> > > >>>>>>>>> user code is being built on top of something that needs to
> > > change,
> > > >>>>>>>>> the
> > > >>>>>>>>>
> > > >>>>>>>>> can
> > > >>>>>>>>
> > > >>>>>>>> is being kicked down the road and users will face more changes
> > > >>>>>>>>> later.
> > > >>>>>>>>>
> > > >>>>>>>>>> I would like to propose the following:
> > > >>>>>>>>>>
> > > >>>>>>>>>> 1. All new code has to be submitted under org.apache.apex
> > > packages
> > > >>>>>>>>>>
> > > >>>>>>>>>> 2. Not all code is under backward compatibility restriction
> > and
> > > in
> > > >>>>>>>>>>
> > > >>>>>>>>>> those
> > > >>>>>>>>>>
> > > >>>>>>>>> cases we can rename the packages right away. Examples: buffer
> > > >>>>>>>>> server,
> > > >>>>>>>>>
> > > >>>>>>>>> engine, demos/examples, benchmarks
> > > >>>>>>>>>>
> > > >>>>>>>>>> 3. Discuss when the core API and operators can be changed.
> For
> > > >>>>>>>>>>
> > > >>>>>>>>>> operators
> > > >>>>>>>>>>
> > > >>>>>>>>> we
> > > >>>>>>>>>
> > > >>>>>>>>> have a bit more freedom to do changes before a major release
> as
> > > >>>>>>>>> most
> > > >>>>>>>>>
> > > >>>>>>>>>> of
> > > >>>>>>>>>>
> > > >>>>>>>>> them are marked @Evolving and users have the ability to
> > continue
> > > >>>>>>>>
> > > >>>>>>>> using
> > > >>>>>>>>>
> > > >>>>>>>>> prior version of Malhar with newer engine due to engine
> > backward
> > > >>>>>>>>
> > > >>>>>>>> compatibility guarantee.
> > > >>>>>>>>>
> > > >>>>>>>>>> Thanks,
> > > >>>>>>>>>> Thomas
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >
> > >
> >
>
Re: Java packages: legacy -> org.apache.apex
Posted by Thomas Weise <th...@gmail.com>.
We need to keep the discussion here on topic, if other things are piled on
top then nothing gets done.
Most operators today are not stable, they are @Evolving. So perhaps it is
necessary to have a separate discussion about that, outside of this thread.
My guess is that there are only a few operators that we could declare
stable. Specifically, under contrib the closest one might have been Kafka,
but that is already superseded by the newer versions.
Thomas
On Fri, Jul 14, 2017 at 10:21 AM, Pramod Immaneni <pr...@datatorrent.com>
wrote:
> We had a discussion a while back, agreed to relegate non-stable and
> experimental operators to contrib and also added this to contribution
> guidelines. We aexecuted on this and cleaned up the repo by moving
> operators deemed non-suitable for production use at that time to contrib.
> So I wouldn't say the operators that are at the top level today or the ones
> in library are 0.x.x quality. Granted, we may need to do one more pass as
> some of the operators may no longer be considered the right implementations
> with the advent of the windowed operator.
>
> Since contrib used to be the place that housed operators that required
> third party libraries, there are some production quality operators in there
> that need to be pulled out into top level modules.
>
> I think we are in agreement that for operators that we consider stable, we
> should provide a b/w stub. I would add that we strongly consider creating
> the org.apache.apex counterparts of any stable operators that are in
> contrib out in top level modules and have the com.datatorrent stubs in
> contrib.
>
> For the operators not considered stable, I would prefer we either leave the
> current package path as is or if we change the package path then create the
> b/w stub as I am not sure how widely they are in use (so, in essence,
> preserve semantic versioning). It would be good if there is a followup JIRA
> that takes a look at what other operators can be moved to contrib with the
> advent of the newer frameworks and understanding.
>
> Thanks
>
> On Fri, Jul 14, 2017 at 9:44 AM, Thomas Weise <th...@apache.org> wrote:
>
> > Most of the operators evolve, as is quite clearly indicated by @Evolving
> > annotations. So while perhaps 0.x.x would be a more appropriate version
> > number, I don't think you can change that.
> >
> > Thomas
> >
> > On Fri, Jul 14, 2017 at 9:35 AM, Vlad Rozov <v....@datatorrent.com>
> > wrote:
> >
> > > If entire library is not stable, its version should be 0.x.x and there
> > > should not be any semantic versioning enabled or implied. It evolves.
> If
> > > some operators are stable as 3.8.x version suggests, the library should
> > > follow semantic versioning and it is not OK to make a major change that
> > > breaks semantic versioning across the entire library. It is not a
> finding
> > > an excuse not to make a change. For me semantic versioning and backward
> > > compatibility is more important than a proper package name.
> > >
> > > Thank you,
> > >
> > > Vlad
> > >
> > >
> > > On 7/14/17 09:11, Thomas Weise wrote:
> > >
> > >> Semantic versioning makes sense when you have a stable baseline. As
> long
> > >> as
> > >> frequent fixes need to be made to address basic issues, it makes no
> > sense
> > >> to declare operators stable, which is why they are marked evolving.
> > >>
> > >> Instead of finding reasons to avoid changes that should have been
> made a
> > >> long time ago, why not discuss what a "stable operator" is and which
> > ones
> > >> deserve to be in that category. Those are the ones that will get the
> > >> backward compatibility stub.
> > >>
> > >> Thanks,
> > >> Thomas
> > >>
> > >>
> > >> On Fri, Jul 14, 2017 at 8:34 AM, Vlad Rozov <v....@datatorrent.com>
> > >> wrote:
> > >>
> > >> My preference is to agree that the next Malhar release is 4.0.0 and
> make
> > >>> the proposed changes when 3.8.0 goes out. Otherwise why to keep
> > semantic
> > >>> versioning checks on Malhar in case there is no version
> compatibility.
> > >>>
> > >>> Thank you,
> > >>>
> > >>> Vlad
> > >>>
> > >>>
> > >>> On 7/14/17 08:11, Thomas Weise wrote:
> > >>>
> > >>> next release is 3.8.0
> > >>>>
> > >>>> On Fri, Jul 14, 2017 at 7:55 AM, Vlad Rozov <
> v.rozov@datatorrent.com>
> > >>>> wrote:
> > >>>>
> > >>>> next release - 3.9.0 or 4.0.0?
> > >>>>
> > >>>>> Thank you,
> > >>>>>
> > >>>>> Vlad
> > >>>>>
> > >>>>> On 7/13/17 22:25, Thomas Weise wrote:
> > >>>>>
> > >>>>> It is time to resurrect this thread and get going with the work.
> > >>>>>
> > >>>>>> For the next release, I will sign up to do the package move in
> > Malhar:
> > >>>>>>
> > >>>>>> https://issues.apache.org/jira/browse/APEXMALHAR-2517
> > >>>>>>
> > >>>>>> In general this will be straightforward; most classes in Malhar
> are
> > >>>>>> marked
> > >>>>>> evolving and it is trivial for users to change import statements.
> > >>>>>> However,
> > >>>>>> I would suggest to discuss if there are selected operators that
> are
> > >>>>>> worth
> > >>>>>> keeping a backward compatibility stub in the old location.
> > >>>>>>
> > >>>>>> Here is my plan:
> > >>>>>>
> > >>>>>> 1. Relocate all classes in *lib* and *contrib* within one PR -
> this
> > is
> > >>>>>> all
> > >>>>>>
> > >>>>>> IDE automated work
> > >>>>>> 2. Add backward compatibility classes, if, any in separate PR
> > >>>>>> 3. Create PR for Megh library to reflect moved classes
> > >>>>>>
> > >>>>>> Thanks,
> > >>>>>> Thomas
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> On Wed, Mar 1, 2017 at 2:24 PM, Pramod Immaneni <
> > >>>>>> pramod@datatorrent.com
> > >>>>>> wrote:
> > >>>>>>
> > >>>>>> Inline
> > >>>>>>
> > >>>>>> On Mon, Feb 27, 2017 at 11:03 PM, Thomas Weise <th...@apache.org>
> > >>>>>>> wrote:
> > >>>>>>>
> > >>>>>>> -->
> > >>>>>>>
> > >>>>>>> On Mon, Feb 27, 2017 at 1:27 PM, Pramod Immaneni <
> > >>>>>>>> pramod@datatorrent.com
> > >>>>>>>>
> > >>>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>> For malhar, for existing operators, I prefer we do this as part
> of
> > >>>>>>>> the
> > >>>>>>>>
> > >>>>>>>> planned refactoring for breaking the monolith modules into baby
> > >>>>>>>>>
> > >>>>>>>>> packages
> > >>>>>>>>>
> > >>>>>>>> and would also prefer deprecating the existing operators in
> place.
> > >>>>>>>> Refactor into smaller modules was discussed for malhar-contrib
> and
> > >>>>>>>> given
> > >>>>>>>> the overall state of that module I think it is OK to defer
> package
> > >>>>>>>>
> > >>>>>>>> renaming
> > >>>>>>>>
> > >>>>>>> there. I do however prefer to see the package rename addressed
> for
> > >>>>>>>
> > >>>>>>>> other
> > >>>>>>>> modules, especially for the main library module.
> > >>>>>>>>
> > >>>>>>>> Should we consider breaking the library into smaller modules as
> > >>>>>>>> well,
> > >>>>>>>>
> > >>>>>>>> the
> > >>>>>>> file/block operators for example probably can be in their own
> > module
> > >>>>>>> from
> > >>>>>>> just an organizational perspective.
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> This
> > >>>>>>>
> > >>>>>>>> will help us achieve two things. First, the user will see all
> the
> > >>>>>>>>> new
> > >>>>>>>>> changes at once as opposed to dealing with it twice (with
> package
> > >>>>>>>>>
> > >>>>>>>>> rename
> > >>>>>>>>>
> > >>>>>>>> and dependency changes) and second it will allow for a smoother
> > >>>>>>>> transition
> > >>>>>>>>
> > >>>>>>>> as the existing code will still work in a deprecated state. It
> > will
> > >>>>>>>>
> > >>>>>>>>> also
> > >>>>>>>>>
> > >>>>>>>> give a more consistent structure to malhar. For new operators,
> we
> > >>>>>>>> can
> > >>>>>>>> go
> > >>>>>>>> with the new package path but we need to ensure they will get
> > moved
> > >>>>>>>> into
> > >>>>>>>> the baby packages as well.
> > >>>>>>>>
> > >>>>>>>> I think existing operators should be renamed so that git history
> > >>>>>>>>>
> > >>>>>>>>> remains. A
> > >>>>>>>>
> > >>>>>>> possible solution for backward compatibility could be to
> > subsequently
> > >>>>>>>
> > >>>>>>>> add
> > >>>>>>>> empty subclasses in the previous location (for existing concrete
> > >>>>>>>>
> > >>>>>>>> operators
> > >>>>>>>>
> > >>>>>>> that we know are actually in use) to simplify migration for
> users.
> > >>>>>>>
> > >>>>>>>>
> > >>>>>>>> Yes we can do that.
> > >>>>>>>>
> > >>>>>>>> For demos, we can modify the paths as the apps are typically
> used
> > >>>>>>>
> > >>>>>>> wholesale
> > >>>>>>>>
> > >>>>>>>> and the interface is typically manual interaction.
> > >>>>>>>>
> > >>>>>>>>> For core, if we are adding new api subsystems, like the
> launcher
> > >>>>>>>>> api
> > >>>>>>>>> we
> > >>>>>>>>> added recently for example, we can go with new package path but
> > if
> > >>>>>>>>> we
> > >>>>>>>>>
> > >>>>>>>>> are
> > >>>>>>>>>
> > >>>>>>>> making incremental additions to existing functionality, I feel
> it
> > is
> > >>>>>>>> better
> > >>>>>>>>
> > >>>>>>>> to keep it in the same package. I also prefer we keep the
> package
> > of
> > >>>>>>>>
> > >>>>>>>>> the
> > >>>>>>>>>
> > >>>>>>>> implementation classes consistent with api, for
> understandability
> > >>>>>>>> and
> > >>>>>>>>
> > >>>>>>>> readability of the code. So, for example, we don't change
> package
> > >>>>>>>>> path
> > >>>>>>>>>
> > >>>>>>>>> of
> > >>>>>>>>>
> > >>>>>>>> LogicalPlan as it is an implementation of DAG. It is subjective,
> > but
> > >>>>>>>> it
> > >>>>>>>>
> > >>>>>>>> will be good if we can also do the same with classes closely
> > related
> > >>>>>>>>> to
> > >>>>>>>>>
> > >>>>>>>>> the
> > >>>>>>>>>
> > >>>>>>>> implementation classes as well. Maybe we can moving these on a
> > >>>>>>>> package
> > >>>>>>>>
> > >>>>>>>>> by
> > >>>>>>>>>
> > >>>>>>>> package basis, like everything in com.datatorrent.stram.engine
> > could
> > >>>>>>>> be
> > >>>>>>>>
> > >>>>>>>> moved. For completely internal components like buffer server, we
> > can
> > >>>>>>>>>
> > >>>>>>>>> move
> > >>>>>>>>>
> > >>>>>>>> them wholesale. We can consider moving all api and classes, when
> > we
> > >>>>>>>> go
> > >>>>>>>> to
> > >>>>>>>> next major release but would like to see if we can find a way to
> > >>>>>>>> support
> > >>>>>>>> existing api for one more major release in deprecated mode.
> > >>>>>>>>
> > >>>>>>>> The point of the major release is to enable backward
> incompatible
> > >>>>>>>>>
> > >>>>>>>>> changes
> > >>>>>>>> and I don't think it is realistic to support the existing API
> for
> > >>>>>>>> another
> > >>>>>>>> major release. IMO it is also not necessary as most existing
> > >>>>>>>> application
> > >>>>>>>> code refers to operators, attributes and the application
> > interface.
> > >>>>>>>>
> > >>>>>>>> Perhaps
> > >>>>>>>>
> > >>>>>>> it is possible to keep those around as interface extensions to
> help
> > >>>>>>>
> > >>>>>>>> migration. Custom operators may need to be migrated to reflect
> API
> > >>>>>>>>
> > >>>>>>>> changes,
> > >>>>>>>>
> > >>>>>>> and I would consider that a reasonable task for operator
> developers
> > >>>>>>> as
> > >>>>>>>
> > >>>>>>>> part
> > >>>>>>>>
> > >>>>>>> of a major upgrade.
> > >>>>>>>
> > >>>>>>>> It would be good if we can keep them as deprecated interface
> > >>>>>>>> extensions
> > >>>>>>>>
> > >>>>>>>> for
> > >>>>>>> one release to provide a smoother transition.
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> API and implementation in engine are kept separate intentionally.
> > >>>>>>> They
> > >>>>>>>
> > >>>>>>> reside in different packages today, so I don't see a problem
> > renaming
> > >>>>>>>> com.datatorrent.stram.engine as you say, even when the API
> cannot
> > be
> > >>>>>>>> touched right away.
> > >>>>>>>>
> > >>>>>>>> They are different packages but sharing a common prefix with api
> > >>>>>>>> will
> > >>>>>>>> be
> > >>>>>>>>
> > >>>>>>>> helpful to someone new to codebase in terms of readability. Not
> a
> > >>>>>>> big
> > >>>>>>> deal
> > >>>>>>> and can be changed.
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> Thanks
> > >>>>>>>
> > >>>>>>> On Mon, Feb 27, 2017 at 7:39 AM, Thomas Weise <th...@apache.org>
> > >>>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> Hi,
> > >>>>>>>>>
> > >>>>>>>>> This topic has come up on several PRs and I think it warrants a
> > >>>>>>>>>>
> > >>>>>>>>>> broader
> > >>>>>>>>>>
> > >>>>>>>>> discussion.
> > >>>>>>>>
> > >>>>>>>> At the time of incubation, the decision was to defer change of
> > Java
> > >>>>>>>>>
> > >>>>>>>>>> packages from com.datatorrent to org.apache.apex till next
> major
> > >>>>>>>>>>
> > >>>>>>>>>> release
> > >>>>>>>>>>
> > >>>>>>>>> to
> > >>>>>>>>>
> > >>>>>>>>> ensure backward compatibility for users.
> > >>>>>>>>>
> > >>>>>>>>>> Unfortunately that has lead to some confusion, as contributors
> > >>>>>>>>>>
> > >>>>>>>>>> continue
> > >>>>>>>>>>
> > >>>>>>>>> to
> > >>>>>>>>
> > >>>>>>>> add new code under legacy packages.
> > >>>>>>>>>
> > >>>>>>>>>> It is also a wider issue that examples for using Apex continue
> > to
> > >>>>>>>>>>
> > >>>>>>>>>> refer
> > >>>>>>>>>>
> > >>>>>>>>> to
> > >>>>>>>>
> > >>>>>>>> com.datatorrent packages, nearly one year after graduation. More
> > and
> > >>>>>>>>>
> > >>>>>>>>>> more
> > >>>>>>>>>>
> > >>>>>>>>> user code is being built on top of something that needs to
> > change,
> > >>>>>>>>> the
> > >>>>>>>>>
> > >>>>>>>>> can
> > >>>>>>>>
> > >>>>>>>> is being kicked down the road and users will face more changes
> > >>>>>>>>> later.
> > >>>>>>>>>
> > >>>>>>>>>> I would like to propose the following:
> > >>>>>>>>>>
> > >>>>>>>>>> 1. All new code has to be submitted under org.apache.apex
> > packages
> > >>>>>>>>>>
> > >>>>>>>>>> 2. Not all code is under backward compatibility restriction
> and
> > in
> > >>>>>>>>>>
> > >>>>>>>>>> those
> > >>>>>>>>>>
> > >>>>>>>>> cases we can rename the packages right away. Examples: buffer
> > >>>>>>>>> server,
> > >>>>>>>>>
> > >>>>>>>>> engine, demos/examples, benchmarks
> > >>>>>>>>>>
> > >>>>>>>>>> 3. Discuss when the core API and operators can be changed. For
> > >>>>>>>>>>
> > >>>>>>>>>> operators
> > >>>>>>>>>>
> > >>>>>>>>> we
> > >>>>>>>>>
> > >>>>>>>>> have a bit more freedom to do changes before a major release as
> > >>>>>>>>> most
> > >>>>>>>>>
> > >>>>>>>>>> of
> > >>>>>>>>>>
> > >>>>>>>>> them are marked @Evolving and users have the ability to
> continue
> > >>>>>>>>
> > >>>>>>>> using
> > >>>>>>>>>
> > >>>>>>>>> prior version of Malhar with newer engine due to engine
> backward
> > >>>>>>>>
> > >>>>>>>> compatibility guarantee.
> > >>>>>>>>>
> > >>>>>>>>>> Thanks,
> > >>>>>>>>>> Thomas
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >
> >
>
Re: Java packages: legacy -> org.apache.apex
Posted by Pramod Immaneni <pr...@datatorrent.com>.
We had a discussion a while back, agreed to relegate non-stable and
experimental operators to contrib and also added this to contribution
guidelines. We aexecuted on this and cleaned up the repo by moving
operators deemed non-suitable for production use at that time to contrib.
So I wouldn't say the operators that are at the top level today or the ones
in library are 0.x.x quality. Granted, we may need to do one more pass as
some of the operators may no longer be considered the right implementations
with the advent of the windowed operator.
Since contrib used to be the place that housed operators that required
third party libraries, there are some production quality operators in there
that need to be pulled out into top level modules.
I think we are in agreement that for operators that we consider stable, we
should provide a b/w stub. I would add that we strongly consider creating
the org.apache.apex counterparts of any stable operators that are in
contrib out in top level modules and have the com.datatorrent stubs in
contrib.
For the operators not considered stable, I would prefer we either leave the
current package path as is or if we change the package path then create the
b/w stub as I am not sure how widely they are in use (so, in essence,
preserve semantic versioning). It would be good if there is a followup JIRA
that takes a look at what other operators can be moved to contrib with the
advent of the newer frameworks and understanding.
Thanks
On Fri, Jul 14, 2017 at 9:44 AM, Thomas Weise <th...@apache.org> wrote:
> Most of the operators evolve, as is quite clearly indicated by @Evolving
> annotations. So while perhaps 0.x.x would be a more appropriate version
> number, I don't think you can change that.
>
> Thomas
>
> On Fri, Jul 14, 2017 at 9:35 AM, Vlad Rozov <v....@datatorrent.com>
> wrote:
>
> > If entire library is not stable, its version should be 0.x.x and there
> > should not be any semantic versioning enabled or implied. It evolves. If
> > some operators are stable as 3.8.x version suggests, the library should
> > follow semantic versioning and it is not OK to make a major change that
> > breaks semantic versioning across the entire library. It is not a finding
> > an excuse not to make a change. For me semantic versioning and backward
> > compatibility is more important than a proper package name.
> >
> > Thank you,
> >
> > Vlad
> >
> >
> > On 7/14/17 09:11, Thomas Weise wrote:
> >
> >> Semantic versioning makes sense when you have a stable baseline. As long
> >> as
> >> frequent fixes need to be made to address basic issues, it makes no
> sense
> >> to declare operators stable, which is why they are marked evolving.
> >>
> >> Instead of finding reasons to avoid changes that should have been made a
> >> long time ago, why not discuss what a "stable operator" is and which
> ones
> >> deserve to be in that category. Those are the ones that will get the
> >> backward compatibility stub.
> >>
> >> Thanks,
> >> Thomas
> >>
> >>
> >> On Fri, Jul 14, 2017 at 8:34 AM, Vlad Rozov <v....@datatorrent.com>
> >> wrote:
> >>
> >> My preference is to agree that the next Malhar release is 4.0.0 and make
> >>> the proposed changes when 3.8.0 goes out. Otherwise why to keep
> semantic
> >>> versioning checks on Malhar in case there is no version compatibility.
> >>>
> >>> Thank you,
> >>>
> >>> Vlad
> >>>
> >>>
> >>> On 7/14/17 08:11, Thomas Weise wrote:
> >>>
> >>> next release is 3.8.0
> >>>>
> >>>> On Fri, Jul 14, 2017 at 7:55 AM, Vlad Rozov <v....@datatorrent.com>
> >>>> wrote:
> >>>>
> >>>> next release - 3.9.0 or 4.0.0?
> >>>>
> >>>>> Thank you,
> >>>>>
> >>>>> Vlad
> >>>>>
> >>>>> On 7/13/17 22:25, Thomas Weise wrote:
> >>>>>
> >>>>> It is time to resurrect this thread and get going with the work.
> >>>>>
> >>>>>> For the next release, I will sign up to do the package move in
> Malhar:
> >>>>>>
> >>>>>> https://issues.apache.org/jira/browse/APEXMALHAR-2517
> >>>>>>
> >>>>>> In general this will be straightforward; most classes in Malhar are
> >>>>>> marked
> >>>>>> evolving and it is trivial for users to change import statements.
> >>>>>> However,
> >>>>>> I would suggest to discuss if there are selected operators that are
> >>>>>> worth
> >>>>>> keeping a backward compatibility stub in the old location.
> >>>>>>
> >>>>>> Here is my plan:
> >>>>>>
> >>>>>> 1. Relocate all classes in *lib* and *contrib* within one PR - this
> is
> >>>>>> all
> >>>>>>
> >>>>>> IDE automated work
> >>>>>> 2. Add backward compatibility classes, if, any in separate PR
> >>>>>> 3. Create PR for Megh library to reflect moved classes
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Thomas
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Wed, Mar 1, 2017 at 2:24 PM, Pramod Immaneni <
> >>>>>> pramod@datatorrent.com
> >>>>>> wrote:
> >>>>>>
> >>>>>> Inline
> >>>>>>
> >>>>>> On Mon, Feb 27, 2017 at 11:03 PM, Thomas Weise <th...@apache.org>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>> -->
> >>>>>>>
> >>>>>>> On Mon, Feb 27, 2017 at 1:27 PM, Pramod Immaneni <
> >>>>>>>> pramod@datatorrent.com
> >>>>>>>>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>> For malhar, for existing operators, I prefer we do this as part of
> >>>>>>>> the
> >>>>>>>>
> >>>>>>>> planned refactoring for breaking the monolith modules into baby
> >>>>>>>>>
> >>>>>>>>> packages
> >>>>>>>>>
> >>>>>>>> and would also prefer deprecating the existing operators in place.
> >>>>>>>> Refactor into smaller modules was discussed for malhar-contrib and
> >>>>>>>> given
> >>>>>>>> the overall state of that module I think it is OK to defer package
> >>>>>>>>
> >>>>>>>> renaming
> >>>>>>>>
> >>>>>>> there. I do however prefer to see the package rename addressed for
> >>>>>>>
> >>>>>>>> other
> >>>>>>>> modules, especially for the main library module.
> >>>>>>>>
> >>>>>>>> Should we consider breaking the library into smaller modules as
> >>>>>>>> well,
> >>>>>>>>
> >>>>>>>> the
> >>>>>>> file/block operators for example probably can be in their own
> module
> >>>>>>> from
> >>>>>>> just an organizational perspective.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> This
> >>>>>>>
> >>>>>>>> will help us achieve two things. First, the user will see all the
> >>>>>>>>> new
> >>>>>>>>> changes at once as opposed to dealing with it twice (with package
> >>>>>>>>>
> >>>>>>>>> rename
> >>>>>>>>>
> >>>>>>>> and dependency changes) and second it will allow for a smoother
> >>>>>>>> transition
> >>>>>>>>
> >>>>>>>> as the existing code will still work in a deprecated state. It
> will
> >>>>>>>>
> >>>>>>>>> also
> >>>>>>>>>
> >>>>>>>> give a more consistent structure to malhar. For new operators, we
> >>>>>>>> can
> >>>>>>>> go
> >>>>>>>> with the new package path but we need to ensure they will get
> moved
> >>>>>>>> into
> >>>>>>>> the baby packages as well.
> >>>>>>>>
> >>>>>>>> I think existing operators should be renamed so that git history
> >>>>>>>>>
> >>>>>>>>> remains. A
> >>>>>>>>
> >>>>>>> possible solution for backward compatibility could be to
> subsequently
> >>>>>>>
> >>>>>>>> add
> >>>>>>>> empty subclasses in the previous location (for existing concrete
> >>>>>>>>
> >>>>>>>> operators
> >>>>>>>>
> >>>>>>> that we know are actually in use) to simplify migration for users.
> >>>>>>>
> >>>>>>>>
> >>>>>>>> Yes we can do that.
> >>>>>>>>
> >>>>>>>> For demos, we can modify the paths as the apps are typically used
> >>>>>>>
> >>>>>>> wholesale
> >>>>>>>>
> >>>>>>>> and the interface is typically manual interaction.
> >>>>>>>>
> >>>>>>>>> For core, if we are adding new api subsystems, like the launcher
> >>>>>>>>> api
> >>>>>>>>> we
> >>>>>>>>> added recently for example, we can go with new package path but
> if
> >>>>>>>>> we
> >>>>>>>>>
> >>>>>>>>> are
> >>>>>>>>>
> >>>>>>>> making incremental additions to existing functionality, I feel it
> is
> >>>>>>>> better
> >>>>>>>>
> >>>>>>>> to keep it in the same package. I also prefer we keep the package
> of
> >>>>>>>>
> >>>>>>>>> the
> >>>>>>>>>
> >>>>>>>> implementation classes consistent with api, for understandability
> >>>>>>>> and
> >>>>>>>>
> >>>>>>>> readability of the code. So, for example, we don't change package
> >>>>>>>>> path
> >>>>>>>>>
> >>>>>>>>> of
> >>>>>>>>>
> >>>>>>>> LogicalPlan as it is an implementation of DAG. It is subjective,
> but
> >>>>>>>> it
> >>>>>>>>
> >>>>>>>> will be good if we can also do the same with classes closely
> related
> >>>>>>>>> to
> >>>>>>>>>
> >>>>>>>>> the
> >>>>>>>>>
> >>>>>>>> implementation classes as well. Maybe we can moving these on a
> >>>>>>>> package
> >>>>>>>>
> >>>>>>>>> by
> >>>>>>>>>
> >>>>>>>> package basis, like everything in com.datatorrent.stram.engine
> could
> >>>>>>>> be
> >>>>>>>>
> >>>>>>>> moved. For completely internal components like buffer server, we
> can
> >>>>>>>>>
> >>>>>>>>> move
> >>>>>>>>>
> >>>>>>>> them wholesale. We can consider moving all api and classes, when
> we
> >>>>>>>> go
> >>>>>>>> to
> >>>>>>>> next major release but would like to see if we can find a way to
> >>>>>>>> support
> >>>>>>>> existing api for one more major release in deprecated mode.
> >>>>>>>>
> >>>>>>>> The point of the major release is to enable backward incompatible
> >>>>>>>>>
> >>>>>>>>> changes
> >>>>>>>> and I don't think it is realistic to support the existing API for
> >>>>>>>> another
> >>>>>>>> major release. IMO it is also not necessary as most existing
> >>>>>>>> application
> >>>>>>>> code refers to operators, attributes and the application
> interface.
> >>>>>>>>
> >>>>>>>> Perhaps
> >>>>>>>>
> >>>>>>> it is possible to keep those around as interface extensions to help
> >>>>>>>
> >>>>>>>> migration. Custom operators may need to be migrated to reflect API
> >>>>>>>>
> >>>>>>>> changes,
> >>>>>>>>
> >>>>>>> and I would consider that a reasonable task for operator developers
> >>>>>>> as
> >>>>>>>
> >>>>>>>> part
> >>>>>>>>
> >>>>>>> of a major upgrade.
> >>>>>>>
> >>>>>>>> It would be good if we can keep them as deprecated interface
> >>>>>>>> extensions
> >>>>>>>>
> >>>>>>>> for
> >>>>>>> one release to provide a smoother transition.
> >>>>>>>
> >>>>>>>
> >>>>>>> API and implementation in engine are kept separate intentionally.
> >>>>>>> They
> >>>>>>>
> >>>>>>> reside in different packages today, so I don't see a problem
> renaming
> >>>>>>>> com.datatorrent.stram.engine as you say, even when the API cannot
> be
> >>>>>>>> touched right away.
> >>>>>>>>
> >>>>>>>> They are different packages but sharing a common prefix with api
> >>>>>>>> will
> >>>>>>>> be
> >>>>>>>>
> >>>>>>>> helpful to someone new to codebase in terms of readability. Not a
> >>>>>>> big
> >>>>>>> deal
> >>>>>>> and can be changed.
> >>>>>>>
> >>>>>>>
> >>>>>>> Thanks
> >>>>>>>
> >>>>>>> On Mon, Feb 27, 2017 at 7:39 AM, Thomas Weise <th...@apache.org>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Hi,
> >>>>>>>>>
> >>>>>>>>> This topic has come up on several PRs and I think it warrants a
> >>>>>>>>>>
> >>>>>>>>>> broader
> >>>>>>>>>>
> >>>>>>>>> discussion.
> >>>>>>>>
> >>>>>>>> At the time of incubation, the decision was to defer change of
> Java
> >>>>>>>>>
> >>>>>>>>>> packages from com.datatorrent to org.apache.apex till next major
> >>>>>>>>>>
> >>>>>>>>>> release
> >>>>>>>>>>
> >>>>>>>>> to
> >>>>>>>>>
> >>>>>>>>> ensure backward compatibility for users.
> >>>>>>>>>
> >>>>>>>>>> Unfortunately that has lead to some confusion, as contributors
> >>>>>>>>>>
> >>>>>>>>>> continue
> >>>>>>>>>>
> >>>>>>>>> to
> >>>>>>>>
> >>>>>>>> add new code under legacy packages.
> >>>>>>>>>
> >>>>>>>>>> It is also a wider issue that examples for using Apex continue
> to
> >>>>>>>>>>
> >>>>>>>>>> refer
> >>>>>>>>>>
> >>>>>>>>> to
> >>>>>>>>
> >>>>>>>> com.datatorrent packages, nearly one year after graduation. More
> and
> >>>>>>>>>
> >>>>>>>>>> more
> >>>>>>>>>>
> >>>>>>>>> user code is being built on top of something that needs to
> change,
> >>>>>>>>> the
> >>>>>>>>>
> >>>>>>>>> can
> >>>>>>>>
> >>>>>>>> is being kicked down the road and users will face more changes
> >>>>>>>>> later.
> >>>>>>>>>
> >>>>>>>>>> I would like to propose the following:
> >>>>>>>>>>
> >>>>>>>>>> 1. All new code has to be submitted under org.apache.apex
> packages
> >>>>>>>>>>
> >>>>>>>>>> 2. Not all code is under backward compatibility restriction and
> in
> >>>>>>>>>>
> >>>>>>>>>> those
> >>>>>>>>>>
> >>>>>>>>> cases we can rename the packages right away. Examples: buffer
> >>>>>>>>> server,
> >>>>>>>>>
> >>>>>>>>> engine, demos/examples, benchmarks
> >>>>>>>>>>
> >>>>>>>>>> 3. Discuss when the core API and operators can be changed. For
> >>>>>>>>>>
> >>>>>>>>>> operators
> >>>>>>>>>>
> >>>>>>>>> we
> >>>>>>>>>
> >>>>>>>>> have a bit more freedom to do changes before a major release as
> >>>>>>>>> most
> >>>>>>>>>
> >>>>>>>>>> of
> >>>>>>>>>>
> >>>>>>>>> them are marked @Evolving and users have the ability to continue
> >>>>>>>>
> >>>>>>>> using
> >>>>>>>>>
> >>>>>>>>> prior version of Malhar with newer engine due to engine backward
> >>>>>>>>
> >>>>>>>> compatibility guarantee.
> >>>>>>>>>
> >>>>>>>>>> Thanks,
> >>>>>>>>>> Thomas
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >
>
Re: Java packages: legacy -> org.apache.apex
Posted by Thomas Weise <th...@apache.org>.
Most of the operators evolve, as is quite clearly indicated by @Evolving
annotations. So while perhaps 0.x.x would be a more appropriate version
number, I don't think you can change that.
Thomas
On Fri, Jul 14, 2017 at 9:35 AM, Vlad Rozov <v....@datatorrent.com> wrote:
> If entire library is not stable, its version should be 0.x.x and there
> should not be any semantic versioning enabled or implied. It evolves. If
> some operators are stable as 3.8.x version suggests, the library should
> follow semantic versioning and it is not OK to make a major change that
> breaks semantic versioning across the entire library. It is not a finding
> an excuse not to make a change. For me semantic versioning and backward
> compatibility is more important than a proper package name.
>
> Thank you,
>
> Vlad
>
>
> On 7/14/17 09:11, Thomas Weise wrote:
>
>> Semantic versioning makes sense when you have a stable baseline. As long
>> as
>> frequent fixes need to be made to address basic issues, it makes no sense
>> to declare operators stable, which is why they are marked evolving.
>>
>> Instead of finding reasons to avoid changes that should have been made a
>> long time ago, why not discuss what a "stable operator" is and which ones
>> deserve to be in that category. Those are the ones that will get the
>> backward compatibility stub.
>>
>> Thanks,
>> Thomas
>>
>>
>> On Fri, Jul 14, 2017 at 8:34 AM, Vlad Rozov <v....@datatorrent.com>
>> wrote:
>>
>> My preference is to agree that the next Malhar release is 4.0.0 and make
>>> the proposed changes when 3.8.0 goes out. Otherwise why to keep semantic
>>> versioning checks on Malhar in case there is no version compatibility.
>>>
>>> Thank you,
>>>
>>> Vlad
>>>
>>>
>>> On 7/14/17 08:11, Thomas Weise wrote:
>>>
>>> next release is 3.8.0
>>>>
>>>> On Fri, Jul 14, 2017 at 7:55 AM, Vlad Rozov <v....@datatorrent.com>
>>>> wrote:
>>>>
>>>> next release - 3.9.0 or 4.0.0?
>>>>
>>>>> Thank you,
>>>>>
>>>>> Vlad
>>>>>
>>>>> On 7/13/17 22:25, Thomas Weise wrote:
>>>>>
>>>>> It is time to resurrect this thread and get going with the work.
>>>>>
>>>>>> For the next release, I will sign up to do the package move in Malhar:
>>>>>>
>>>>>> https://issues.apache.org/jira/browse/APEXMALHAR-2517
>>>>>>
>>>>>> In general this will be straightforward; most classes in Malhar are
>>>>>> marked
>>>>>> evolving and it is trivial for users to change import statements.
>>>>>> However,
>>>>>> I would suggest to discuss if there are selected operators that are
>>>>>> worth
>>>>>> keeping a backward compatibility stub in the old location.
>>>>>>
>>>>>> Here is my plan:
>>>>>>
>>>>>> 1. Relocate all classes in *lib* and *contrib* within one PR - this is
>>>>>> all
>>>>>>
>>>>>> IDE automated work
>>>>>> 2. Add backward compatibility classes, if, any in separate PR
>>>>>> 3. Create PR for Megh library to reflect moved classes
>>>>>>
>>>>>> Thanks,
>>>>>> Thomas
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Mar 1, 2017 at 2:24 PM, Pramod Immaneni <
>>>>>> pramod@datatorrent.com
>>>>>> wrote:
>>>>>>
>>>>>> Inline
>>>>>>
>>>>>> On Mon, Feb 27, 2017 at 11:03 PM, Thomas Weise <th...@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>> -->
>>>>>>>
>>>>>>> On Mon, Feb 27, 2017 at 1:27 PM, Pramod Immaneni <
>>>>>>>> pramod@datatorrent.com
>>>>>>>>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> For malhar, for existing operators, I prefer we do this as part of
>>>>>>>> the
>>>>>>>>
>>>>>>>> planned refactoring for breaking the monolith modules into baby
>>>>>>>>>
>>>>>>>>> packages
>>>>>>>>>
>>>>>>>> and would also prefer deprecating the existing operators in place.
>>>>>>>> Refactor into smaller modules was discussed for malhar-contrib and
>>>>>>>> given
>>>>>>>> the overall state of that module I think it is OK to defer package
>>>>>>>>
>>>>>>>> renaming
>>>>>>>>
>>>>>>> there. I do however prefer to see the package rename addressed for
>>>>>>>
>>>>>>>> other
>>>>>>>> modules, especially for the main library module.
>>>>>>>>
>>>>>>>> Should we consider breaking the library into smaller modules as
>>>>>>>> well,
>>>>>>>>
>>>>>>>> the
>>>>>>> file/block operators for example probably can be in their own module
>>>>>>> from
>>>>>>> just an organizational perspective.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> This
>>>>>>>
>>>>>>>> will help us achieve two things. First, the user will see all the
>>>>>>>>> new
>>>>>>>>> changes at once as opposed to dealing with it twice (with package
>>>>>>>>>
>>>>>>>>> rename
>>>>>>>>>
>>>>>>>> and dependency changes) and second it will allow for a smoother
>>>>>>>> transition
>>>>>>>>
>>>>>>>> as the existing code will still work in a deprecated state. It will
>>>>>>>>
>>>>>>>>> also
>>>>>>>>>
>>>>>>>> give a more consistent structure to malhar. For new operators, we
>>>>>>>> can
>>>>>>>> go
>>>>>>>> with the new package path but we need to ensure they will get moved
>>>>>>>> into
>>>>>>>> the baby packages as well.
>>>>>>>>
>>>>>>>> I think existing operators should be renamed so that git history
>>>>>>>>>
>>>>>>>>> remains. A
>>>>>>>>
>>>>>>> possible solution for backward compatibility could be to subsequently
>>>>>>>
>>>>>>>> add
>>>>>>>> empty subclasses in the previous location (for existing concrete
>>>>>>>>
>>>>>>>> operators
>>>>>>>>
>>>>>>> that we know are actually in use) to simplify migration for users.
>>>>>>>
>>>>>>>>
>>>>>>>> Yes we can do that.
>>>>>>>>
>>>>>>>> For demos, we can modify the paths as the apps are typically used
>>>>>>>
>>>>>>> wholesale
>>>>>>>>
>>>>>>>> and the interface is typically manual interaction.
>>>>>>>>
>>>>>>>>> For core, if we are adding new api subsystems, like the launcher
>>>>>>>>> api
>>>>>>>>> we
>>>>>>>>> added recently for example, we can go with new package path but if
>>>>>>>>> we
>>>>>>>>>
>>>>>>>>> are
>>>>>>>>>
>>>>>>>> making incremental additions to existing functionality, I feel it is
>>>>>>>> better
>>>>>>>>
>>>>>>>> to keep it in the same package. I also prefer we keep the package of
>>>>>>>>
>>>>>>>>> the
>>>>>>>>>
>>>>>>>> implementation classes consistent with api, for understandability
>>>>>>>> and
>>>>>>>>
>>>>>>>> readability of the code. So, for example, we don't change package
>>>>>>>>> path
>>>>>>>>>
>>>>>>>>> of
>>>>>>>>>
>>>>>>>> LogicalPlan as it is an implementation of DAG. It is subjective, but
>>>>>>>> it
>>>>>>>>
>>>>>>>> will be good if we can also do the same with classes closely related
>>>>>>>>> to
>>>>>>>>>
>>>>>>>>> the
>>>>>>>>>
>>>>>>>> implementation classes as well. Maybe we can moving these on a
>>>>>>>> package
>>>>>>>>
>>>>>>>>> by
>>>>>>>>>
>>>>>>>> package basis, like everything in com.datatorrent.stram.engine could
>>>>>>>> be
>>>>>>>>
>>>>>>>> moved. For completely internal components like buffer server, we can
>>>>>>>>>
>>>>>>>>> move
>>>>>>>>>
>>>>>>>> them wholesale. We can consider moving all api and classes, when we
>>>>>>>> go
>>>>>>>> to
>>>>>>>> next major release but would like to see if we can find a way to
>>>>>>>> support
>>>>>>>> existing api for one more major release in deprecated mode.
>>>>>>>>
>>>>>>>> The point of the major release is to enable backward incompatible
>>>>>>>>>
>>>>>>>>> changes
>>>>>>>> and I don't think it is realistic to support the existing API for
>>>>>>>> another
>>>>>>>> major release. IMO it is also not necessary as most existing
>>>>>>>> application
>>>>>>>> code refers to operators, attributes and the application interface.
>>>>>>>>
>>>>>>>> Perhaps
>>>>>>>>
>>>>>>> it is possible to keep those around as interface extensions to help
>>>>>>>
>>>>>>>> migration. Custom operators may need to be migrated to reflect API
>>>>>>>>
>>>>>>>> changes,
>>>>>>>>
>>>>>>> and I would consider that a reasonable task for operator developers
>>>>>>> as
>>>>>>>
>>>>>>>> part
>>>>>>>>
>>>>>>> of a major upgrade.
>>>>>>>
>>>>>>>> It would be good if we can keep them as deprecated interface
>>>>>>>> extensions
>>>>>>>>
>>>>>>>> for
>>>>>>> one release to provide a smoother transition.
>>>>>>>
>>>>>>>
>>>>>>> API and implementation in engine are kept separate intentionally.
>>>>>>> They
>>>>>>>
>>>>>>> reside in different packages today, so I don't see a problem renaming
>>>>>>>> com.datatorrent.stram.engine as you say, even when the API cannot be
>>>>>>>> touched right away.
>>>>>>>>
>>>>>>>> They are different packages but sharing a common prefix with api
>>>>>>>> will
>>>>>>>> be
>>>>>>>>
>>>>>>>> helpful to someone new to codebase in terms of readability. Not a
>>>>>>> big
>>>>>>> deal
>>>>>>> and can be changed.
>>>>>>>
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>> On Mon, Feb 27, 2017 at 7:39 AM, Thomas Weise <th...@apache.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> This topic has come up on several PRs and I think it warrants a
>>>>>>>>>>
>>>>>>>>>> broader
>>>>>>>>>>
>>>>>>>>> discussion.
>>>>>>>>
>>>>>>>> At the time of incubation, the decision was to defer change of Java
>>>>>>>>>
>>>>>>>>>> packages from com.datatorrent to org.apache.apex till next major
>>>>>>>>>>
>>>>>>>>>> release
>>>>>>>>>>
>>>>>>>>> to
>>>>>>>>>
>>>>>>>>> ensure backward compatibility for users.
>>>>>>>>>
>>>>>>>>>> Unfortunately that has lead to some confusion, as contributors
>>>>>>>>>>
>>>>>>>>>> continue
>>>>>>>>>>
>>>>>>>>> to
>>>>>>>>
>>>>>>>> add new code under legacy packages.
>>>>>>>>>
>>>>>>>>>> It is also a wider issue that examples for using Apex continue to
>>>>>>>>>>
>>>>>>>>>> refer
>>>>>>>>>>
>>>>>>>>> to
>>>>>>>>
>>>>>>>> com.datatorrent packages, nearly one year after graduation. More and
>>>>>>>>>
>>>>>>>>>> more
>>>>>>>>>>
>>>>>>>>> user code is being built on top of something that needs to change,
>>>>>>>>> the
>>>>>>>>>
>>>>>>>>> can
>>>>>>>>
>>>>>>>> is being kicked down the road and users will face more changes
>>>>>>>>> later.
>>>>>>>>>
>>>>>>>>>> I would like to propose the following:
>>>>>>>>>>
>>>>>>>>>> 1. All new code has to be submitted under org.apache.apex packages
>>>>>>>>>>
>>>>>>>>>> 2. Not all code is under backward compatibility restriction and in
>>>>>>>>>>
>>>>>>>>>> those
>>>>>>>>>>
>>>>>>>>> cases we can rename the packages right away. Examples: buffer
>>>>>>>>> server,
>>>>>>>>>
>>>>>>>>> engine, demos/examples, benchmarks
>>>>>>>>>>
>>>>>>>>>> 3. Discuss when the core API and operators can be changed. For
>>>>>>>>>>
>>>>>>>>>> operators
>>>>>>>>>>
>>>>>>>>> we
>>>>>>>>>
>>>>>>>>> have a bit more freedom to do changes before a major release as
>>>>>>>>> most
>>>>>>>>>
>>>>>>>>>> of
>>>>>>>>>>
>>>>>>>>> them are marked @Evolving and users have the ability to continue
>>>>>>>>
>>>>>>>> using
>>>>>>>>>
>>>>>>>>> prior version of Malhar with newer engine due to engine backward
>>>>>>>>
>>>>>>>> compatibility guarantee.
>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Thomas
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>
Re: Java packages: legacy -> org.apache.apex
Posted by Vlad Rozov <v....@datatorrent.com>.
If entire library is not stable, its version should be 0.x.x and there
should not be any semantic versioning enabled or implied. It evolves. If
some operators are stable as 3.8.x version suggests, the library should
follow semantic versioning and it is not OK to make a major change that
breaks semantic versioning across the entire library. It is not a
finding an excuse not to make a change. For me semantic versioning and
backward compatibility is more important than a proper package name.
Thank you,
Vlad
On 7/14/17 09:11, Thomas Weise wrote:
> Semantic versioning makes sense when you have a stable baseline. As long as
> frequent fixes need to be made to address basic issues, it makes no sense
> to declare operators stable, which is why they are marked evolving.
>
> Instead of finding reasons to avoid changes that should have been made a
> long time ago, why not discuss what a "stable operator" is and which ones
> deserve to be in that category. Those are the ones that will get the
> backward compatibility stub.
>
> Thanks,
> Thomas
>
>
> On Fri, Jul 14, 2017 at 8:34 AM, Vlad Rozov <v....@datatorrent.com> wrote:
>
>> My preference is to agree that the next Malhar release is 4.0.0 and make
>> the proposed changes when 3.8.0 goes out. Otherwise why to keep semantic
>> versioning checks on Malhar in case there is no version compatibility.
>>
>> Thank you,
>>
>> Vlad
>>
>>
>> On 7/14/17 08:11, Thomas Weise wrote:
>>
>>> next release is 3.8.0
>>>
>>> On Fri, Jul 14, 2017 at 7:55 AM, Vlad Rozov <v....@datatorrent.com>
>>> wrote:
>>>
>>> next release - 3.9.0 or 4.0.0?
>>>> Thank you,
>>>>
>>>> Vlad
>>>>
>>>> On 7/13/17 22:25, Thomas Weise wrote:
>>>>
>>>> It is time to resurrect this thread and get going with the work.
>>>>> For the next release, I will sign up to do the package move in Malhar:
>>>>>
>>>>> https://issues.apache.org/jira/browse/APEXMALHAR-2517
>>>>>
>>>>> In general this will be straightforward; most classes in Malhar are
>>>>> marked
>>>>> evolving and it is trivial for users to change import statements.
>>>>> However,
>>>>> I would suggest to discuss if there are selected operators that are
>>>>> worth
>>>>> keeping a backward compatibility stub in the old location.
>>>>>
>>>>> Here is my plan:
>>>>>
>>>>> 1. Relocate all classes in *lib* and *contrib* within one PR - this is
>>>>> all
>>>>>
>>>>> IDE automated work
>>>>> 2. Add backward compatibility classes, if, any in separate PR
>>>>> 3. Create PR for Megh library to reflect moved classes
>>>>>
>>>>> Thanks,
>>>>> Thomas
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Mar 1, 2017 at 2:24 PM, Pramod Immaneni <pramod@datatorrent.com
>>>>> wrote:
>>>>>
>>>>> Inline
>>>>>
>>>>>> On Mon, Feb 27, 2017 at 11:03 PM, Thomas Weise <th...@apache.org> wrote:
>>>>>>
>>>>>> -->
>>>>>>
>>>>>>> On Mon, Feb 27, 2017 at 1:27 PM, Pramod Immaneni <
>>>>>>> pramod@datatorrent.com
>>>>>>>
>>>>>>> wrote:
>>>>>>>
>>>>>>> For malhar, for existing operators, I prefer we do this as part of the
>>>>>>>
>>>>>>>> planned refactoring for breaking the monolith modules into baby
>>>>>>>>
>>>>>>>> packages
>>>>>>> and would also prefer deprecating the existing operators in place.
>>>>>>> Refactor into smaller modules was discussed for malhar-contrib and
>>>>>>> given
>>>>>>> the overall state of that module I think it is OK to defer package
>>>>>>>
>>>>>>> renaming
>>>>>> there. I do however prefer to see the package rename addressed for
>>>>>>> other
>>>>>>> modules, especially for the main library module.
>>>>>>>
>>>>>>> Should we consider breaking the library into smaller modules as well,
>>>>>>>
>>>>>> the
>>>>>> file/block operators for example probably can be in their own module
>>>>>> from
>>>>>> just an organizational perspective.
>>>>>>
>>>>>>
>>>>>>
>>>>>> This
>>>>>>>> will help us achieve two things. First, the user will see all the new
>>>>>>>> changes at once as opposed to dealing with it twice (with package
>>>>>>>>
>>>>>>>> rename
>>>>>>> and dependency changes) and second it will allow for a smoother
>>>>>>> transition
>>>>>>>
>>>>>>> as the existing code will still work in a deprecated state. It will
>>>>>>>> also
>>>>>>> give a more consistent structure to malhar. For new operators, we can
>>>>>>> go
>>>>>>> with the new package path but we need to ensure they will get moved
>>>>>>> into
>>>>>>> the baby packages as well.
>>>>>>>
>>>>>>>> I think existing operators should be renamed so that git history
>>>>>>>>
>>>>>>> remains. A
>>>>>> possible solution for backward compatibility could be to subsequently
>>>>>>> add
>>>>>>> empty subclasses in the previous location (for existing concrete
>>>>>>>
>>>>>>> operators
>>>>>> that we know are actually in use) to simplify migration for users.
>>>>>>>
>>>>>>> Yes we can do that.
>>>>>>>
>>>>>> For demos, we can modify the paths as the apps are typically used
>>>>>>
>>>>>>> wholesale
>>>>>>>
>>>>>>> and the interface is typically manual interaction.
>>>>>>>> For core, if we are adding new api subsystems, like the launcher api
>>>>>>>> we
>>>>>>>> added recently for example, we can go with new package path but if we
>>>>>>>>
>>>>>>>> are
>>>>>>> making incremental additions to existing functionality, I feel it is
>>>>>>> better
>>>>>>>
>>>>>>> to keep it in the same package. I also prefer we keep the package of
>>>>>>>> the
>>>>>>> implementation classes consistent with api, for understandability and
>>>>>>>
>>>>>>>> readability of the code. So, for example, we don't change package
>>>>>>>> path
>>>>>>>>
>>>>>>>> of
>>>>>>> LogicalPlan as it is an implementation of DAG. It is subjective, but
>>>>>>> it
>>>>>>>
>>>>>>>> will be good if we can also do the same with classes closely related
>>>>>>>> to
>>>>>>>>
>>>>>>>> the
>>>>>>> implementation classes as well. Maybe we can moving these on a package
>>>>>>>> by
>>>>>>> package basis, like everything in com.datatorrent.stram.engine could
>>>>>>> be
>>>>>>>
>>>>>>>> moved. For completely internal components like buffer server, we can
>>>>>>>>
>>>>>>>> move
>>>>>>> them wholesale. We can consider moving all api and classes, when we go
>>>>>>> to
>>>>>>> next major release but would like to see if we can find a way to
>>>>>>> support
>>>>>>> existing api for one more major release in deprecated mode.
>>>>>>>
>>>>>>>> The point of the major release is to enable backward incompatible
>>>>>>>>
>>>>>>> changes
>>>>>>> and I don't think it is realistic to support the existing API for
>>>>>>> another
>>>>>>> major release. IMO it is also not necessary as most existing
>>>>>>> application
>>>>>>> code refers to operators, attributes and the application interface.
>>>>>>>
>>>>>>> Perhaps
>>>>>> it is possible to keep those around as interface extensions to help
>>>>>>> migration. Custom operators may need to be migrated to reflect API
>>>>>>>
>>>>>>> changes,
>>>>>> and I would consider that a reasonable task for operator developers as
>>>>>>> part
>>>>>> of a major upgrade.
>>>>>>> It would be good if we can keep them as deprecated interface
>>>>>>> extensions
>>>>>>>
>>>>>> for
>>>>>> one release to provide a smoother transition.
>>>>>>
>>>>>>
>>>>>> API and implementation in engine are kept separate intentionally. They
>>>>>>
>>>>>>> reside in different packages today, so I don't see a problem renaming
>>>>>>> com.datatorrent.stram.engine as you say, even when the API cannot be
>>>>>>> touched right away.
>>>>>>>
>>>>>>> They are different packages but sharing a common prefix with api will
>>>>>>> be
>>>>>>>
>>>>>> helpful to someone new to codebase in terms of readability. Not a big
>>>>>> deal
>>>>>> and can be changed.
>>>>>>
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>>> On Mon, Feb 27, 2017 at 7:39 AM, Thomas Weise <th...@apache.org> wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>>> This topic has come up on several PRs and I think it warrants a
>>>>>>>>>
>>>>>>>>> broader
>>>>>>> discussion.
>>>>>>>
>>>>>>>> At the time of incubation, the decision was to defer change of Java
>>>>>>>>> packages from com.datatorrent to org.apache.apex till next major
>>>>>>>>>
>>>>>>>>> release
>>>>>>>> to
>>>>>>>>
>>>>>>>> ensure backward compatibility for users.
>>>>>>>>> Unfortunately that has lead to some confusion, as contributors
>>>>>>>>>
>>>>>>>>> continue
>>>>>>> to
>>>>>>>
>>>>>>>> add new code under legacy packages.
>>>>>>>>> It is also a wider issue that examples for using Apex continue to
>>>>>>>>>
>>>>>>>>> refer
>>>>>>> to
>>>>>>>
>>>>>>>> com.datatorrent packages, nearly one year after graduation. More and
>>>>>>>>> more
>>>>>>>> user code is being built on top of something that needs to change,
>>>>>>>> the
>>>>>>>>
>>>>>>> can
>>>>>>>
>>>>>>>> is being kicked down the road and users will face more changes later.
>>>>>>>>> I would like to propose the following:
>>>>>>>>>
>>>>>>>>> 1. All new code has to be submitted under org.apache.apex packages
>>>>>>>>>
>>>>>>>>> 2. Not all code is under backward compatibility restriction and in
>>>>>>>>>
>>>>>>>>> those
>>>>>>>> cases we can rename the packages right away. Examples: buffer server,
>>>>>>>>
>>>>>>>>> engine, demos/examples, benchmarks
>>>>>>>>>
>>>>>>>>> 3. Discuss when the core API and operators can be changed. For
>>>>>>>>>
>>>>>>>>> operators
>>>>>>>> we
>>>>>>>>
>>>>>>>> have a bit more freedom to do changes before a major release as most
>>>>>>>>> of
>>>>>>> them are marked @Evolving and users have the ability to continue
>>>>>>>
>>>>>>>> using
>>>>>>>>
>>>>>>> prior version of Malhar with newer engine due to engine backward
>>>>>>>
>>>>>>>> compatibility guarantee.
>>>>>>>>> Thanks,
>>>>>>>>> Thomas
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
Re: Java packages: legacy -> org.apache.apex
Posted by Thomas Weise <th...@apache.org>.
Semantic versioning makes sense when you have a stable baseline. As long as
frequent fixes need to be made to address basic issues, it makes no sense
to declare operators stable, which is why they are marked evolving.
Instead of finding reasons to avoid changes that should have been made a
long time ago, why not discuss what a "stable operator" is and which ones
deserve to be in that category. Those are the ones that will get the
backward compatibility stub.
Thanks,
Thomas
On Fri, Jul 14, 2017 at 8:34 AM, Vlad Rozov <v....@datatorrent.com> wrote:
> My preference is to agree that the next Malhar release is 4.0.0 and make
> the proposed changes when 3.8.0 goes out. Otherwise why to keep semantic
> versioning checks on Malhar in case there is no version compatibility.
>
> Thank you,
>
> Vlad
>
>
> On 7/14/17 08:11, Thomas Weise wrote:
>
>> next release is 3.8.0
>>
>> On Fri, Jul 14, 2017 at 7:55 AM, Vlad Rozov <v....@datatorrent.com>
>> wrote:
>>
>> next release - 3.9.0 or 4.0.0?
>>>
>>> Thank you,
>>>
>>> Vlad
>>>
>>> On 7/13/17 22:25, Thomas Weise wrote:
>>>
>>> It is time to resurrect this thread and get going with the work.
>>>>
>>>> For the next release, I will sign up to do the package move in Malhar:
>>>>
>>>> https://issues.apache.org/jira/browse/APEXMALHAR-2517
>>>>
>>>> In general this will be straightforward; most classes in Malhar are
>>>> marked
>>>> evolving and it is trivial for users to change import statements.
>>>> However,
>>>> I would suggest to discuss if there are selected operators that are
>>>> worth
>>>> keeping a backward compatibility stub in the old location.
>>>>
>>>> Here is my plan:
>>>>
>>>> 1. Relocate all classes in *lib* and *contrib* within one PR - this is
>>>> all
>>>>
>>>> IDE automated work
>>>> 2. Add backward compatibility classes, if, any in separate PR
>>>> 3. Create PR for Megh library to reflect moved classes
>>>>
>>>> Thanks,
>>>> Thomas
>>>>
>>>>
>>>>
>>>> On Wed, Mar 1, 2017 at 2:24 PM, Pramod Immaneni <pramod@datatorrent.com
>>>> >
>>>> wrote:
>>>>
>>>> Inline
>>>>
>>>>> On Mon, Feb 27, 2017 at 11:03 PM, Thomas Weise <th...@apache.org> wrote:
>>>>>
>>>>> -->
>>>>>
>>>>>> On Mon, Feb 27, 2017 at 1:27 PM, Pramod Immaneni <
>>>>>> pramod@datatorrent.com
>>>>>>
>>>>>> wrote:
>>>>>>
>>>>>> For malhar, for existing operators, I prefer we do this as part of the
>>>>>>
>>>>>>> planned refactoring for breaking the monolith modules into baby
>>>>>>>
>>>>>>> packages
>>>>>> and would also prefer deprecating the existing operators in place.
>>>>>> Refactor into smaller modules was discussed for malhar-contrib and
>>>>>> given
>>>>>> the overall state of that module I think it is OK to defer package
>>>>>>
>>>>>> renaming
>>>>>
>>>>> there. I do however prefer to see the package rename addressed for
>>>>>> other
>>>>>> modules, especially for the main library module.
>>>>>>
>>>>>> Should we consider breaking the library into smaller modules as well,
>>>>>>
>>>>> the
>>>>> file/block operators for example probably can be in their own module
>>>>> from
>>>>> just an organizational perspective.
>>>>>
>>>>>
>>>>>
>>>>> This
>>>>>>
>>>>>>> will help us achieve two things. First, the user will see all the new
>>>>>>> changes at once as opposed to dealing with it twice (with package
>>>>>>>
>>>>>>> rename
>>>>>> and dependency changes) and second it will allow for a smoother
>>>>>> transition
>>>>>>
>>>>>> as the existing code will still work in a deprecated state. It will
>>>>>>>
>>>>>>> also
>>>>>> give a more consistent structure to malhar. For new operators, we can
>>>>>> go
>>>>>> with the new package path but we need to ensure they will get moved
>>>>>> into
>>>>>> the baby packages as well.
>>>>>>
>>>>>>> I think existing operators should be renamed so that git history
>>>>>>>
>>>>>> remains. A
>>>>>
>>>>> possible solution for backward compatibility could be to subsequently
>>>>>> add
>>>>>> empty subclasses in the previous location (for existing concrete
>>>>>>
>>>>>> operators
>>>>>
>>>>> that we know are actually in use) to simplify migration for users.
>>>>>>
>>>>>>
>>>>>> Yes we can do that.
>>>>>>
>>>>>
>>>>> For demos, we can modify the paths as the apps are typically used
>>>>>
>>>>>> wholesale
>>>>>>
>>>>>> and the interface is typically manual interaction.
>>>>>>>
>>>>>>> For core, if we are adding new api subsystems, like the launcher api
>>>>>>> we
>>>>>>> added recently for example, we can go with new package path but if we
>>>>>>>
>>>>>>> are
>>>>>> making incremental additions to existing functionality, I feel it is
>>>>>> better
>>>>>>
>>>>>> to keep it in the same package. I also prefer we keep the package of
>>>>>>>
>>>>>>> the
>>>>>> implementation classes consistent with api, for understandability and
>>>>>>
>>>>>>> readability of the code. So, for example, we don't change package
>>>>>>> path
>>>>>>>
>>>>>>> of
>>>>>> LogicalPlan as it is an implementation of DAG. It is subjective, but
>>>>>> it
>>>>>>
>>>>>>> will be good if we can also do the same with classes closely related
>>>>>>> to
>>>>>>>
>>>>>>> the
>>>>>>
>>>>>> implementation classes as well. Maybe we can moving these on a package
>>>>>>>
>>>>>>> by
>>>>>> package basis, like everything in com.datatorrent.stram.engine could
>>>>>> be
>>>>>>
>>>>>>> moved. For completely internal components like buffer server, we can
>>>>>>>
>>>>>>> move
>>>>>> them wholesale. We can consider moving all api and classes, when we go
>>>>>> to
>>>>>> next major release but would like to see if we can find a way to
>>>>>> support
>>>>>> existing api for one more major release in deprecated mode.
>>>>>>
>>>>>>>
>>>>>>> The point of the major release is to enable backward incompatible
>>>>>>>
>>>>>> changes
>>>>>> and I don't think it is realistic to support the existing API for
>>>>>> another
>>>>>> major release. IMO it is also not necessary as most existing
>>>>>> application
>>>>>> code refers to operators, attributes and the application interface.
>>>>>>
>>>>>> Perhaps
>>>>>
>>>>> it is possible to keep those around as interface extensions to help
>>>>>> migration. Custom operators may need to be migrated to reflect API
>>>>>>
>>>>>> changes,
>>>>>
>>>>> and I would consider that a reasonable task for operator developers as
>>>>>>
>>>>>> part
>>>>>
>>>>> of a major upgrade.
>>>>>>
>>>>>> It would be good if we can keep them as deprecated interface
>>>>>> extensions
>>>>>>
>>>>> for
>>>>> one release to provide a smoother transition.
>>>>>
>>>>>
>>>>> API and implementation in engine are kept separate intentionally. They
>>>>>
>>>>>> reside in different packages today, so I don't see a problem renaming
>>>>>> com.datatorrent.stram.engine as you say, even when the API cannot be
>>>>>> touched right away.
>>>>>>
>>>>>> They are different packages but sharing a common prefix with api will
>>>>>> be
>>>>>>
>>>>> helpful to someone new to codebase in terms of readability. Not a big
>>>>> deal
>>>>> and can be changed.
>>>>>
>>>>>
>>>>> Thanks
>>>>>
>>>>>> On Mon, Feb 27, 2017 at 7:39 AM, Thomas Weise <th...@apache.org> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>>> This topic has come up on several PRs and I think it warrants a
>>>>>>>>
>>>>>>>> broader
>>>>>>>
>>>>>> discussion.
>>>>>>
>>>>>>> At the time of incubation, the decision was to defer change of Java
>>>>>>>> packages from com.datatorrent to org.apache.apex till next major
>>>>>>>>
>>>>>>>> release
>>>>>>> to
>>>>>>>
>>>>>>> ensure backward compatibility for users.
>>>>>>>>
>>>>>>>> Unfortunately that has lead to some confusion, as contributors
>>>>>>>>
>>>>>>>> continue
>>>>>>>
>>>>>> to
>>>>>>
>>>>>>> add new code under legacy packages.
>>>>>>>>
>>>>>>>> It is also a wider issue that examples for using Apex continue to
>>>>>>>>
>>>>>>>> refer
>>>>>>>
>>>>>> to
>>>>>>
>>>>>>> com.datatorrent packages, nearly one year after graduation. More and
>>>>>>>>
>>>>>>>> more
>>>>>>> user code is being built on top of something that needs to change,
>>>>>>> the
>>>>>>>
>>>>>> can
>>>>>>
>>>>>>> is being kicked down the road and users will face more changes later.
>>>>>>>>
>>>>>>>> I would like to propose the following:
>>>>>>>>
>>>>>>>> 1. All new code has to be submitted under org.apache.apex packages
>>>>>>>>
>>>>>>>> 2. Not all code is under backward compatibility restriction and in
>>>>>>>>
>>>>>>>> those
>>>>>>> cases we can rename the packages right away. Examples: buffer server,
>>>>>>>
>>>>>>>> engine, demos/examples, benchmarks
>>>>>>>>
>>>>>>>> 3. Discuss when the core API and operators can be changed. For
>>>>>>>>
>>>>>>>> operators
>>>>>>> we
>>>>>>>
>>>>>>> have a bit more freedom to do changes before a major release as most
>>>>>>>>
>>>>>>>> of
>>>>>>>
>>>>>> them are marked @Evolving and users have the ability to continue
>>>>>>
>>>>>>> using
>>>>>>>
>>>>>> prior version of Malhar with newer engine due to engine backward
>>>>>>
>>>>>>> compatibility guarantee.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Thomas
>>>>>>>>
>>>>>>>>
>>>>>>>>
>
Re: Java packages: legacy -> org.apache.apex
Posted by Vlad Rozov <v....@datatorrent.com>.
My preference is to agree that the next Malhar release is 4.0.0 and make
the proposed changes when 3.8.0 goes out. Otherwise why to keep semantic
versioning checks on Malhar in case there is no version compatibility.
Thank you,
Vlad
On 7/14/17 08:11, Thomas Weise wrote:
> next release is 3.8.0
>
> On Fri, Jul 14, 2017 at 7:55 AM, Vlad Rozov <v....@datatorrent.com> wrote:
>
>> next release - 3.9.0 or 4.0.0?
>>
>> Thank you,
>>
>> Vlad
>>
>> On 7/13/17 22:25, Thomas Weise wrote:
>>
>>> It is time to resurrect this thread and get going with the work.
>>>
>>> For the next release, I will sign up to do the package move in Malhar:
>>>
>>> https://issues.apache.org/jira/browse/APEXMALHAR-2517
>>>
>>> In general this will be straightforward; most classes in Malhar are marked
>>> evolving and it is trivial for users to change import statements. However,
>>> I would suggest to discuss if there are selected operators that are worth
>>> keeping a backward compatibility stub in the old location.
>>>
>>> Here is my plan:
>>>
>>> 1. Relocate all classes in *lib* and *contrib* within one PR - this is all
>>>
>>> IDE automated work
>>> 2. Add backward compatibility classes, if, any in separate PR
>>> 3. Create PR for Megh library to reflect moved classes
>>>
>>> Thanks,
>>> Thomas
>>>
>>>
>>>
>>> On Wed, Mar 1, 2017 at 2:24 PM, Pramod Immaneni <pr...@datatorrent.com>
>>> wrote:
>>>
>>> Inline
>>>> On Mon, Feb 27, 2017 at 11:03 PM, Thomas Weise <th...@apache.org> wrote:
>>>>
>>>> -->
>>>>> On Mon, Feb 27, 2017 at 1:27 PM, Pramod Immaneni <
>>>>> pramod@datatorrent.com
>>>>>
>>>>> wrote:
>>>>>
>>>>> For malhar, for existing operators, I prefer we do this as part of the
>>>>>> planned refactoring for breaking the monolith modules into baby
>>>>>>
>>>>> packages
>>>>> and would also prefer deprecating the existing operators in place.
>>>>> Refactor into smaller modules was discussed for malhar-contrib and given
>>>>> the overall state of that module I think it is OK to defer package
>>>>>
>>>> renaming
>>>>
>>>>> there. I do however prefer to see the package rename addressed for other
>>>>> modules, especially for the main library module.
>>>>>
>>>>> Should we consider breaking the library into smaller modules as well,
>>>> the
>>>> file/block operators for example probably can be in their own module from
>>>> just an organizational perspective.
>>>>
>>>>
>>>>
>>>>> This
>>>>>> will help us achieve two things. First, the user will see all the new
>>>>>> changes at once as opposed to dealing with it twice (with package
>>>>>>
>>>>> rename
>>>>> and dependency changes) and second it will allow for a smoother
>>>>> transition
>>>>>
>>>>>> as the existing code will still work in a deprecated state. It will
>>>>>>
>>>>> also
>>>>> give a more consistent structure to malhar. For new operators, we can
>>>>> go
>>>>> with the new package path but we need to ensure they will get moved
>>>>> into
>>>>> the baby packages as well.
>>>>>> I think existing operators should be renamed so that git history
>>>> remains. A
>>>>
>>>>> possible solution for backward compatibility could be to subsequently
>>>>> add
>>>>> empty subclasses in the previous location (for existing concrete
>>>>>
>>>> operators
>>>>
>>>>> that we know are actually in use) to simplify migration for users.
>>>>>
>>>>>
>>>>> Yes we can do that.
>>>>
>>>> For demos, we can modify the paths as the apps are typically used
>>>>> wholesale
>>>>>
>>>>>> and the interface is typically manual interaction.
>>>>>>
>>>>>> For core, if we are adding new api subsystems, like the launcher api we
>>>>>> added recently for example, we can go with new package path but if we
>>>>>>
>>>>> are
>>>>> making incremental additions to existing functionality, I feel it is
>>>>> better
>>>>>
>>>>>> to keep it in the same package. I also prefer we keep the package of
>>>>>>
>>>>> the
>>>>> implementation classes consistent with api, for understandability and
>>>>>> readability of the code. So, for example, we don't change package path
>>>>>>
>>>>> of
>>>>> LogicalPlan as it is an implementation of DAG. It is subjective, but it
>>>>>> will be good if we can also do the same with classes closely related to
>>>>>>
>>>>> the
>>>>>
>>>>>> implementation classes as well. Maybe we can moving these on a package
>>>>>>
>>>>> by
>>>>> package basis, like everything in com.datatorrent.stram.engine could be
>>>>>> moved. For completely internal components like buffer server, we can
>>>>>>
>>>>> move
>>>>> them wholesale. We can consider moving all api and classes, when we go
>>>>> to
>>>>> next major release but would like to see if we can find a way to
>>>>> support
>>>>> existing api for one more major release in deprecated mode.
>>>>>>
>>>>>> The point of the major release is to enable backward incompatible
>>>>> changes
>>>>> and I don't think it is realistic to support the existing API for
>>>>> another
>>>>> major release. IMO it is also not necessary as most existing application
>>>>> code refers to operators, attributes and the application interface.
>>>>>
>>>> Perhaps
>>>>
>>>>> it is possible to keep those around as interface extensions to help
>>>>> migration. Custom operators may need to be migrated to reflect API
>>>>>
>>>> changes,
>>>>
>>>>> and I would consider that a reasonable task for operator developers as
>>>>>
>>>> part
>>>>
>>>>> of a major upgrade.
>>>>>
>>>>> It would be good if we can keep them as deprecated interface extensions
>>>> for
>>>> one release to provide a smoother transition.
>>>>
>>>>
>>>> API and implementation in engine are kept separate intentionally. They
>>>>> reside in different packages today, so I don't see a problem renaming
>>>>> com.datatorrent.stram.engine as you say, even when the API cannot be
>>>>> touched right away.
>>>>>
>>>>> They are different packages but sharing a common prefix with api will be
>>>> helpful to someone new to codebase in terms of readability. Not a big
>>>> deal
>>>> and can be changed.
>>>>
>>>>
>>>> Thanks
>>>>>> On Mon, Feb 27, 2017 at 7:39 AM, Thomas Weise <th...@apache.org> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>> This topic has come up on several PRs and I think it warrants a
>>>>>>>
>>>>>> broader
>>>>> discussion.
>>>>>>> At the time of incubation, the decision was to defer change of Java
>>>>>>> packages from com.datatorrent to org.apache.apex till next major
>>>>>>>
>>>>>> release
>>>>>> to
>>>>>>
>>>>>>> ensure backward compatibility for users.
>>>>>>>
>>>>>>> Unfortunately that has lead to some confusion, as contributors
>>>>>>>
>>>>>> continue
>>>>> to
>>>>>>> add new code under legacy packages.
>>>>>>>
>>>>>>> It is also a wider issue that examples for using Apex continue to
>>>>>>>
>>>>>> refer
>>>>> to
>>>>>>> com.datatorrent packages, nearly one year after graduation. More and
>>>>>>>
>>>>>> more
>>>>>> user code is being built on top of something that needs to change,
>>>>>> the
>>>>> can
>>>>>>> is being kicked down the road and users will face more changes later.
>>>>>>>
>>>>>>> I would like to propose the following:
>>>>>>>
>>>>>>> 1. All new code has to be submitted under org.apache.apex packages
>>>>>>>
>>>>>>> 2. Not all code is under backward compatibility restriction and in
>>>>>>>
>>>>>> those
>>>>>> cases we can rename the packages right away. Examples: buffer server,
>>>>>>> engine, demos/examples, benchmarks
>>>>>>>
>>>>>>> 3. Discuss when the core API and operators can be changed. For
>>>>>>>
>>>>>> operators
>>>>>> we
>>>>>>
>>>>>>> have a bit more freedom to do changes before a major release as most
>>>>>>>
>>>>>> of
>>>>> them are marked @Evolving and users have the ability to continue
>>>>>> using
>>>>> prior version of Malhar with newer engine due to engine backward
>>>>>>> compatibility guarantee.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Thomas
>>>>>>>
>>>>>>>
Re: Java packages: legacy -> org.apache.apex
Posted by Thomas Weise <th...@apache.org>.
next release is 3.8.0
On Fri, Jul 14, 2017 at 7:55 AM, Vlad Rozov <v....@datatorrent.com> wrote:
> next release - 3.9.0 or 4.0.0?
>
> Thank you,
>
> Vlad
>
> On 7/13/17 22:25, Thomas Weise wrote:
>
>> It is time to resurrect this thread and get going with the work.
>>
>> For the next release, I will sign up to do the package move in Malhar:
>>
>> https://issues.apache.org/jira/browse/APEXMALHAR-2517
>>
>> In general this will be straightforward; most classes in Malhar are marked
>> evolving and it is trivial for users to change import statements. However,
>> I would suggest to discuss if there are selected operators that are worth
>> keeping a backward compatibility stub in the old location.
>>
>> Here is my plan:
>>
>> 1. Relocate all classes in *lib* and *contrib* within one PR - this is all
>>
>> IDE automated work
>> 2. Add backward compatibility classes, if, any in separate PR
>> 3. Create PR for Megh library to reflect moved classes
>>
>> Thanks,
>> Thomas
>>
>>
>>
>> On Wed, Mar 1, 2017 at 2:24 PM, Pramod Immaneni <pr...@datatorrent.com>
>> wrote:
>>
>> Inline
>>>
>>> On Mon, Feb 27, 2017 at 11:03 PM, Thomas Weise <th...@apache.org> wrote:
>>>
>>> -->
>>>>
>>>> On Mon, Feb 27, 2017 at 1:27 PM, Pramod Immaneni <
>>>> pramod@datatorrent.com
>>>>
>>>> wrote:
>>>>
>>>> For malhar, for existing operators, I prefer we do this as part of the
>>>>> planned refactoring for breaking the monolith modules into baby
>>>>>
>>>> packages
>>>
>>>> and would also prefer deprecating the existing operators in place.
>>>>>
>>>>
>>>> Refactor into smaller modules was discussed for malhar-contrib and given
>>>> the overall state of that module I think it is OK to defer package
>>>>
>>> renaming
>>>
>>>> there. I do however prefer to see the package rename addressed for other
>>>> modules, especially for the main library module.
>>>>
>>>> Should we consider breaking the library into smaller modules as well,
>>> the
>>> file/block operators for example probably can be in their own module from
>>> just an organizational perspective.
>>>
>>>
>>>
>>>> This
>>>>> will help us achieve two things. First, the user will see all the new
>>>>> changes at once as opposed to dealing with it twice (with package
>>>>>
>>>> rename
>>>
>>>> and dependency changes) and second it will allow for a smoother
>>>>>
>>>> transition
>>>>
>>>>> as the existing code will still work in a deprecated state. It will
>>>>>
>>>> also
>>>
>>>> give a more consistent structure to malhar. For new operators, we can
>>>>>
>>>> go
>>>
>>>> with the new package path but we need to ensure they will get moved
>>>>>
>>>> into
>>>
>>>> the baby packages as well.
>>>>>
>>>>> I think existing operators should be renamed so that git history
>>>>
>>> remains. A
>>>
>>>> possible solution for backward compatibility could be to subsequently
>>>> add
>>>> empty subclasses in the previous location (for existing concrete
>>>>
>>> operators
>>>
>>>> that we know are actually in use) to simplify migration for users.
>>>>
>>>>
>>>> Yes we can do that.
>>>
>>>
>>> For demos, we can modify the paths as the apps are typically used
>>>>>
>>>> wholesale
>>>>
>>>>> and the interface is typically manual interaction.
>>>>>
>>>>> For core, if we are adding new api subsystems, like the launcher api we
>>>>> added recently for example, we can go with new package path but if we
>>>>>
>>>> are
>>>
>>>> making incremental additions to existing functionality, I feel it is
>>>>>
>>>> better
>>>>
>>>>> to keep it in the same package. I also prefer we keep the package of
>>>>>
>>>> the
>>>
>>>> implementation classes consistent with api, for understandability and
>>>>> readability of the code. So, for example, we don't change package path
>>>>>
>>>> of
>>>
>>>> LogicalPlan as it is an implementation of DAG. It is subjective, but it
>>>>> will be good if we can also do the same with classes closely related to
>>>>>
>>>> the
>>>>
>>>>> implementation classes as well. Maybe we can moving these on a package
>>>>>
>>>> by
>>>
>>>> package basis, like everything in com.datatorrent.stram.engine could be
>>>>> moved. For completely internal components like buffer server, we can
>>>>>
>>>> move
>>>
>>>> them wholesale. We can consider moving all api and classes, when we go
>>>>>
>>>> to
>>>
>>>> next major release but would like to see if we can find a way to
>>>>>
>>>> support
>>>
>>>> existing api for one more major release in deprecated mode.
>>>>>
>>>>>
>>>>> The point of the major release is to enable backward incompatible
>>>> changes
>>>> and I don't think it is realistic to support the existing API for
>>>> another
>>>> major release. IMO it is also not necessary as most existing application
>>>> code refers to operators, attributes and the application interface.
>>>>
>>> Perhaps
>>>
>>>> it is possible to keep those around as interface extensions to help
>>>> migration. Custom operators may need to be migrated to reflect API
>>>>
>>> changes,
>>>
>>>> and I would consider that a reasonable task for operator developers as
>>>>
>>> part
>>>
>>>> of a major upgrade.
>>>>
>>>> It would be good if we can keep them as deprecated interface extensions
>>> for
>>> one release to provide a smoother transition.
>>>
>>>
>>> API and implementation in engine are kept separate intentionally. They
>>>> reside in different packages today, so I don't see a problem renaming
>>>> com.datatorrent.stram.engine as you say, even when the API cannot be
>>>> touched right away.
>>>>
>>>> They are different packages but sharing a common prefix with api will be
>>> helpful to someone new to codebase in terms of readability. Not a big
>>> deal
>>> and can be changed.
>>>
>>>
>>> Thanks
>>>>
>>>>> On Mon, Feb 27, 2017 at 7:39 AM, Thomas Weise <th...@apache.org> wrote:
>>>>>
>>>>> Hi,
>>>>>>
>>>>>> This topic has come up on several PRs and I think it warrants a
>>>>>>
>>>>> broader
>>>
>>>> discussion.
>>>>>>
>>>>>> At the time of incubation, the decision was to defer change of Java
>>>>>> packages from com.datatorrent to org.apache.apex till next major
>>>>>>
>>>>> release
>>>>
>>>>> to
>>>>>
>>>>>> ensure backward compatibility for users.
>>>>>>
>>>>>> Unfortunately that has lead to some confusion, as contributors
>>>>>>
>>>>> continue
>>>
>>>> to
>>>>>
>>>>>> add new code under legacy packages.
>>>>>>
>>>>>> It is also a wider issue that examples for using Apex continue to
>>>>>>
>>>>> refer
>>>
>>>> to
>>>>>
>>>>>> com.datatorrent packages, nearly one year after graduation. More and
>>>>>>
>>>>> more
>>>>
>>>>> user code is being built on top of something that needs to change,
>>>>>>
>>>>> the
>>>
>>>> can
>>>>>
>>>>>> is being kicked down the road and users will face more changes later.
>>>>>>
>>>>>> I would like to propose the following:
>>>>>>
>>>>>> 1. All new code has to be submitted under org.apache.apex packages
>>>>>>
>>>>>> 2. Not all code is under backward compatibility restriction and in
>>>>>>
>>>>> those
>>>>
>>>>> cases we can rename the packages right away. Examples: buffer server,
>>>>>> engine, demos/examples, benchmarks
>>>>>>
>>>>>> 3. Discuss when the core API and operators can be changed. For
>>>>>>
>>>>> operators
>>>>
>>>>> we
>>>>>
>>>>>> have a bit more freedom to do changes before a major release as most
>>>>>>
>>>>> of
>>>
>>>> them are marked @Evolving and users have the ability to continue
>>>>>>
>>>>> using
>>>
>>>> prior version of Malhar with newer engine due to engine backward
>>>>>> compatibility guarantee.
>>>>>>
>>>>>> Thanks,
>>>>>> Thomas
>>>>>>
>>>>>>
>
Re: Java packages: legacy -> org.apache.apex
Posted by Pramod Immaneni <pr...@datatorrent.com>.
Would prefer to have bw compatibility stubs for the regularly used
operators such as kafka (0.8), file, db etc and mark them deprecated. Would
like to have at least one 3.x release before going to 4.0 where we can
remove all the deprecated code.
Thanks
On Fri, Jul 14, 2017 at 7:55 AM, Vlad Rozov <v....@datatorrent.com> wrote:
> next release - 3.9.0 or 4.0.0?
>
> Thank you,
>
> Vlad
>
>
> On 7/13/17 22:25, Thomas Weise wrote:
>
>> It is time to resurrect this thread and get going with the work.
>>
>> For the next release, I will sign up to do the package move in Malhar:
>>
>> https://issues.apache.org/jira/browse/APEXMALHAR-2517
>>
>> In general this will be straightforward; most classes in Malhar are marked
>> evolving and it is trivial for users to change import statements. However,
>> I would suggest to discuss if there are selected operators that are worth
>> keeping a backward compatibility stub in the old location.
>>
>> Here is my plan:
>>
>> 1. Relocate all classes in *lib* and *contrib* within one PR - this is all
>> IDE automated work
>> 2. Add backward compatibility classes, if, any in separate PR
>> 3. Create PR for Megh library to reflect moved classes
>>
>> Thanks,
>> Thomas
>>
>>
>>
>> On Wed, Mar 1, 2017 at 2:24 PM, Pramod Immaneni <pr...@datatorrent.com>
>> wrote:
>>
>> Inline
>>>
>>> On Mon, Feb 27, 2017 at 11:03 PM, Thomas Weise <th...@apache.org> wrote:
>>>
>>> -->
>>>>
>>>> On Mon, Feb 27, 2017 at 1:27 PM, Pramod Immaneni <
>>>> pramod@datatorrent.com
>>>>
>>>> wrote:
>>>>
>>>> For malhar, for existing operators, I prefer we do this as part of the
>>>>> planned refactoring for breaking the monolith modules into baby
>>>>>
>>>> packages
>>>
>>>> and would also prefer deprecating the existing operators in place.
>>>>>
>>>>
>>>> Refactor into smaller modules was discussed for malhar-contrib and given
>>>> the overall state of that module I think it is OK to defer package
>>>>
>>> renaming
>>>
>>>> there. I do however prefer to see the package rename addressed for other
>>>> modules, especially for the main library module.
>>>>
>>>> Should we consider breaking the library into smaller modules as well,
>>> the
>>> file/block operators for example probably can be in their own module from
>>> just an organizational perspective.
>>>
>>>
>>>
>>>> This
>>>>> will help us achieve two things. First, the user will see all the new
>>>>> changes at once as opposed to dealing with it twice (with package
>>>>>
>>>> rename
>>>
>>>> and dependency changes) and second it will allow for a smoother
>>>>>
>>>> transition
>>>>
>>>>> as the existing code will still work in a deprecated state. It will
>>>>>
>>>> also
>>>
>>>> give a more consistent structure to malhar. For new operators, we can
>>>>>
>>>> go
>>>
>>>> with the new package path but we need to ensure they will get moved
>>>>>
>>>> into
>>>
>>>> the baby packages as well.
>>>>>
>>>>> I think existing operators should be renamed so that git history
>>>>
>>> remains. A
>>>
>>>> possible solution for backward compatibility could be to subsequently
>>>> add
>>>> empty subclasses in the previous location (for existing concrete
>>>>
>>> operators
>>>
>>>> that we know are actually in use) to simplify migration for users.
>>>>
>>>>
>>>> Yes we can do that.
>>>
>>>
>>> For demos, we can modify the paths as the apps are typically used
>>>>>
>>>> wholesale
>>>>
>>>>> and the interface is typically manual interaction.
>>>>>
>>>>> For core, if we are adding new api subsystems, like the launcher api we
>>>>> added recently for example, we can go with new package path but if we
>>>>>
>>>> are
>>>
>>>> making incremental additions to existing functionality, I feel it is
>>>>>
>>>> better
>>>>
>>>>> to keep it in the same package. I also prefer we keep the package of
>>>>>
>>>> the
>>>
>>>> implementation classes consistent with api, for understandability and
>>>>> readability of the code. So, for example, we don't change package path
>>>>>
>>>> of
>>>
>>>> LogicalPlan as it is an implementation of DAG. It is subjective, but it
>>>>> will be good if we can also do the same with classes closely related to
>>>>>
>>>> the
>>>>
>>>>> implementation classes as well. Maybe we can moving these on a package
>>>>>
>>>> by
>>>
>>>> package basis, like everything in com.datatorrent.stram.engine could be
>>>>> moved. For completely internal components like buffer server, we can
>>>>>
>>>> move
>>>
>>>> them wholesale. We can consider moving all api and classes, when we go
>>>>>
>>>> to
>>>
>>>> next major release but would like to see if we can find a way to
>>>>>
>>>> support
>>>
>>>> existing api for one more major release in deprecated mode.
>>>>>
>>>>>
>>>>> The point of the major release is to enable backward incompatible
>>>> changes
>>>> and I don't think it is realistic to support the existing API for
>>>> another
>>>> major release. IMO it is also not necessary as most existing application
>>>> code refers to operators, attributes and the application interface.
>>>>
>>> Perhaps
>>>
>>>> it is possible to keep those around as interface extensions to help
>>>> migration. Custom operators may need to be migrated to reflect API
>>>>
>>> changes,
>>>
>>>> and I would consider that a reasonable task for operator developers as
>>>>
>>> part
>>>
>>>> of a major upgrade.
>>>>
>>>> It would be good if we can keep them as deprecated interface extensions
>>> for
>>> one release to provide a smoother transition.
>>>
>>>
>>> API and implementation in engine are kept separate intentionally. They
>>>> reside in different packages today, so I don't see a problem renaming
>>>> com.datatorrent.stram.engine as you say, even when the API cannot be
>>>> touched right away.
>>>>
>>>> They are different packages but sharing a common prefix with api will be
>>> helpful to someone new to codebase in terms of readability. Not a big
>>> deal
>>> and can be changed.
>>>
>>>
>>> Thanks
>>>>
>>>>> On Mon, Feb 27, 2017 at 7:39 AM, Thomas Weise <th...@apache.org> wrote:
>>>>>
>>>>> Hi,
>>>>>>
>>>>>> This topic has come up on several PRs and I think it warrants a
>>>>>>
>>>>> broader
>>>
>>>> discussion.
>>>>>>
>>>>>> At the time of incubation, the decision was to defer change of Java
>>>>>> packages from com.datatorrent to org.apache.apex till next major
>>>>>>
>>>>> release
>>>>
>>>>> to
>>>>>
>>>>>> ensure backward compatibility for users.
>>>>>>
>>>>>> Unfortunately that has lead to some confusion, as contributors
>>>>>>
>>>>> continue
>>>
>>>> to
>>>>>
>>>>>> add new code under legacy packages.
>>>>>>
>>>>>> It is also a wider issue that examples for using Apex continue to
>>>>>>
>>>>> refer
>>>
>>>> to
>>>>>
>>>>>> com.datatorrent packages, nearly one year after graduation. More and
>>>>>>
>>>>> more
>>>>
>>>>> user code is being built on top of something that needs to change,
>>>>>>
>>>>> the
>>>
>>>> can
>>>>>
>>>>>> is being kicked down the road and users will face more changes later.
>>>>>>
>>>>>> I would like to propose the following:
>>>>>>
>>>>>> 1. All new code has to be submitted under org.apache.apex packages
>>>>>>
>>>>>> 2. Not all code is under backward compatibility restriction and in
>>>>>>
>>>>> those
>>>>
>>>>> cases we can rename the packages right away. Examples: buffer server,
>>>>>> engine, demos/examples, benchmarks
>>>>>>
>>>>>> 3. Discuss when the core API and operators can be changed. For
>>>>>>
>>>>> operators
>>>>
>>>>> we
>>>>>
>>>>>> have a bit more freedom to do changes before a major release as most
>>>>>>
>>>>> of
>>>
>>>> them are marked @Evolving and users have the ability to continue
>>>>>>
>>>>> using
>>>
>>>> prior version of Malhar with newer engine due to engine backward
>>>>>> compatibility guarantee.
>>>>>>
>>>>>> Thanks,
>>>>>> Thomas
>>>>>>
>>>>>>
>
Re: Java packages: legacy -> org.apache.apex
Posted by Vlad Rozov <v....@datatorrent.com>.
next release - 3.9.0 or 4.0.0?
Thank you,
Vlad
On 7/13/17 22:25, Thomas Weise wrote:
> It is time to resurrect this thread and get going with the work.
>
> For the next release, I will sign up to do the package move in Malhar:
>
> https://issues.apache.org/jira/browse/APEXMALHAR-2517
>
> In general this will be straightforward; most classes in Malhar are marked
> evolving and it is trivial for users to change import statements. However,
> I would suggest to discuss if there are selected operators that are worth
> keeping a backward compatibility stub in the old location.
>
> Here is my plan:
>
> 1. Relocate all classes in *lib* and *contrib* within one PR - this is all
> IDE automated work
> 2. Add backward compatibility classes, if, any in separate PR
> 3. Create PR for Megh library to reflect moved classes
>
> Thanks,
> Thomas
>
>
>
> On Wed, Mar 1, 2017 at 2:24 PM, Pramod Immaneni <pr...@datatorrent.com>
> wrote:
>
>> Inline
>>
>> On Mon, Feb 27, 2017 at 11:03 PM, Thomas Weise <th...@apache.org> wrote:
>>
>>> -->
>>>
>>> On Mon, Feb 27, 2017 at 1:27 PM, Pramod Immaneni <pramod@datatorrent.com
>>>
>>> wrote:
>>>
>>>> For malhar, for existing operators, I prefer we do this as part of the
>>>> planned refactoring for breaking the monolith modules into baby
>> packages
>>>> and would also prefer deprecating the existing operators in place.
>>>
>>> Refactor into smaller modules was discussed for malhar-contrib and given
>>> the overall state of that module I think it is OK to defer package
>> renaming
>>> there. I do however prefer to see the package rename addressed for other
>>> modules, especially for the main library module.
>>>
>> Should we consider breaking the library into smaller modules as well, the
>> file/block operators for example probably can be in their own module from
>> just an organizational perspective.
>>
>>
>>>
>>>> This
>>>> will help us achieve two things. First, the user will see all the new
>>>> changes at once as opposed to dealing with it twice (with package
>> rename
>>>> and dependency changes) and second it will allow for a smoother
>>> transition
>>>> as the existing code will still work in a deprecated state. It will
>> also
>>>> give a more consistent structure to malhar. For new operators, we can
>> go
>>>> with the new package path but we need to ensure they will get moved
>> into
>>>> the baby packages as well.
>>>>
>>> I think existing operators should be renamed so that git history
>> remains. A
>>> possible solution for backward compatibility could be to subsequently add
>>> empty subclasses in the previous location (for existing concrete
>> operators
>>> that we know are actually in use) to simplify migration for users.
>>>
>>>
>> Yes we can do that.
>>
>>
>>>> For demos, we can modify the paths as the apps are typically used
>>> wholesale
>>>> and the interface is typically manual interaction.
>>>>
>>>> For core, if we are adding new api subsystems, like the launcher api we
>>>> added recently for example, we can go with new package path but if we
>> are
>>>> making incremental additions to existing functionality, I feel it is
>>> better
>>>> to keep it in the same package. I also prefer we keep the package of
>> the
>>>> implementation classes consistent with api, for understandability and
>>>> readability of the code. So, for example, we don't change package path
>> of
>>>> LogicalPlan as it is an implementation of DAG. It is subjective, but it
>>>> will be good if we can also do the same with classes closely related to
>>> the
>>>> implementation classes as well. Maybe we can moving these on a package
>> by
>>>> package basis, like everything in com.datatorrent.stram.engine could be
>>>> moved. For completely internal components like buffer server, we can
>> move
>>>> them wholesale. We can consider moving all api and classes, when we go
>> to
>>>> next major release but would like to see if we can find a way to
>> support
>>>> existing api for one more major release in deprecated mode.
>>>>
>>>>
>>> The point of the major release is to enable backward incompatible changes
>>> and I don't think it is realistic to support the existing API for another
>>> major release. IMO it is also not necessary as most existing application
>>> code refers to operators, attributes and the application interface.
>> Perhaps
>>> it is possible to keep those around as interface extensions to help
>>> migration. Custom operators may need to be migrated to reflect API
>> changes,
>>> and I would consider that a reasonable task for operator developers as
>> part
>>> of a major upgrade.
>>>
>> It would be good if we can keep them as deprecated interface extensions for
>> one release to provide a smoother transition.
>>
>>
>>> API and implementation in engine are kept separate intentionally. They
>>> reside in different packages today, so I don't see a problem renaming
>>> com.datatorrent.stram.engine as you say, even when the API cannot be
>>> touched right away.
>>>
>> They are different packages but sharing a common prefix with api will be
>> helpful to someone new to codebase in terms of readability. Not a big deal
>> and can be changed.
>>
>>
>>> Thanks
>>>> On Mon, Feb 27, 2017 at 7:39 AM, Thomas Weise <th...@apache.org> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> This topic has come up on several PRs and I think it warrants a
>> broader
>>>>> discussion.
>>>>>
>>>>> At the time of incubation, the decision was to defer change of Java
>>>>> packages from com.datatorrent to org.apache.apex till next major
>>> release
>>>> to
>>>>> ensure backward compatibility for users.
>>>>>
>>>>> Unfortunately that has lead to some confusion, as contributors
>> continue
>>>> to
>>>>> add new code under legacy packages.
>>>>>
>>>>> It is also a wider issue that examples for using Apex continue to
>> refer
>>>> to
>>>>> com.datatorrent packages, nearly one year after graduation. More and
>>> more
>>>>> user code is being built on top of something that needs to change,
>> the
>>>> can
>>>>> is being kicked down the road and users will face more changes later.
>>>>>
>>>>> I would like to propose the following:
>>>>>
>>>>> 1. All new code has to be submitted under org.apache.apex packages
>>>>>
>>>>> 2. Not all code is under backward compatibility restriction and in
>>> those
>>>>> cases we can rename the packages right away. Examples: buffer server,
>>>>> engine, demos/examples, benchmarks
>>>>>
>>>>> 3. Discuss when the core API and operators can be changed. For
>>> operators
>>>> we
>>>>> have a bit more freedom to do changes before a major release as most
>> of
>>>>> them are marked @Evolving and users have the ability to continue
>> using
>>>>> prior version of Malhar with newer engine due to engine backward
>>>>> compatibility guarantee.
>>>>>
>>>>> Thanks,
>>>>> Thomas
>>>>>
Re: Java packages: legacy -> org.apache.apex
Posted by Thomas Weise <th...@apache.org>.
It is time to resurrect this thread and get going with the work.
For the next release, I will sign up to do the package move in Malhar:
https://issues.apache.org/jira/browse/APEXMALHAR-2517
In general this will be straightforward; most classes in Malhar are marked
evolving and it is trivial for users to change import statements. However,
I would suggest to discuss if there are selected operators that are worth
keeping a backward compatibility stub in the old location.
Here is my plan:
1. Relocate all classes in *lib* and *contrib* within one PR - this is all
IDE automated work
2. Add backward compatibility classes, if, any in separate PR
3. Create PR for Megh library to reflect moved classes
Thanks,
Thomas
On Wed, Mar 1, 2017 at 2:24 PM, Pramod Immaneni <pr...@datatorrent.com>
wrote:
> Inline
>
> On Mon, Feb 27, 2017 at 11:03 PM, Thomas Weise <th...@apache.org> wrote:
>
> > -->
> >
> > On Mon, Feb 27, 2017 at 1:27 PM, Pramod Immaneni <pramod@datatorrent.com
> >
> > wrote:
> >
> > > For malhar, for existing operators, I prefer we do this as part of the
> > > planned refactoring for breaking the monolith modules into baby
> packages
> > > and would also prefer deprecating the existing operators in place.
> >
> >
> > Refactor into smaller modules was discussed for malhar-contrib and given
> > the overall state of that module I think it is OK to defer package
> renaming
> > there. I do however prefer to see the package rename addressed for other
> > modules, especially for the main library module.
> >
>
> Should we consider breaking the library into smaller modules as well, the
> file/block operators for example probably can be in their own module from
> just an organizational perspective.
>
>
> >
> >
> > > This
> > > will help us achieve two things. First, the user will see all the new
> > > changes at once as opposed to dealing with it twice (with package
> rename
> > > and dependency changes) and second it will allow for a smoother
> > transition
> > > as the existing code will still work in a deprecated state. It will
> also
> > > give a more consistent structure to malhar. For new operators, we can
> go
> > > with the new package path but we need to ensure they will get moved
> into
> > > the baby packages as well.
> > >
> >
> > I think existing operators should be renamed so that git history
> remains. A
> > possible solution for backward compatibility could be to subsequently add
> > empty subclasses in the previous location (for existing concrete
> operators
> > that we know are actually in use) to simplify migration for users.
> >
> >
> Yes we can do that.
>
>
> >
> > > For demos, we can modify the paths as the apps are typically used
> > wholesale
> > > and the interface is typically manual interaction.
> > >
> > > For core, if we are adding new api subsystems, like the launcher api we
> > > added recently for example, we can go with new package path but if we
> are
> > > making incremental additions to existing functionality, I feel it is
> > better
> > > to keep it in the same package. I also prefer we keep the package of
> the
> > > implementation classes consistent with api, for understandability and
> > > readability of the code. So, for example, we don't change package path
> of
> > > LogicalPlan as it is an implementation of DAG. It is subjective, but it
> > > will be good if we can also do the same with classes closely related to
> > the
> > > implementation classes as well. Maybe we can moving these on a package
> by
> > > package basis, like everything in com.datatorrent.stram.engine could be
> > > moved. For completely internal components like buffer server, we can
> move
> > > them wholesale. We can consider moving all api and classes, when we go
> to
> > > next major release but would like to see if we can find a way to
> support
> > > existing api for one more major release in deprecated mode.
> > >
> > >
> > The point of the major release is to enable backward incompatible changes
> > and I don't think it is realistic to support the existing API for another
> > major release. IMO it is also not necessary as most existing application
> > code refers to operators, attributes and the application interface.
> Perhaps
> > it is possible to keep those around as interface extensions to help
> > migration. Custom operators may need to be migrated to reflect API
> changes,
> > and I would consider that a reasonable task for operator developers as
> part
> > of a major upgrade.
> >
>
> It would be good if we can keep them as deprecated interface extensions for
> one release to provide a smoother transition.
>
>
> >
> > API and implementation in engine are kept separate intentionally. They
> > reside in different packages today, so I don't see a problem renaming
> > com.datatorrent.stram.engine as you say, even when the API cannot be
> > touched right away.
> >
>
> They are different packages but sharing a common prefix with api will be
> helpful to someone new to codebase in terms of readability. Not a big deal
> and can be changed.
>
>
> >
> > Thanks
> > >
> > > On Mon, Feb 27, 2017 at 7:39 AM, Thomas Weise <th...@apache.org> wrote:
> > >
> > > > Hi,
> > > >
> > > > This topic has come up on several PRs and I think it warrants a
> broader
> > > > discussion.
> > > >
> > > > At the time of incubation, the decision was to defer change of Java
> > > > packages from com.datatorrent to org.apache.apex till next major
> > release
> > > to
> > > > ensure backward compatibility for users.
> > > >
> > > > Unfortunately that has lead to some confusion, as contributors
> continue
> > > to
> > > > add new code under legacy packages.
> > > >
> > > > It is also a wider issue that examples for using Apex continue to
> refer
> > > to
> > > > com.datatorrent packages, nearly one year after graduation. More and
> > more
> > > > user code is being built on top of something that needs to change,
> the
> > > can
> > > > is being kicked down the road and users will face more changes later.
> > > >
> > > > I would like to propose the following:
> > > >
> > > > 1. All new code has to be submitted under org.apache.apex packages
> > > >
> > > > 2. Not all code is under backward compatibility restriction and in
> > those
> > > > cases we can rename the packages right away. Examples: buffer server,
> > > > engine, demos/examples, benchmarks
> > > >
> > > > 3. Discuss when the core API and operators can be changed. For
> > operators
> > > we
> > > > have a bit more freedom to do changes before a major release as most
> of
> > > > them are marked @Evolving and users have the ability to continue
> using
> > > > prior version of Malhar with newer engine due to engine backward
> > > > compatibility guarantee.
> > > >
> > > > Thanks,
> > > > Thomas
> > > >
> > >
> >
>