Re: status update

[Davor Bonaci <da...@google.com.INVALID>]

Just to close the loop -- I have updated the document based on the latest
round of feedback.

Now that the initial code has been dropped and we have a few pull requests,
we'll start merging those according to the plan.

On Wed, Feb 24, 2016 at 5:39 AM, Sandeep Deshmukh <sa...@datatorrent.com>
wrote:

> Thanks Devor for detailed explanation of the thought process. That
> certainly helps.
>
> I do favor feature branches as they are must of major work. The proposed
> development/feature branch for the time being is perfectly fine. Just a
> caution that keeping the master updated could be taxing. We can certainly
> revisit this later once the project is stable in Incubation.
>
> re: timing of major releases
> Beam being new and rapidly evolving, makes sense to favor growth right now.
>
> Regards
> Sandeep
>
>
> On Fri, Feb 19, 2016 at 1:42 AM, Davor Bonaci <da...@google.com.invalid>
> wrote:
>
> > Thanks Sandeep for your insightful comments! I'll add a little more
> details
> > to follow up on Frances' email to clarify what we were proposing.
> >
> > We were indeed proposing the concept of the development / feature
> branches
> > (for the time being). I think the goal of having the master branch in a
> > "working state" is a clear, non-contentions goal that equally benefits
> all
> > users and all developers. "Working state" doesn't necessarily mean
> > production-ready, but means that "most" things work out of the box. Not
> > having this goal would be hugely taxing for everyone.
> >
> > Now, this goal can be achieved either via:
> >
> >    - direct commits to master with a proper presubmit coverage,
> >    - direct commits to a separate branch(es) with regular integration to
> >    the master, or
> >    - a combination of these approaches on a per-pull request basis.
> >
> > I think everyone universally prefers the first choice. Google committers
> > certainly do.
> >
> > However, we start this project from a point of zero presubmit coverage,
> and
> > expectation of a ton of refactoring and breaking changes. We felt letting
> > all these changes into master right away without proper coverage would
> make
> > everyone's life a little sad. Here's a very simple example -- the vision
> we
> > proposed involves completely changing the runner-facing APIs. Every
> commit
> > of this work will break parts of our project -- such works needs to be
> > separated, otherwise *everyone* gets blocked.
> >
> > [ Additionally, Beam is a somewhat unique project -- it makes sense with
> > services, such as Dataflow Service, Flink or Spark, that are maintained
> as
> > separate projects -- so, we need a decent cross-project integration
> > coverage. ]
> >
> > With this in mind, our proposal is keep the development / feature
> branches
> > *for the time being* -- until we replicate a portion of the presubmit
> > coverage we have internally at Google. This is the "default" place for
> > executing against the vision, cleaning runner API, etc. As Frances said,
> > there are classes of commits, such as a new IO connector or a new
> > domain-specific PTransform, where we can simplify this and go directly to
> > master. In any case, master should never be behind the development branch
> > by more than a week or so.
> >
> > --
> >
> > re: timing of major releases
> > This is often a contentions topic. The other side of the argument is
> > two-fold:
> >
> >    - Releasing rarely creates a bigger cliff. Hence, it becomes a higher
> >    burden and less likely to happen.
> >    - Releasing rarely optimizes experience for existing users, not new
> >    users. If we are growing quickly, that may not be the best metric to
> >    optimize for.
> >
> > In reality, we are likely to release more often as we mature. Later on,
> we
> > are likely to release less often.
> >
> > I think decisions in this space should be delayed until the time we need
> to
> > make a choice and consider the payload we want to release. That said,
> > having some timeline in mind is often beneficial to make right design
> > tradeoffs.
> >
> > --
> >
> > re: @evolving idea
> > We are already doing this and planning to continue -- this works really
> > well. See @Experimental here
> > <
> >
> https://github.com/GoogleCloudPlatform/DataflowJavaSDK/blob/a8347d186700e83edb91a3c673200abec1b1d4e7/sdk/src/main/java/com/google/cloud/dataflow/sdk/annotations/Experimental.java
> > >
> > .
> >
> > On Thu, Feb 18, 2016 at 10:17 AM, Robert Bradshaw <
> > robertwb@google.com.invalid> wrote:
> >
> > > Yes, (b) would be a so-called feature branch. (The proposal here is to
> > > discard the idea of having a separate long-lived "develop" branch.)
> > >
> > > On Thu, Feb 18, 2016 at 10:08 AM, Frances Perry <fjp@google.com.invalid
> >
> > > wrote:
> > >
> > > > Our developers are going to be a varied group -- so "main
> development"
> > > will
> > > > look quite different to different developers. In particular, look at:
> > > > (a) a developer writing a java sdk extension for a new IO connector
> > > > (b) a developer changing the beam model
> > > >
> > > > I think it's fine for work like (a) to occur on master, but I think
> > > things
> > > > like (b) should  happen on a development branch so that we can keep
> the
> > > > master branch in a working state. There are going to be a number of
> > > large,
> > > > backwards incompatible, churn-y changes to Runner APIs in the near
> > > future.
> > > > I'd like us to be able to do those in a way that doesn't affect folks
> > who
> > > > are attempting more surface level contributions.
> > > >
> > > > Frances
> > > >
> > > > On Thu, Feb 18, 2016 at 8:07 AM, Robert Bradshaw <
> > > > robertwb@google.com.invalid> wrote:
> > > >
> > > > > +1 to using master for main development (and most non-ASF projects
> > use
> > > > > master like this too). Not having master (the default when one
> > clones,
> > > > > etc.) be at HEAD is often surprising. Tags are easy enough to use
> > when
> > > > one
> > > > > wants a stable version.
> > > > >
> > > > > - Robert
> > > > >
> > > > >
> > > > > On Wed, Feb 17, 2016 at 11:38 PM, Jean-Baptiste Onofré <
> > > jb@nanthrax.net>
> > > > > wrote:
> > > > >
> > > > > > Thanks Henry, I remember now, and Frances posted the link.
> > > > > >
> > > > > > I agree: we should use the master branch as dev branch as all
> other
> > > ASF
> > > > > > projects do.
> > > > > >
> > > > > > Regards
> > > > > > JB
> > > > > >
> > > > > >
> > > > > > On 02/18/2016 08:04 AM, Henry Saputra wrote:
> > > > > >
> > > > > >> Actually no, it is a bit different.
> > > > > >> The concept of develop branch is following the "successful git
> > > > branching
> > > > > >> model" blog post [1] that introduce using develop branch as
> active
> > > > > branch
> > > > > >> for development and use master as stable branch.
> > > > > >>
> > > > > >> I would recommend using master branch instead as default branch
> to
> > > do
> > > > > >> active development to match other ASF projects.
> > > > > >>
> > > > > >> Some projects using develop from origin company, like Twill [2],
> > had
> > > > > also
> > > > > >> moved to using master as default active branch.
> > > > > >>
> > > > > >> Just my 2 cents.
> > > > > >>
> > > > > >> Thx.
> > > > > >>
> > > > > >> Henry
> > > > > >>
> > > > > >>
> > > > > >> [1] http://nvie.com/posts/a-successful-git-branching-model/
> > > > > >> [2] http://twill.incubator.apache.org/HowToContribute.html
> > > > > >>
> > > > > >> On Wed, Feb 17, 2016 at 10:52 PM, Jean-Baptiste Onofré <
> > > > jb@nanthrax.net
> > > > > >
> > > > > >> wrote:
> > > > > >>
> > > > > >> Hi,
> > > > > >>>
> > > > > >>> Correct me if I'm wrong, but I'm assuming that develop ==
> master
> > > > (from
> > > > > a
> > > > > >>> git perspective).
> > > > > >>>
> > > > > >>> I configured Jenkins this way as it's the "regular" naming ;)
> > > > > >>>
> > > > > >>> I think Frances said "develop" from a dev perspective. All
> > projects
> > > > use
> > > > > >>> master (it's what I'm doing in Falcon, Lens, Karaf, Camel, etc,
> > > etc).
> > > > > >>>
> > > > > >>> Maybe I'm wrong ;)
> > > > > >>>
> > > > > >>> Regards
> > > > > >>> JB
> > > > > >>>
> > > > > >>>
> > > > > >>> On 02/18/2016 06:46 AM, Sandeep Deshmukh wrote:
> > > > > >>>
> > > > > >>> Hi All,
> > > > > >>>>
> > > > > >>>> I have some comments on the repository structure and most of
> > them
> > > > are
> > > > > >>>> wrt
> > > > > >>>> my experience in another Apache incubating project.
> > > > > >>>>
> > > > > >>>>
> > > > > >>>>      1. Most active projects use *master* as default
> development
> > > > > branch
> > > > > >>>> than
> > > > > >>>>      *develop*.  For example, Flink, Spark, Storm, Samza, Pig,
> > > Hive,
> > > > > and
> > > > > >>>>      Hadoop use master branch.
> > > > > >>>>      2. Released artifacts are always hosted on downloads
> > > page.Maser
> > > > > >>>> need
> > > > > >>>> not
> > > > > >>>>      be the one with production ready state.
> > > > > >>>>      3. It is quite intuitive to use *master* otherwise new
> > > > > contributors
> > > > > >>>>      needs to go through documentation to understand process
> of
> > > each
> > > > > >>>> project.
> > > > > >>>>      4. Overall, the process becomes simple if *master* is the
> > > > default
> > > > > >>>> branch.
> > > > > >>>>
> > > > > >>>>
> > > > > >>>> Another suggestion is related to release with major version
> > > change.
> > > > > >>>> Major
> > > > > >>>> release twice a year is a lot of burden on the end user if
> they
> > > want
> > > > > to
> > > > > >>>> upgrade to a newer version. To address this issue, newly added
> > > APIs
> > > > > can
> > > > > >>>> be
> > > > > >>>> marked as @evolving so that users are aware of possible change
> > in
> > > > the
> > > > > >>>> upcoming release but the stable one should be carefully
> changed.
> > > > > >>>>
> > > > > >>>> Regards,
> > > > > >>>> Sandeep
> > > > > >>>>
> > > > > >>>> On Sat, Feb 13, 2016 at 2:34 AM, Frances Perry
> > > > <fjp@google.com.invalid
> > > > > >
> > > > > >>>> wrote:
> > > > > >>>>
> > > > > >>>> Thanks for all the feedback! Please keep it coming as needed.
> > > > > >>>>
> > > > > >>>>>
> > > > > >>>>> We've gone ahead and created components matching this
> > structure:
> > > > > >>>>>
> > > > > >>>>>
> > > > > >>>>>
> > > > > >>>>>
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/BEAM/?selectedTab=com.atlassian.jira.jira-projects-plugin:components-panel
> > > > > >>>>>
> > > > > >>>>> We'll work on transition existing state from Google-internal
> > > tools
> > > > > into
> > > > > >>>>> this over the next few weeks.
> > > > > >>>>>
> > > > > >>>>>
> > > > > >>>>> On Fri, Feb 12, 2016 at 7:47 AM, Kenneth Knowles
> > > > > >>>>> <klk@google.com.invalid
> > > > > >>>>>
> > > > > >>>>>>
> > > > > >>>>>> wrote:
> > > > > >>>>>
> > > > > >>>>> On Thu, Feb 11, 2016 at 8:53 AM, Maximilian Michels <
> > > > mxm@apache.org>
> > > > > >>>>>
> > > > > >>>>>> wrote:
> > > > > >>>>>>
> > > > > >>>>>> As for the /develop branch, I would suggest to
> > > > > >>>>>>
> > > > > >>>>>>> make it mandatory to have it in a usable state at all
> times.
> > > > > >>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>> +1
> > > > > >>>>>>
> > > > > >>>>>> If breakage is accidentally committed (as will happen) then
> a
> > > CTR
> > > > > >>>>>>
> > > > > >>>>>> rollback
> > > > > >>>>>
> > > > > >>>>> is a encouraged.
> > > > > >>>>>>
> > > > > >>>>>> Kenn
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>>
> > > > > >>>> --
> > > > > >>> Jean-Baptiste Onofré
> > > > > >>> jbonofre@apache.org
> > > > > >>> http://blog.nanthrax.net
> > > > > >>> Talend - http://www.talend.com
> > > > > >>>
> > > > > >>>
> > > > > >>
> > > > > > --
> > > > > > Jean-Baptiste Onofré
> > > > > > jbonofre@apache.org
> > > > > > http://blog.nanthrax.net
> > > > > > Talend - http://www.talend.com
> > > > > >
> > > > >
> > > >
> > >
> >
>