Re: The Go SDK got accidentally merged - options to deal with the pain

[Henning Rohde <he...@google.com>]

Hi everyone,

 Thank you all for your patience. The last major identified feature (Go
windowing) is now in review: https://github.com/apache/beam/pull/5179. The
remaining work listed under

     https://issues.apache.org/jira/browse/BEAM-2083

is integration tests and documentation (quickstart, etc). I expect that
will take a few weeks after which we should be in a position to do a vote
about making the Go SDK an official Beam SDK. To this end, please do take a
look at the listed tasks and let me know if there us anything missing.

Lastly, I have a practical question: how should we order the PRs to the
beam site documentation wrt the vote? Should we get PRs accepted, but not
committed before a vote? Or just commit them as they are ready to avoid
potential merge conflicts?

Thanks!

Henning




On Sat, Mar 10, 2018 at 10:45 AM Henning Rohde <he...@google.com> wrote:

> Thank you all! I've added the remaining work -- as I understand it -- as
> dependencies to the overall Go SDK issue (tracking the "official" merge to
> master):
>
>     https://issues.apache.org/jira/browse/BEAM-2083
>
> Please feel free to add to this list or expand the items, if there is
> anything I overlooked. If this presence of the Go SDK in master cause
> issues for other modules, please simply file a bug against me and I'll take
> care of it.
>
> Robert - I understand your last reply as addressing Davor's points. Please
> let me know if there is anything I need to do in that regard.
>
> Henning
>
>
>
> On Fri, Mar 9, 2018 at 8:39 AM, Ismaël Mejía <ie...@gmail.com> wrote:
>
>> +1 to let it evolve in master (+Davor points), having ongoing work on
>> master makes sense given the state of advance + the hope that this
>> won't add any issue for the other modules.
>>
>> On Thu, Mar 8, 2018 at 7:30 PM, Robert Bradshaw <ro...@google.com>
>> wrote:
>> > +1 to both of these points. SGA should have probably already been
>> filed, and
>> > excising this from releases should be easy, but I added a line item to
>> the
>> > validation checklist template to make sure we don't forget.
>> >
>> > On Thu, Mar 8, 2018 at 7:13 AM Davor Bonaci <da...@apache.org> wrote:
>> >>
>> >> I support leaving things as they stand now -- thanks for finding a good
>> >> way out of an uncomfortable situation.
>> >>
>> >> That said, two things need to happen:
>> >> (1) SGA needs to be filed asap, per Board feedback in the last report,
>> and
>> >> (2) releases cannot contain any code from the Go SDK before formally
>> voted
>> >> on the new component and accepted. This includes source releases that
>> are
>> >> created through "assembly", so manual exclusion in the configuration is
>> >> likely needed.
>> >>
>> >> On Wed, Mar 7, 2018 at 1:54 PM, Kenneth Knowles <kl...@google.com>
>> wrote:
>> >>>
>> >>> Re-reading the old thread, I see these desirata:
>> >>>
>> >>>  - "enough IO to write end-to-end examples such as WordCount and
>> >>> demonstrate what IOs would look like"
>> >>>  - "accounting and tracking the fact that each element has an
>> associated
>> >>> window and timestamp"
>> >>>  - "test suites and test utilities"
>> >>>
>> >>> Browsing the code, it looks like these each exist to some level of
>> >>> completion.
>> >>>
>> >>> Kenn
>> >>>
>> >>>
>> >>> On Wed, Mar 7, 2018 at 1:38 PM Robert Bradshaw <ro...@google.com>
>> >>> wrote:
>> >>>>
>> >>>> I was actually thinking along the same lines: what was yet lacking to
>> >>>> "officially" merge the Go branch in? The thread we started on this
>> seems to
>> >>>> have fizzled out over the holidays, but windowing support is the only
>> >>>> must-have missing technical feature in my book (assuming
>> documentation and
>> >>>> testing are, or are brought up to snuff).
>> >>>>
>> >>>>
>> >>>> On Wed, Mar 7, 2018 at 1:35 PM Henning Rohde <he...@google.com>
>> wrote:
>> >>>>>
>> >>>>> One thought: the Go SDK is actually not that far away from
>> satisfying
>> >>>>> the guidelines for merging to master anyway (as discussed here
>> [1]). If we
>> >>>>> decide to simply leave the code in master -- which seems to be what
>> this
>> >>>>> thread is leaning towards -- I'll gladly sign up to do the
>> remaining aspects
>> >>>>> (I believe it's only windowing, validation tests and documentation)
>> >>>>> reasonably quickly to get to an official vote for accepting it and
>> in turn
>> >>>>> get master into a sound state. It would seem like the path of least
>> hassle.
>> >>>>> Of course, I'm happy to go with whatever the community is
>> comfortable with
>> >>>>> -- just trying to make lemonade out of the merge lemon.
>> >>>>>
>> >>>>> Henning
>> >>>>>
>> >>>>> [1]
>> >>>>>
>> https://lists.apache.org/thread.html/fd4201980d7a6e67248b1f183ee06b0ff1305bd46f1291495679fc0a@%3Cdev.beam.apache.org%3E
>> >>>>>
>> >>>>> On Tue, Mar 6, 2018 at 3:40 PM, Kenneth Knowles <kl...@google.com>
>> wrote:
>> >>>>>>
>> >>>>>> I think a very easy fix to unblock everyone is
>> >>>>>> https://github.com/apache/beam/pull/4809. It just updates one
>> line of a pom.
>> >>>>>>
>> >>>>>>
>> >>>>>> On Tue, Mar 6, 2018 at 3:33 PM Robert Bradshaw <
>> robertwb@google.com>
>> >>>>>> wrote:
>> >>>>>>>
>> >>>>>>> I'm not sure what value there is in preserving this accidental
>> merge
>> >>>>>>> in history, but all options proposed seem fine to me. We should
>> resolve this
>> >>>>>>> (or at least unblock other dev work) quickly though.
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> On Tue, Mar 6, 2018 at 3:16 PM Kenneth Knowles <kl...@google.com>
>> >>>>>>> wrote:
>> >>>>>>>>
>> >>>>>>>> My own vote is for leaving the history immutable, which is the
>> case
>> >>>>>>>> for the full rollback or leaving it there disabled.
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> On Tue, Mar 6, 2018 at 3:01 PM Thomas Weise <th...@apache.org>
>> wrote:
>> >>>>>>>>>
>> >>>>>>>>> +1 for (1), assuming it is straightforward to exclude from the
>> >>>>>>>>> build and eventually will end up in master anyways.
>> >>>>>>>>>
>> >>>>>>>>> On Tue, Mar 6, 2018 at 2:59 PM, Robert Bradshaw
>> >>>>>>>>> <ro...@google.com> wrote:
>> >>>>>>>>>>
>> >>>>>>>>>> I would opt for (2), but I'm not sure who has permissions to do
>> >>>>>>>>>> that. It should be easy to re-merge the couple of things that
>> have gone in
>> >>>>>>>>>> since then.
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> On Tue, Mar 6, 2018 at 2:43 PM Kenneth Knowles <klk@google.com
>> >
>> >>>>>>>>>> wrote:
>> >>>>>>>>>>>
>> >>>>>>>>>>> Hi all,
>> >>>>>>>>>>>
>> >>>>>>>>>>> You may have noticed that our tests are red. A pull request
>> that
>> >>>>>>>>>>> was meant for the Go SDK branch accidentally got merged onto
>> the master
>> >>>>>>>>>>> branch. Things have been merged to master since then.
>> >>>>>>>>>>>
>> >>>>>>>>>>> I've opened a revert at
>> https://github.com/apache/beam/pull/4808
>> >>>>>>>>>>>
>> >>>>>>>>>>> The next time there is a master to go-sdk merge it will need
>> to
>> >>>>>>>>>>> be re-reverted.
>> >>>>>>>>>>>
>> >>>>>>>>>>> Two other options are (1) leave it there and disable it in
>> >>>>>>>>>>> whatever way and (2) rebase dropping the commit and force
>> push master
>> >>>>>>>>>>> (breaks all checkouts that are past it).
>> >>>>>>>>>>>
>> >>>>>>>>>>> Kenn
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>
>> >>
>> >
>>
>
>

Re: The Go SDK got accidentally merged - options to deal with the pain

[Henning Rohde <he...@google.com>]

Hi everyone,

 Thanks again for your patience. The last remaining Go SDK items are now
resolved and the beam website has been updated! I'll start a separate
thread for the formal vote shortly.

Thanks,
 Henning


On Thu, Apr 19, 2018 at 5:42 PM Henning Rohde <he...@google.com> wrote:

> Hi everyone,
>
>  Thank you all for your patience. The last major identified feature (Go
> windowing) is now in review: https://github.com/apache/beam/pull/5179.
> The remaining work listed under
>
>      https://issues.apache.org/jira/browse/BEAM-2083
>
> is integration tests and documentation (quickstart, etc). I expect that
> will take a few weeks after which we should be in a position to do a vote
> about making the Go SDK an official Beam SDK. To this end, please do take a
> look at the listed tasks and let me know if there us anything missing.
>
> Lastly, I have a practical question: how should we order the PRs to the
> beam site documentation wrt the vote? Should we get PRs accepted, but not
> committed before a vote? Or just commit them as they are ready to avoid
> potential merge conflicts?
>
> Thanks!
>
> Henning
>
>
>
>
> On Sat, Mar 10, 2018 at 10:45 AM Henning Rohde <he...@google.com> wrote:
>
>> Thank you all! I've added the remaining work -- as I understand it -- as
>> dependencies to the overall Go SDK issue (tracking the "official" merge to
>> master):
>>
>>     https://issues.apache.org/jira/browse/BEAM-2083
>>
>> Please feel free to add to this list or expand the items, if there is
>> anything I overlooked. If this presence of the Go SDK in master cause
>> issues for other modules, please simply file a bug against me and I'll take
>> care of it.
>>
>> Robert - I understand your last reply as addressing Davor's points.
>> Please let me know if there is anything I need to do in that regard.
>>
>> Henning
>>
>>
>>
>> On Fri, Mar 9, 2018 at 8:39 AM, Ismaël Mejía <ie...@gmail.com> wrote:
>>
>>> +1 to let it evolve in master (+Davor points), having ongoing work on
>>> master makes sense given the state of advance + the hope that this
>>> won't add any issue for the other modules.
>>>
>>> On Thu, Mar 8, 2018 at 7:30 PM, Robert Bradshaw <ro...@google.com>
>>> wrote:
>>> > +1 to both of these points. SGA should have probably already been
>>> filed, and
>>> > excising this from releases should be easy, but I added a line item to
>>> the
>>> > validation checklist template to make sure we don't forget.
>>> >
>>> > On Thu, Mar 8, 2018 at 7:13 AM Davor Bonaci <da...@apache.org> wrote:
>>> >>
>>> >> I support leaving things as they stand now -- thanks for finding a
>>> good
>>> >> way out of an uncomfortable situation.
>>> >>
>>> >> That said, two things need to happen:
>>> >> (1) SGA needs to be filed asap, per Board feedback in the last
>>> report, and
>>> >> (2) releases cannot contain any code from the Go SDK before formally
>>> voted
>>> >> on the new component and accepted. This includes source releases that
>>> are
>>> >> created through "assembly", so manual exclusion in the configuration
>>> is
>>> >> likely needed.
>>> >>
>>> >> On Wed, Mar 7, 2018 at 1:54 PM, Kenneth Knowles <kl...@google.com>
>>> wrote:
>>> >>>
>>> >>> Re-reading the old thread, I see these desirata:
>>> >>>
>>> >>>  - "enough IO to write end-to-end examples such as WordCount and
>>> >>> demonstrate what IOs would look like"
>>> >>>  - "accounting and tracking the fact that each element has an
>>> associated
>>> >>> window and timestamp"
>>> >>>  - "test suites and test utilities"
>>> >>>
>>> >>> Browsing the code, it looks like these each exist to some level of
>>> >>> completion.
>>> >>>
>>> >>> Kenn
>>> >>>
>>> >>>
>>> >>> On Wed, Mar 7, 2018 at 1:38 PM Robert Bradshaw <ro...@google.com>
>>> >>> wrote:
>>> >>>>
>>> >>>> I was actually thinking along the same lines: what was yet lacking
>>> to
>>> >>>> "officially" merge the Go branch in? The thread we started on this
>>> seems to
>>> >>>> have fizzled out over the holidays, but windowing support is the
>>> only
>>> >>>> must-have missing technical feature in my book (assuming
>>> documentation and
>>> >>>> testing are, or are brought up to snuff).
>>> >>>>
>>> >>>>
>>> >>>> On Wed, Mar 7, 2018 at 1:35 PM Henning Rohde <he...@google.com>
>>> wrote:
>>> >>>>>
>>> >>>>> One thought: the Go SDK is actually not that far away from
>>> satisfying
>>> >>>>> the guidelines for merging to master anyway (as discussed here
>>> [1]). If we
>>> >>>>> decide to simply leave the code in master -- which seems to be
>>> what this
>>> >>>>> thread is leaning towards -- I'll gladly sign up to do the
>>> remaining aspects
>>> >>>>> (I believe it's only windowing, validation tests and documentation)
>>> >>>>> reasonably quickly to get to an official vote for accepting it and
>>> in turn
>>> >>>>> get master into a sound state. It would seem like the path of
>>> least hassle.
>>> >>>>> Of course, I'm happy to go with whatever the community is
>>> comfortable with
>>> >>>>> -- just trying to make lemonade out of the merge lemon.
>>> >>>>>
>>> >>>>> Henning
>>> >>>>>
>>> >>>>> [1]
>>> >>>>>
>>> https://lists.apache.org/thread.html/fd4201980d7a6e67248b1f183ee06b0ff1305bd46f1291495679fc0a@%3Cdev.beam.apache.org%3E
>>> >>>>>
>>> >>>>> On Tue, Mar 6, 2018 at 3:40 PM, Kenneth Knowles <kl...@google.com>
>>> wrote:
>>> >>>>>>
>>> >>>>>> I think a very easy fix to unblock everyone is
>>> >>>>>> https://github.com/apache/beam/pull/4809. It just updates one
>>> line of a pom.
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> On Tue, Mar 6, 2018 at 3:33 PM Robert Bradshaw <
>>> robertwb@google.com>
>>> >>>>>> wrote:
>>> >>>>>>>
>>> >>>>>>> I'm not sure what value there is in preserving this accidental
>>> merge
>>> >>>>>>> in history, but all options proposed seem fine to me. We should
>>> resolve this
>>> >>>>>>> (or at least unblock other dev work) quickly though.
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>> On Tue, Mar 6, 2018 at 3:16 PM Kenneth Knowles <kl...@google.com>
>>> >>>>>>> wrote:
>>> >>>>>>>>
>>> >>>>>>>> My own vote is for leaving the history immutable, which is the
>>> case
>>> >>>>>>>> for the full rollback or leaving it there disabled.
>>> >>>>>>>>
>>> >>>>>>>>
>>> >>>>>>>> On Tue, Mar 6, 2018 at 3:01 PM Thomas Weise <th...@apache.org>
>>> wrote:
>>> >>>>>>>>>
>>> >>>>>>>>> +1 for (1), assuming it is straightforward to exclude from the
>>> >>>>>>>>> build and eventually will end up in master anyways.
>>> >>>>>>>>>
>>> >>>>>>>>> On Tue, Mar 6, 2018 at 2:59 PM, Robert Bradshaw
>>> >>>>>>>>> <ro...@google.com> wrote:
>>> >>>>>>>>>>
>>> >>>>>>>>>> I would opt for (2), but I'm not sure who has permissions to
>>> do
>>> >>>>>>>>>> that. It should be easy to re-merge the couple of things that
>>> have gone in
>>> >>>>>>>>>> since then.
>>> >>>>>>>>>>
>>> >>>>>>>>>>
>>> >>>>>>>>>> On Tue, Mar 6, 2018 at 2:43 PM Kenneth Knowles <
>>> klk@google.com>
>>> >>>>>>>>>> wrote:
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> Hi all,
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> You may have noticed that our tests are red. A pull request
>>> that
>>> >>>>>>>>>>> was meant for the Go SDK branch accidentally got merged onto
>>> the master
>>> >>>>>>>>>>> branch. Things have been merged to master since then.
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> I've opened a revert at
>>> https://github.com/apache/beam/pull/4808
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> The next time there is a master to go-sdk merge it will need
>>> to
>>> >>>>>>>>>>> be re-reverted.
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> Two other options are (1) leave it there and disable it in
>>> >>>>>>>>>>> whatever way and (2) rebase dropping the commit and force
>>> push master
>>> >>>>>>>>>>> (breaks all checkouts that are past it).
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> Kenn
>>> >>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>
>>> >>
>>> >
>>>
>>
>>