You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Konstantinos Katsiapis <ka...@google.com> on 2018/03/01 01:59:14 UTC
Re: Beam 2.4.0
Hi Jean-Baptiste,
I can speak from the perspective of tf.transform
<https://github.com/tensorflow/transform> (TFT) in particular and TFX
<https://research.google.com/pubs/pub46484.html> libs in general, in case
it is useful.
TFX distributed computation has 2 "large" dependencies, namely TensorFlow
and Apache Beam, each on their own release schedule.
As such, releasing of new TFX functionality often (but not always) depends
on (and is blocked by) releases of *both* TensorFlow *and* Apache Beam.
Synchronizing releases across such large projects and organizations is
likely hard, so from our perspective having *frequent* releases of
Tensorflow or Apache Beam (and better yet both) decreases the time for
which we are blocked on releasing our features.
In light of this, I would vote for more frequent releases in general, and
for a Beam 2.4 release soon in particular (as TFT 0.6 depends on it).
Thanks,
Gus
On Tue, Feb 27, 2018 at 10:29 PM, Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> By the way, if third party projects based on Beam (Google Dataflow, Talend
> DataStream, and others) need a release (including some features), it's
> better to
> clearly state this on the mailing list.
>
> At Apache Karaf, I have lot of projects based on it (OpenDaylight, OpenHAB,
> Websphere, ...). They just ask for the release schedule and they align
> with
> these release. As a best effort, I'm always trying to move fast when a
> release
> is asked.
>
> So, if 2.4.0 is required by third party, no problem to "ask for a release".
>
> Regards
> JB
>
> On 02/28/2018 04:17 AM, Reuven Lax wrote:
> > It's been six weeks since you proposed beam 2.3.0. so assuming the same
> time
> > scale for this release, that's 1.5 months between releases. Slightly
> faster than
> > 2 months, but not by much.
> >
> > I do seem to remember that the original goal for beam was monthly
> releases though.
> >
> > Reuven
> >
> > On Tue, Feb 27, 2018, 9:12 PM Jean-Baptiste Onofré <jb@nanthrax.net
> > <ma...@nanthrax.net>> wrote:
> >
> > Hi Reuven,
> >
> > In a previous thread (about Beam project execution), I proposed a
> release every
> > two months (as a best effort), I will find the e-mail.
> >
> > Beam 2.3.0 has been released "officially" on February 16th, so two
> week ago
> > roughly. I would have expected 2.4.0 not before end of March.
> >
> > If we have issue we want to fix fast, then 2.3.1 is good. If it's a
> new release
> > in the pace, it's pretty fast and might "confuse" our users.
> >
> > That's why I'm curious ;)
> >
> > Regards
> > JB
> >
> > On 02/28/2018 03:50 AM, Reuven Lax wrote:
> > > Wasn't the original statement monthly releases? We've never
> realistically
> > > managed that, but Robert's proposed cut will be on a 6-week pace.
> > >
> > > On Tue, Feb 27, 2018, 8:48 PM Jean-Baptiste Onofré <
> jb@nanthrax.net
> > <ma...@nanthrax.net>
> > > <mailto:jb@nanthrax.net <ma...@nanthrax.net>>> wrote:
> > >
> > > Hi Robert,
> > >
> > > I'm just curious: it's pretty fast compared to the original
> plan of a
> > release
> > > every two months. What's the reason to cut 2.4.0 now instead
> of end of
> > March ?
> > >
> > > I will do the Jira triage and update today.
> > >
> > > Regards
> > > JB
> > >
> > > On 02/27/2018 09:21 PM, Robert Bradshaw wrote:
> > > > I'm planning on cutting the 2.4.0 release branch soon
> (tomorrow?). I
> > see 13
> > > > open issues on JIRA [1], none of which are labeled as
> blockers. If there
> > > > are any that cannot be bumped to the next release, let me
> know soon.
> > > >
> > > > - Robert
> > > >
> > > >
> > > > [1]
> > > >
> > >
> > https://issues.apache.org/jira/browse/BEAM-3749?jql=
> project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%
> 22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%202.4.0
> > > >
> > >
> > > --
> > > Jean-Baptiste Onofré
> > > jbonofre@apache.org <ma...@apache.org>
> > <mailto:jbonofre@apache.org <ma...@apache.org>>
> > > http://blog.nanthrax.net
> > > Talend - http://www.talend.com
> > >
> >
> > --
> > Jean-Baptiste Onofré
> > jbonofre@apache.org <ma...@apache.org>
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
> >
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
--
Gus Katsiapis | Software Engineer | katsiapis@google.com | 650-918-7487
Re: Beam 2.4.0
Posted by Romain Manni-Bucau <rm...@gmail.com>.
Done, thanks.
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<https://rmannibucau.metawerx.net/> | Old Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
<https://www.packtpub.com/application-development/java-ee-8-high-performance>
2018-03-15 2:26 GMT+01:00 Jean-Baptiste Onofré <jb...@nanthrax.net>:
> Fully agree.
>
> If nobody jumps on it, I will tackle that tomorrow.
>
> Regards
> JB
> Le 14 mars 2018, à 18:03, Reuven Lax <re...@google.com> a écrit:
>>
>> Can you add a description and assign a reviewer (using R:). I think this
>> was basically ready to merge before modulo breaking compilation.
>>
>>
>> On Wed, Mar 14, 2018 at 10:01 AM Romain Manni-Bucau <
>> rmannibucau@gmail.com> wrote:
>>
>>> up, know it missed the 2.4 but can https://github.com/apache/
>>> beam/pull/4790 have some love? it really makes beam pretty unusable
>>> with direct runner,
>>> I start to have "// workaround for BEAM-3409" everywhere in my codebase
>>> which is quite bothering after 3 months :(
>>>
>>>
>>> Romain Manni-Bucau
>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>> <https://rmannibucau.metawerx.net/> | Old Blog
>>> <http://rmannibucau.wordpress.com> | Github
>>> <https://github.com/rmannibucau> | LinkedIn
>>> <https://www.linkedin.com/in/rmannibucau> | Book
>>> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>>
>>> 2018-03-06 14:44 GMT+01:00 Romain Manni-Bucau <rm...@gmail.com>:
>>>
>>>> Hi guys,
>>>>
>>>> tried to reapply the waitUntilFinish fix - without breaking the
>>>> compilation this time ;) - in https://github.com/apache/beam/pull/4790,
>>>> anyone able to help to review and move that forward?
>>>>
>>>>
>>>> Romain Manni-Bucau
>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>>> <https://rmannibucau.metawerx.net/> | Old Blog
>>>> <http://rmannibucau.wordpress.com> | Github
>>>> <https://github.com/rmannibucau> | LinkedIn
>>>> <https://www.linkedin.com/in/rmannibucau> | Book
>>>> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>>>
>>>> 2018-03-02 9:28 GMT+01:00 Robert Bradshaw <ro...@google.com>:
>>>>
>>>>> On Fri, Mar 2, 2018 at 12:01 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
>>>>> wrote:
>>>>>
>>>>>> Hi Ismaël,
>>>>>>
>>>>>> that's a good idea to show history.
>>>>>>
>>>>>> For me, the vote duration doesn't matter as we are in the release
>>>>>> process already.
>>>>>>
>>>>>
>>>>> A more relevant duration to track would probably be cut to final
>>>>> release. This both measures our investment in the release process, as well
>>>>> as how behind head is when we finally get the release out.
>>>>>
>>>>>
>>>>>> The gap between two releases is more significant.
>>>>>
>>>>>
>>>>> +1, this is what users see. (To clarify terminology, the "time between
>>>>> release" is the time between actually releasing x.y and x.y+1 that is most
>>>>> visible to end users, regardless of intermediate process like cutting and
>>>>> voting that we have.) Of course this gets thrown off if our
>>>>> time-to-prepare-a-release becomes a significant fraction of the desired
>>>>> time-between-releases.
>>>>>
>>>>> And clearly with an average of
>>>>>> 80 days (~ 3 months) it's two long. The idea is to reduce this
>>>>>> clearly. I
>>>>>> propose two months previously (including the vote period), so meaning
>>>>>> 6 weeks
>>>>>> between releases.
>>>>>>
>>>>>
>>>>> It seems there have been proposals for monthly, every 6 weeks
>>>>> (sesquimonthly?), and bimonthly.
>>>>>
>>>>>
>>>>>> Regarding the time we take for the PR review and merge, I think it's
>>>>>> a fair time
>>>>>> to give us time to include new improvements and features, but also to
>>>>>> fix bugs.
>>>>>>
>>>>>
>>>>> Concrete deadlines can provide good motivation to get around to doing
>>>>> reviews, cleaning up PRs, fixing bugs, etc. that you've been meaning to do
>>>>> but for whatever reason keep putting off. So I think it's still good
>>>>> practice to have some lead time that a release is coming for a chance for
>>>>> folks to get stuff in, while still being clear that we're not holding
>>>>> things back for new features and if you don't make the cut another one is
>>>>> close behind.
>>>>>
>>>>> Regards
>>>>>> JB
>>>>>>
>>>>>> On 03/01/2018 06:17 PM, Ismaël Mejía wrote:
>>>>>> > The average time just in the vote process for Beam since we are out
>>>>>> of the
>>>>>> > incubator is 17.5 days with an average of 75 days between versions.
>>>>>> >
>>>>>> > Version Vote Period No. days
>>>>>> > 2.3.0 30/01-16/02 17 days (83 days since last)
>>>>>> > 2.2.0 27/10-25/11 29 days (101 days since last)
>>>>>> > 2.1.0 11/07-16/08 36 days (92 days since last)
>>>>>> > 2.0.0 08/05-16/05 8 days (62 days since last)
>>>>>> > 0.6.0 10/03-15/03 5 days (37 days since last)
>>>>>> > 0.5.0 27/01-06/02 10 days
>>>>>> >
>>>>>> > I think we should have these numbers into account to refine the
>>>>>> distance between
>>>>>> > releases. If we want to follow strict time-based releases, what we
>>>>>> can probably
>>>>>> > refine is how we cut the release so we try to reduce release
>>>>>> overlaps and avoid
>>>>>> > rushing unnecessarily.
>>>>>> >
>>>>>> > Maybe we should follow the proposed 6 weeks for the next release
>>>>>> like this:
>>>>>> >
>>>>>> > - 4 weeks let’s say just after succesful vote and then cut the
>>>>>> release branch.
>>>>>> > - 1 week to burn the blocker list (good to have ones that don’t
>>>>>> make will be
>>>>>> > moved to the next release).
>>>>>> > - 1 week for the vote + RCs (in case the vote takes longer at least
>>>>>> the overlap
>>>>>> > between vote + next dev cycle will be smaller).
>>>>>> >
>>>>>> > If we do this for the next cycle we will have a 6 week ‘dev’ period
>>>>>> and then we
>>>>>> > will have optimistically an average of 2 weeks of ‘releasing’ and 6
>>>>>> weeks ‘dev’
>>>>>> > cycles.
>>>>>> >
>>>>>> > On Thu, Mar 1, 2018 at 6:46 AM, Jean-Baptiste Onofré <
>>>>>> jb@nanthrax.net> wrote:
>>>>>> >> About BEAM-3409, I did a review yesterday and it looks good to me.
>>>>>> We are
>>>>>> >> waiting for Thomas' feedback.
>>>>>> >>
>>>>>> >> Regards
>>>>>> >> JB
>>>>>> >> Le 1 mars 2018, à 09:38, Robert Bradshaw <ro...@google.com> a
>>>>>> écrit:
>>>>>> >>>
>>>>>> >>> Looking at the burn-down list, we have 5 remaining issues. None
>>>>>> of these
>>>>>> >>> are blockers, but all look like they're really close (reviewed,
>>>>>> review
>>>>>> >>> comments were addressed, waiting for a final LGTM). Specifically:
>>>>>> >>>
>>>>>> >>> BEAM-3409 (teardown issues): Thomas Groh had some concerns, could
>>>>>> you
>>>>>> >>> verify they have all been addressed?
>>>>>> >>> BEAM-3479 (DoFn classloader regression test): Kenn Knowles had
>>>>>> minor
>>>>>> >>> comments, looks like they were addressed, could you confirm?
>>>>>> >>> BEAM-3735 (Missing gaming release archetypes): Lukasz Cwik had
>>>>>> minor
>>>>>> >>> comments, looks like they were addressed, could you confirm?
>>>>>> >>> BEAM-3611 (KafkaIO.java splitting): Looks like this was resolved.
>>>>>> >>> BEAM-3762 (unlimited JCE for Dataflow Worker): LGTM pending
>>>>>> (currently
>>>>>> >>> running) tests passing.
>>>>>> >>>
>>>>>> >>> Let's see how many of these we can get in by, say, noon PST
>>>>>> tomorrow.
>>>>>> >>>
>>>>>> >>>
>>>>>> >>>
>>>>>> >>>
>>>>>> >>>
>>>>>> >>> On Wed, Feb 28, 2018 at 9:26 PM Robert Bradshaw <
>>>>>> robertwb@google.com>
>>>>>> >>> wrote:
>>>>>> >>>>
>>>>>> >>>> I tend to fall into the "release early, release often" camp in
>>>>>> general,
>>>>>> >>>> but for this one I'm particularly anxious to get the faster
>>>>>> Python direct
>>>>>> >>>> runner out in the hands of TFT/TFX users (and in particular have
>>>>>> an eye on
>>>>>> >>>> https://www.tensorflow.org/dev-summit/ which I think can be a
>>>>>> healthy source
>>>>>> >>>> of Beam users).
>>>>>> >>>>
>>>>>> >>>>
>>>>>> >>>> On Wed, Feb 28, 2018 at 7:01 PM Jean-Baptiste Onofré <
>>>>>> jb@nanthrax.net>
>>>>>> >>>> wrote:
>>>>>> >>>>>
>>>>>> >>>>> Hi Gus,
>>>>>> >>>>>
>>>>>> >>>>> Thanks for the update, it makes sense.
>>>>>> >>>>>
>>>>>> >>>>> Regards
>>>>>> >>>>> JB
>>>>>> >>>>>
>>>>>> >>>>> On 03/01/2018 02:59 AM, Konstantinos Katsiapis wrote:
>>>>>> >>>>>> Hi Jean-Baptiste,
>>>>>> >>>>>>
>>>>>> >>>>>> I can speak from the perspective of tf.transform
>>>>>> >>>>>> < https://github.com/tensorflow/transform> (TFT) in
>>>>>> particular and TFX
>>>>>> >>>>>> < https://research.google.com/pubs/pub46484.html> libs in
>>>>>> general, in
>>>>>> >>>>>> case it is
>>>>>> >>>>>> useful.
>>>>>> >>>>>>
>>>>>> >>>>>> TFX distributed computation has 2 "large" dependencies, namely
>>>>>> >>>>>> TensorFlow and
>>>>>> >>>>>> Apache Beam, each on their own release schedule.
>>>>>> >>>>>> As such, releasing of new TFX functionality often (but not
>>>>>> always)
>>>>>> >>>>>> depends on
>>>>>> >>>>>> (and is blocked by) releases of *both* TensorFlow *and* Apache
>>>>>> Beam.
>>>>>> >>>>>>
>>>>>> >>>>>> Synchronizing releases across such large projects and
>>>>>> organizations is
>>>>>> >>>>>> likely
>>>>>> >>>>>> hard, so from our perspective having *frequent* releases of
>>>>>> Tensorflow
>>>>>> >>>>>> or Apache
>>>>>> >>>>>> Beam (and better yet both) decreases the time for which we are
>>>>>> blocked
>>>>>> >>>>>> on
>>>>>> >>>>>> releasing our features.
>>>>>> >>>>>>
>>>>>> >>>>>> In light of this, I would vote for more frequent releases in
>>>>>> general,
>>>>>> >>>>>> and for a
>>>>>> >>>>>> Beam 2.4 release soon in particular (as TFT 0.6 depends on it).
>>>>>> >>>>>>
>>>>>> >>>>>> Thanks,
>>>>>> >>>>>> Gus
>>>>>> >>>>>>
>>>>>> >>>>>> On Tue, Feb 27, 2018 at 10:29 PM, Jean-Baptiste Onofré <
>>>>>> >>>>>> jb@nanthrax.net
>>>>>> >>>>>> <mailto: jb@nanthrax.net>> wrote:
>>>>>> >>>>>>
>>>>>> >>>>>> By the way, if third party projects based on Beam (Google
>>>>>> >>>>>> Dataflow, Talend
>>>>>> >>>>>> DataStream, and others) need a release (including some
>>>>>> features),
>>>>>> >>>>>> it's better to
>>>>>> >>>>>> clearly state this on the mailing list.
>>>>>> >>>>>>
>>>>>> >>>>>> At Apache Karaf, I have lot of projects based on it
>>>>>> (OpenDaylight,
>>>>>> >>>>>> OpenHAB,
>>>>>> >>>>>> Websphere, ...). They just ask for the release schedule
>>>>>> and they
>>>>>> >>>>>> align with
>>>>>> >>>>>> these release. As a best effort, I'm always trying to move
>>>>>> fast
>>>>>> >>>>>> when a release
>>>>>> >>>>>> is asked.
>>>>>> >>>>>>
>>>>>> >>>>>> So, if 2.4.0 is required by third party, no problem to
>>>>>> "ask for a
>>>>>> >>>>>> release".
>>>>>> >>>>>>
>>>>>> >>>>>> Regards
>>>>>> >>>>>> JB
>>>>>> >>>>>>
>>>>>> >>>>>> On 02/28/2018 04:17 AM, Reuven Lax wrote:
>>>>>> >>>>>> > It's been six weeks since you proposed beam 2.3.0. so
>>>>>> assuming
>>>>>> >>>>>> the same time
>>>>>> >>>>>> > scale for this release, that's 1.5 months between
>>>>>> releases.
>>>>>> >>>>>> Slightly faster than
>>>>>> >>>>>> > 2 months, but not by much.
>>>>>> >>>>>> >
>>>>>> >>>>>> > I do seem to remember that the original goal for beam was
>>>>>> >>>>>> monthly releases though.
>>>>>> >>>>>> >
>>>>>> >>>>>> > Reuven
>>>>>> >>>>>> >
>>>>>> >>>>>> > On Tue, Feb 27, 2018, 9:12 PM Jean-Baptiste Onofré <
>>>>>> >>>>>> jb@nanthrax.net <mailto: jb@nanthrax.net>
>>>>>> >>>>>> > <mailto: jb@nanthrax.net <mailto: jb@nanthrax.net>>>
>>>>>> wrote:
>>>>>> >>>>>> >
>>>>>> >>>>>> > Hi Reuven,
>>>>>> >>>>>> >
>>>>>> >>>>>> > In a previous thread (about Beam project execution),
>>>>>> I
>>>>>> >>>>>> proposed a release every
>>>>>> >>>>>> > two months (as a best effort), I will find the
>>>>>> e-mail.
>>>>>> >>>>>> >
>>>>>> >>>>>> > Beam 2.3.0 has been released "officially" on
>>>>>> February 16th,
>>>>>> >>>>>> so two week ago
>>>>>> >>>>>> > roughly. I would have expected 2.4.0 not before end
>>>>>> of
>>>>>> >>>>>> March.
>>>>>> >>>>>> >
>>>>>> >>>>>> > If we have issue we want to fix fast, then 2.3.1 is
>>>>>> good. If
>>>>>> >>>>>> it's a new release
>>>>>> >>>>>> > in the pace, it's pretty fast and might "confuse"
>>>>>> our users.
>>>>>> >>>>>> >
>>>>>> >>>>>> > That's why I'm curious ;)
>>>>>> >>>>>> >
>>>>>> >>>>>> > Regards
>>>>>> >>>>>> > JB
>>>>>> >>>>>> >
>>>>>> >>>>>> > On 02/28/2018 03:50 AM, Reuven Lax wrote:
>>>>>> >>>>>> > > Wasn't the original statement monthly releases?
>>>>>> We've
>>>>>> >>>>>> never realistically
>>>>>> >>>>>> > > managed that, but Robert's proposed cut will be on
>>>>>> a
>>>>>> >>>>>> 6-week pace.
>>>>>> >>>>>> > >
>>>>>> >>>>>> > > On Tue, Feb 27, 2018, 8:48 PM Jean-Baptiste Onofré
>>>>>> <
>>>>>> >>>>>> jb@nanthrax.net <mailto: jb@nanthrax.net>
>>>>>> >>>>>> > <mailto: jb@nanthrax.net <mailto: jb@nanthrax.net>>
>>>>>> >>>>>> > > <mailto: jb@nanthrax.net <mailto: jb@nanthrax.net>
>>>>>> >>>>>> <mailto: jb@nanthrax.net
>>>>>> >>>>>> <mailto: jb@nanthrax.net>>>> wrote:
>>>>>> >>>>>> > >
>>>>>> >>>>>> > > Hi Robert,
>>>>>> >>>>>> > >
>>>>>> >>>>>> > > I'm just curious: it's pretty fast compared to
>>>>>> the
>>>>>> >>>>>> original plan of a
>>>>>> >>>>>> > release
>>>>>> >>>>>> > > every two months. What's the reason to cut
>>>>>> 2.4.0 now
>>>>>> >>>>>> instead of end of
>>>>>> >>>>>> > March ?
>>>>>> >>>>>> > >
>>>>>> >>>>>> > > I will do the Jira triage and update today.
>>>>>> >>>>>> > >
>>>>>> >>>>>> > > Regards
>>>>>> >>>>>> > > JB
>>>>>> >>>>>> > >
>>>>>> >>>>>> > > On 02/27/2018 09:21 PM, Robert Bradshaw wrote:
>>>>>> >>>>>> > > > I'm planning on cutting the 2.4.0 release
>>>>>> branch
>>>>>> >>>>>> soon (tomorrow?). I
>>>>>> >>>>>> > see 13
>>>>>> >>>>>> > > > open issues on JIRA [1], none of which are
>>>>>> labeled
>>>>>> >>>>>> as blockers. If there
>>>>>> >>>>>> > > > are any that cannot be bumped to the next
>>>>>> release,
>>>>>> >>>>>> let me know soon.
>>>>>> >>>>>> > > >
>>>>>> >>>>>> > > > - Robert
>>>>>> >>>>>> > > >
>>>>>> >>>>>> > > >
>>>>>> >>>>>> > > > [1]
>>>>>> >>>>>> > > >
>>>>>> >>>>>> > >
>>>>>> >>>>>> >
>>>>>> >>>>>> https://issues.apache.org/jira/browse/BEAM-3749?jql=
>>>>>> project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%
>>>>>> 22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%202.4.0
>>>>>> >>>>>> <
>>>>>> >>>>>> https://issues.apache.org/jira/browse/BEAM-3749?jql=
>>>>>> project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%
>>>>>> 22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%202.4.0>
>>>>>> >>>>>> > > >
>>>>>> >>>>>> > >
>>>>>> >>>>>> > > --
>>>>>> >>>>>> > > Jean-Baptiste Onofré
>>>>>> >>>>>> > > jbonofre@apache.org <mailto:
>>>>>> jbonofre@apache.org>
>>>>>> >>>>>> <mailto: jbonofre@apache.org
>>>>>> >>>>>> <mailto: jbonofre@apache.org>>
>>>>>> >>>>>> > <mailto: jbonofre@apache.org <mailto:
>>>>>> jbonofre@apache.org>
>>>>>> >>>>>> <mailto: jbonofre@apache.org <mailto: jbonofre@apache.org
>>>>>> >>>
>>>>>> >>>>>> > > http://blog.nanthrax.net
>>>>>> >>>>>> > > Talend - http://www.talend.com
>>>>>> >>>>>> > >
>>>>>> >>>>>> >
>>>>>> >>>>>> > --
>>>>>> >>>>>> > Jean-Baptiste Onofré
>>>>>> >>>>>> > jbonofre@apache.org <mailto: jbonofre@apache.org>
>>>>>> >>>>>> <mailto: jbonofre@apache.org <mailto: jbonofre@apache.org
>>>>>> >>
>>>>>> >>>>>> > http://blog.nanthrax.net
>>>>>> >>>>>> > Talend - http://www.talend.com
>>>>>> >>>>>> >
>>>>>> >>>>>>
>>>>>> >>>>>> --
>>>>>> >>>>>> Jean-Baptiste Onofré
>>>>>> >>>>>> jbonofre@apache.org <mailto: jbonofre@apache.org>
>>>>>> >>>>>> http://blog.nanthrax.net
>>>>>> >>>>>> Talend - http://www.talend.com
>>>>>> >>>>>>
>>>>>> >>>>>>
>>>>>> >>>>>>
>>>>>> >>>>>>
>>>>>> >>>>>> --
>>>>>> >>>>>> Gus Katsiapis | Software Engineer | katsiapis@google.com
>>>>>> >>>>>> <mailto: katsiapis@google.com> | 650-918-7487
>>>>>> >>>>>
>>>>>> >>>>> --
>>>>>> >>>>> Jean-Baptiste Onofré
>>>>>> >>>>> jbonofre@apache.org
>>>>>> >>>>> http://blog.nanthrax.net
>>>>>> >>>>> Talend - http://www.talend.com
>>>>>>
>>>>>> --
>>>>>> Jean-Baptiste Onofré
>>>>>> jbonofre@apache.org
>>>>>> http://blog.nanthrax.net
>>>>>> Talend - http://www.talend.com
>>>>>>
>>>>>
>>>>
>>>
Re: Beam 2.4.0
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Fully agree.
If nobody jumps on it, I will tackle that tomorrow.
Regards
JB
Le 14 mars 2018 à 18:03, à 18:03, Reuven Lax <re...@google.com> a écrit:
>Can you add a description and assign a reviewer (using R:). I think
>this
>was basically ready to merge before modulo breaking compilation.
>
>
>On Wed, Mar 14, 2018 at 10:01 AM Romain Manni-Bucau
><rm...@gmail.com>
>wrote:
>
>> up, know it missed the 2.4 but can
>> https://github.com/apache/beam/pull/4790 have some love? it really
>makes
>> beam pretty unusable with direct runner,
>> I start to have "// workaround for BEAM-3409" everywhere in my
>codebase
>> which is quite bothering after 3 months :(
>>
>>
>> Romain Manni-Bucau
>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>> <https://rmannibucau.metawerx.net/> | Old Blog
>> <http://rmannibucau.wordpress.com> | Github
>> <https://github.com/rmannibucau> | LinkedIn
>> <https://www.linkedin.com/in/rmannibucau> | Book
>>
><https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>
>> 2018-03-06 14:44 GMT+01:00 Romain Manni-Bucau
><rm...@gmail.com>:
>>
>>> Hi guys,
>>>
>>> tried to reapply the waitUntilFinish fix - without breaking the
>>> compilation this time ;) - in
>https://github.com/apache/beam/pull/4790,
>>> anyone able to help to review and move that forward?
>>>
>>>
>>> Romain Manni-Bucau
>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>> <https://rmannibucau.metawerx.net/> | Old Blog
>>> <http://rmannibucau.wordpress.com> | Github
>>> <https://github.com/rmannibucau> | LinkedIn
>>> <https://www.linkedin.com/in/rmannibucau> | Book
>>>
><https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>>
>>> 2018-03-02 9:28 GMT+01:00 Robert Bradshaw <ro...@google.com>:
>>>
>>>> On Fri, Mar 2, 2018 at 12:01 AM Jean-Baptiste Onofré
><jb...@nanthrax.net>
>>>> wrote:
>>>>
>>>>> Hi Ismaël,
>>>>>
>>>>> that's a good idea to show history.
>>>>>
>>>>> For me, the vote duration doesn't matter as we are in the release
>>>>> process already.
>>>>>
>>>>
>>>> A more relevant duration to track would probably be cut to final
>>>> release. This both measures our investment in the release process,
>as well
>>>> as how behind head is when we finally get the release out.
>>>>
>>>>
>>>>> The gap between two releases is more significant.
>>>>
>>>>
>>>> +1, this is what users see. (To clarify terminology, the "time
>between
>>>> release" is the time between actually releasing x.y and x.y+1 that
>is most
>>>> visible to end users, regardless of intermediate process like
>cutting and
>>>> voting that we have.) Of course this gets thrown off if our
>>>> time-to-prepare-a-release becomes a significant fraction of the
>desired
>>>> time-between-releases.
>>>>
>>>> And clearly with an average of
>>>>> 80 days (~ 3 months) it's two long. The idea is to reduce this
>clearly.
>>>>> I
>>>>> propose two months previously (including the vote period), so
>meaning 6
>>>>> weeks
>>>>> between releases.
>>>>>
>>>>
>>>> It seems there have been proposals for monthly, every 6 weeks
>>>> (sesquimonthly?), and bimonthly.
>>>>
>>>>
>>>>> Regarding the time we take for the PR review and merge, I think
>it's a
>>>>> fair time
>>>>> to give us time to include new improvements and features, but also
>to
>>>>> fix bugs.
>>>>>
>>>>
>>>> Concrete deadlines can provide good motivation to get around to
>doing
>>>> reviews, cleaning up PRs, fixing bugs, etc. that you've been
>meaning to do
>>>> but for whatever reason keep putting off. So I think it's still
>good
>>>> practice to have some lead time that a release is coming for a
>chance for
>>>> folks to get stuff in, while still being clear that we're not
>holding
>>>> things back for new features and if you don't make the cut another
>one is
>>>> close behind.
>>>>
>>>> Regards
>>>>> JB
>>>>>
>>>>> On 03/01/2018 06:17 PM, Ismaël Mejía wrote:
>>>>> > The average time just in the vote process for Beam since we are
>out
>>>>> of the
>>>>> > incubator is 17.5 days with an average of 75 days between
>versions.
>>>>> >
>>>>> > Version Vote Period No. days
>>>>> > 2.3.0 30/01-16/02 17 days (83 days since last)
>>>>> > 2.2.0 27/10-25/11 29 days (101 days since last)
>>>>> > 2.1.0 11/07-16/08 36 days (92 days since last)
>>>>> > 2.0.0 08/05-16/05 8 days (62 days since last)
>>>>> > 0.6.0 10/03-15/03 5 days (37 days since last)
>>>>> > 0.5.0 27/01-06/02 10 days
>>>>> >
>>>>> > I think we should have these numbers into account to refine the
>>>>> distance between
>>>>> > releases. If we want to follow strict time-based releases, what
>we
>>>>> can probably
>>>>> > refine is how we cut the release so we try to reduce release
>overlaps
>>>>> and avoid
>>>>> > rushing unnecessarily.
>>>>> >
>>>>> > Maybe we should follow the proposed 6 weeks for the next release
>like
>>>>> this:
>>>>> >
>>>>> > - 4 weeks let’s say just after succesful vote and then cut the
>>>>> release branch.
>>>>> > - 1 week to burn the blocker list (good to have ones that don’t
>make
>>>>> will be
>>>>> > moved to the next release).
>>>>> > - 1 week for the vote + RCs (in case the vote takes longer at
>least
>>>>> the overlap
>>>>> > between vote + next dev cycle will be smaller).
>>>>> >
>>>>> > If we do this for the next cycle we will have a 6 week ‘dev’
>period
>>>>> and then we
>>>>> > will have optimistically an average of 2 weeks of ‘releasing’
>and 6
>>>>> weeks ‘dev’
>>>>> > cycles.
>>>>> >
>>>>> > On Thu, Mar 1, 2018 at 6:46 AM, Jean-Baptiste Onofré
><jb...@nanthrax.net>
>>>>> wrote:
>>>>> >> About BEAM-3409, I did a review yesterday and it looks good to
>me.
>>>>> We are
>>>>> >> waiting for Thomas' feedback.
>>>>> >>
>>>>> >> Regards
>>>>> >> JB
>>>>> >> Le 1 mars 2018, à 09:38, Robert Bradshaw <ro...@google.com>
>a
>>>>> écrit:
>>>>> >>>
>>>>> >>> Looking at the burn-down list, we have 5 remaining issues.
>None of
>>>>> these
>>>>> >>> are blockers, but all look like they're really close
>(reviewed,
>>>>> review
>>>>> >>> comments were addressed, waiting for a final LGTM).
>Specifically:
>>>>> >>>
>>>>> >>> BEAM-3409 (teardown issues): Thomas Groh had some concerns,
>could
>>>>> you
>>>>> >>> verify they have all been addressed?
>>>>> >>> BEAM-3479 (DoFn classloader regression test): Kenn Knowles
>had
>>>>> minor
>>>>> >>> comments, looks like they were addressed, could you confirm?
>>>>> >>> BEAM-3735 (Missing gaming release archetypes): Lukasz Cwik had
>minor
>>>>> >>> comments, looks like they were addressed, could you confirm?
>>>>> >>> BEAM-3611 (KafkaIO.java splitting): Looks like this was
>resolved.
>>>>> >>> BEAM-3762 (unlimited JCE for Dataflow Worker): LGTM pending
>>>>> (currently
>>>>> >>> running) tests passing.
>>>>> >>>
>>>>> >>> Let's see how many of these we can get in by, say, noon PST
>>>>> tomorrow.
>>>>> >>>
>>>>> >>>
>>>>> >>>
>>>>> >>>
>>>>> >>>
>>>>> >>> On Wed, Feb 28, 2018 at 9:26 PM Robert Bradshaw <
>>>>> robertwb@google.com>
>>>>> >>> wrote:
>>>>> >>>>
>>>>> >>>> I tend to fall into the "release early, release often" camp
>in
>>>>> general,
>>>>> >>>> but for this one I'm particularly anxious to get the faster
>Python
>>>>> direct
>>>>> >>>> runner out in the hands of TFT/TFX users (and in particular
>have
>>>>> an eye on
>>>>> >>>> https://www.tensorflow.org/dev-summit/ which I think can be a
>>>>> healthy source
>>>>> >>>> of Beam users).
>>>>> >>>>
>>>>> >>>>
>>>>> >>>> On Wed, Feb 28, 2018 at 7:01 PM Jean-Baptiste Onofré <
>>>>> jb@nanthrax.net>
>>>>> >>>> wrote:
>>>>> >>>>>
>>>>> >>>>> Hi Gus,
>>>>> >>>>>
>>>>> >>>>> Thanks for the update, it makes sense.
>>>>> >>>>>
>>>>> >>>>> Regards
>>>>> >>>>> JB
>>>>> >>>>>
>>>>> >>>>> On 03/01/2018 02:59 AM, Konstantinos Katsiapis wrote:
>>>>> >>>>>> Hi Jean-Baptiste,
>>>>> >>>>>>
>>>>> >>>>>> I can speak from the perspective of tf.transform
>>>>> >>>>>> < https://github.com/tensorflow/transform> (TFT) in
>particular
>>>>> and TFX
>>>>> >>>>>> < https://research.google.com/pubs/pub46484.html> libs in
>>>>> general, in
>>>>> >>>>>> case it is
>>>>> >>>>>> useful.
>>>>> >>>>>>
>>>>> >>>>>> TFX distributed computation has 2 "large" dependencies,
>namely
>>>>> >>>>>> TensorFlow and
>>>>> >>>>>> Apache Beam, each on their own release schedule.
>>>>> >>>>>> As such, releasing of new TFX functionality often (but not
>>>>> always)
>>>>> >>>>>> depends on
>>>>> >>>>>> (and is blocked by) releases of *both* TensorFlow *and*
>Apache
>>>>> Beam.
>>>>> >>>>>>
>>>>> >>>>>> Synchronizing releases across such large projects and
>>>>> organizations is
>>>>> >>>>>> likely
>>>>> >>>>>> hard, so from our perspective having *frequent* releases of
>>>>> Tensorflow
>>>>> >>>>>> or Apache
>>>>> >>>>>> Beam (and better yet both) decreases the time for which we
>are
>>>>> blocked
>>>>> >>>>>> on
>>>>> >>>>>> releasing our features.
>>>>> >>>>>>
>>>>> >>>>>> In light of this, I would vote for more frequent releases
>in
>>>>> general,
>>>>> >>>>>> and for a
>>>>> >>>>>> Beam 2.4 release soon in particular (as TFT 0.6 depends on
>it).
>>>>> >>>>>>
>>>>> >>>>>> Thanks,
>>>>> >>>>>> Gus
>>>>> >>>>>>
>>>>> >>>>>> On Tue, Feb 27, 2018 at 10:29 PM, Jean-Baptiste Onofré <
>>>>> >>>>>> jb@nanthrax.net
>>>>> >>>>>> <mailto: jb@nanthrax.net>> wrote:
>>>>> >>>>>>
>>>>> >>>>>> By the way, if third party projects based on Beam
>(Google
>>>>> >>>>>> Dataflow, Talend
>>>>> >>>>>> DataStream, and others) need a release (including some
>>>>> features),
>>>>> >>>>>> it's better to
>>>>> >>>>>> clearly state this on the mailing list.
>>>>> >>>>>>
>>>>> >>>>>> At Apache Karaf, I have lot of projects based on it
>>>>> (OpenDaylight,
>>>>> >>>>>> OpenHAB,
>>>>> >>>>>> Websphere, ...). They just ask for the release
>schedule and
>>>>> they
>>>>> >>>>>> align with
>>>>> >>>>>> these release. As a best effort, I'm always trying to
>move
>>>>> fast
>>>>> >>>>>> when a release
>>>>> >>>>>> is asked.
>>>>> >>>>>>
>>>>> >>>>>> So, if 2.4.0 is required by third party, no problem to
>"ask
>>>>> for a
>>>>> >>>>>> release".
>>>>> >>>>>>
>>>>> >>>>>> Regards
>>>>> >>>>>> JB
>>>>> >>>>>>
>>>>> >>>>>> On 02/28/2018 04:17 AM, Reuven Lax wrote:
>>>>> >>>>>> > It's been six weeks since you proposed beam 2.3.0. so
>>>>> assuming
>>>>> >>>>>> the same time
>>>>> >>>>>> > scale for this release, that's 1.5 months between
>releases.
>>>>> >>>>>> Slightly faster than
>>>>> >>>>>> > 2 months, but not by much.
>>>>> >>>>>> >
>>>>> >>>>>> > I do seem to remember that the original goal for beam
>was
>>>>> >>>>>> monthly releases though.
>>>>> >>>>>> >
>>>>> >>>>>> > Reuven
>>>>> >>>>>> >
>>>>> >>>>>> > On Tue, Feb 27, 2018, 9:12 PM Jean-Baptiste Onofré <
>>>>> >>>>>> jb@nanthrax.net <mailto: jb@nanthrax.net>
>>>>> >>>>>> > <mailto: jb@nanthrax.net <mailto: jb@nanthrax.net>>>
>>>>> wrote:
>>>>> >>>>>> >
>>>>> >>>>>> > Hi Reuven,
>>>>> >>>>>> >
>>>>> >>>>>> > In a previous thread (about Beam project
>execution), I
>>>>> >>>>>> proposed a release every
>>>>> >>>>>> > two months (as a best effort), I will find the
>e-mail.
>>>>> >>>>>> >
>>>>> >>>>>> > Beam 2.3.0 has been released "officially" on
>February
>>>>> 16th,
>>>>> >>>>>> so two week ago
>>>>> >>>>>> > roughly. I would have expected 2.4.0 not before
>end of
>>>>> >>>>>> March.
>>>>> >>>>>> >
>>>>> >>>>>> > If we have issue we want to fix fast, then 2.3.1
>is
>>>>> good. If
>>>>> >>>>>> it's a new release
>>>>> >>>>>> > in the pace, it's pretty fast and might "confuse"
>our
>>>>> users.
>>>>> >>>>>> >
>>>>> >>>>>> > That's why I'm curious ;)
>>>>> >>>>>> >
>>>>> >>>>>> > Regards
>>>>> >>>>>> > JB
>>>>> >>>>>> >
>>>>> >>>>>> > On 02/28/2018 03:50 AM, Reuven Lax wrote:
>>>>> >>>>>> > > Wasn't the original statement monthly releases?
>We've
>>>>> >>>>>> never realistically
>>>>> >>>>>> > > managed that, but Robert's proposed cut will be
>on a
>>>>> >>>>>> 6-week pace.
>>>>> >>>>>> > >
>>>>> >>>>>> > > On Tue, Feb 27, 2018, 8:48 PM Jean-Baptiste
>Onofré <
>>>>> >>>>>> jb@nanthrax.net <mailto: jb@nanthrax.net>
>>>>> >>>>>> > <mailto: jb@nanthrax.net <mailto:
>jb@nanthrax.net>>
>>>>> >>>>>> > > <mailto: jb@nanthrax.net <mailto:
>jb@nanthrax.net>
>>>>> >>>>>> <mailto: jb@nanthrax.net
>>>>> >>>>>> <mailto: jb@nanthrax.net>>>> wrote:
>>>>> >>>>>> > >
>>>>> >>>>>> > > Hi Robert,
>>>>> >>>>>> > >
>>>>> >>>>>> > > I'm just curious: it's pretty fast compared
>to
>>>>> the
>>>>> >>>>>> original plan of a
>>>>> >>>>>> > release
>>>>> >>>>>> > > every two months. What's the reason to cut
>2.4.0
>>>>> now
>>>>> >>>>>> instead of end of
>>>>> >>>>>> > March ?
>>>>> >>>>>> > >
>>>>> >>>>>> > > I will do the Jira triage and update today.
>>>>> >>>>>> > >
>>>>> >>>>>> > > Regards
>>>>> >>>>>> > > JB
>>>>> >>>>>> > >
>>>>> >>>>>> > > On 02/27/2018 09:21 PM, Robert Bradshaw
>wrote:
>>>>> >>>>>> > > > I'm planning on cutting the 2.4.0 release
>>>>> branch
>>>>> >>>>>> soon (tomorrow?). I
>>>>> >>>>>> > see 13
>>>>> >>>>>> > > > open issues on JIRA [1], none of which
>are
>>>>> labeled
>>>>> >>>>>> as blockers. If there
>>>>> >>>>>> > > > are any that cannot be bumped to the next
>>>>> release,
>>>>> >>>>>> let me know soon.
>>>>> >>>>>> > > >
>>>>> >>>>>> > > > - Robert
>>>>> >>>>>> > > >
>>>>> >>>>>> > > >
>>>>> >>>>>> > > > [1]
>>>>> >>>>>> > > >
>>>>> >>>>>> > >
>>>>> >>>>>> >
>>>>> >>>>>>
>>>>>
>https://issues.apache.org/jira/browse/BEAM-3749?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%202.4.0
>>>>> >>>>>> <
>>>>> >>>>>>
>>>>>
>https://issues.apache.org/jira/browse/BEAM-3749?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%202.4.0
>>>>> >
>>>>> >>>>>> > > >
>>>>> >>>>>> > >
>>>>> >>>>>> > > --
>>>>> >>>>>> > > Jean-Baptiste Onofré
>>>>> >>>>>> > > jbonofre@apache.org <mailto:
>>>>> jbonofre@apache.org>
>>>>> >>>>>> <mailto: jbonofre@apache.org
>>>>> >>>>>> <mailto: jbonofre@apache.org>>
>>>>> >>>>>> > <mailto: jbonofre@apache.org <mailto:
>>>>> jbonofre@apache.org>
>>>>> >>>>>> <mailto: jbonofre@apache.org <mailto:
>jbonofre@apache.org>>>
>>>>> >>>>>> > > http://blog.nanthrax.net
>>>>> >>>>>> > > Talend - http://www.talend.com
>>>>> >>>>>> > >
>>>>> >>>>>> >
>>>>> >>>>>> > --
>>>>> >>>>>> > Jean-Baptiste Onofré
>>>>> >>>>>> > jbonofre@apache.org <mailto:
>jbonofre@apache.org>
>>>>> >>>>>> <mailto: jbonofre@apache.org <mailto:
>jbonofre@apache.org>>
>>>>> >>>>>> > http://blog.nanthrax.net
>>>>> >>>>>> > Talend - http://www.talend.com
>>>>> >>>>>> >
>>>>> >>>>>>
>>>>> >>>>>> --
>>>>> >>>>>> Jean-Baptiste Onofré
>>>>> >>>>>> jbonofre@apache.org <mailto: jbonofre@apache.org>
>>>>> >>>>>> http://blog.nanthrax.net
>>>>> >>>>>> Talend - http://www.talend.com
>>>>> >>>>>>
>>>>> >>>>>>
>>>>> >>>>>>
>>>>> >>>>>>
>>>>> >>>>>> --
>>>>> >>>>>> Gus Katsiapis | Software Engineer | katsiapis@google.com
>>>>> >>>>>> <mailto: katsiapis@google.com> | 650-918-7487
>>>>> >>>>>
>>>>> >>>>> --
>>>>> >>>>> Jean-Baptiste Onofré
>>>>> >>>>> jbonofre@apache.org
>>>>> >>>>> http://blog.nanthrax.net
>>>>> >>>>> Talend - http://www.talend.com
>>>>>
>>>>> --
>>>>> Jean-Baptiste Onofré
>>>>> jbonofre@apache.org
>>>>> http://blog.nanthrax.net
>>>>> Talend - http://www.talend.com
>>>>>
>>>>
>>>
>>
Re: Beam 2.4.0
Posted by Reuven Lax <re...@google.com>.
Can you add a description and assign a reviewer (using R:). I think this
was basically ready to merge before modulo breaking compilation.
On Wed, Mar 14, 2018 at 10:01 AM Romain Manni-Bucau <rm...@gmail.com>
wrote:
> up, know it missed the 2.4 but can
> https://github.com/apache/beam/pull/4790 have some love? it really makes
> beam pretty unusable with direct runner,
> I start to have "// workaround for BEAM-3409" everywhere in my codebase
> which is quite bothering after 3 months :(
>
>
> Romain Manni-Bucau
> @rmannibucau <https://twitter.com/rmannibucau> | Blog
> <https://rmannibucau.metawerx.net/> | Old Blog
> <http://rmannibucau.wordpress.com> | Github
> <https://github.com/rmannibucau> | LinkedIn
> <https://www.linkedin.com/in/rmannibucau> | Book
> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>
> 2018-03-06 14:44 GMT+01:00 Romain Manni-Bucau <rm...@gmail.com>:
>
>> Hi guys,
>>
>> tried to reapply the waitUntilFinish fix - without breaking the
>> compilation this time ;) - in https://github.com/apache/beam/pull/4790,
>> anyone able to help to review and move that forward?
>>
>>
>> Romain Manni-Bucau
>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>> <https://rmannibucau.metawerx.net/> | Old Blog
>> <http://rmannibucau.wordpress.com> | Github
>> <https://github.com/rmannibucau> | LinkedIn
>> <https://www.linkedin.com/in/rmannibucau> | Book
>> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>
>> 2018-03-02 9:28 GMT+01:00 Robert Bradshaw <ro...@google.com>:
>>
>>> On Fri, Mar 2, 2018 at 12:01 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
>>> wrote:
>>>
>>>> Hi Ismaël,
>>>>
>>>> that's a good idea to show history.
>>>>
>>>> For me, the vote duration doesn't matter as we are in the release
>>>> process already.
>>>>
>>>
>>> A more relevant duration to track would probably be cut to final
>>> release. This both measures our investment in the release process, as well
>>> as how behind head is when we finally get the release out.
>>>
>>>
>>>> The gap between two releases is more significant.
>>>
>>>
>>> +1, this is what users see. (To clarify terminology, the "time between
>>> release" is the time between actually releasing x.y and x.y+1 that is most
>>> visible to end users, regardless of intermediate process like cutting and
>>> voting that we have.) Of course this gets thrown off if our
>>> time-to-prepare-a-release becomes a significant fraction of the desired
>>> time-between-releases.
>>>
>>> And clearly with an average of
>>>> 80 days (~ 3 months) it's two long. The idea is to reduce this clearly.
>>>> I
>>>> propose two months previously (including the vote period), so meaning 6
>>>> weeks
>>>> between releases.
>>>>
>>>
>>> It seems there have been proposals for monthly, every 6 weeks
>>> (sesquimonthly?), and bimonthly.
>>>
>>>
>>>> Regarding the time we take for the PR review and merge, I think it's a
>>>> fair time
>>>> to give us time to include new improvements and features, but also to
>>>> fix bugs.
>>>>
>>>
>>> Concrete deadlines can provide good motivation to get around to doing
>>> reviews, cleaning up PRs, fixing bugs, etc. that you've been meaning to do
>>> but for whatever reason keep putting off. So I think it's still good
>>> practice to have some lead time that a release is coming for a chance for
>>> folks to get stuff in, while still being clear that we're not holding
>>> things back for new features and if you don't make the cut another one is
>>> close behind.
>>>
>>> Regards
>>>> JB
>>>>
>>>> On 03/01/2018 06:17 PM, Ismaël Mejía wrote:
>>>> > The average time just in the vote process for Beam since we are out
>>>> of the
>>>> > incubator is 17.5 days with an average of 75 days between versions.
>>>> >
>>>> > Version Vote Period No. days
>>>> > 2.3.0 30/01-16/02 17 days (83 days since last)
>>>> > 2.2.0 27/10-25/11 29 days (101 days since last)
>>>> > 2.1.0 11/07-16/08 36 days (92 days since last)
>>>> > 2.0.0 08/05-16/05 8 days (62 days since last)
>>>> > 0.6.0 10/03-15/03 5 days (37 days since last)
>>>> > 0.5.0 27/01-06/02 10 days
>>>> >
>>>> > I think we should have these numbers into account to refine the
>>>> distance between
>>>> > releases. If we want to follow strict time-based releases, what we
>>>> can probably
>>>> > refine is how we cut the release so we try to reduce release overlaps
>>>> and avoid
>>>> > rushing unnecessarily.
>>>> >
>>>> > Maybe we should follow the proposed 6 weeks for the next release like
>>>> this:
>>>> >
>>>> > - 4 weeks let’s say just after succesful vote and then cut the
>>>> release branch.
>>>> > - 1 week to burn the blocker list (good to have ones that don’t make
>>>> will be
>>>> > moved to the next release).
>>>> > - 1 week for the vote + RCs (in case the vote takes longer at least
>>>> the overlap
>>>> > between vote + next dev cycle will be smaller).
>>>> >
>>>> > If we do this for the next cycle we will have a 6 week ‘dev’ period
>>>> and then we
>>>> > will have optimistically an average of 2 weeks of ‘releasing’ and 6
>>>> weeks ‘dev’
>>>> > cycles.
>>>> >
>>>> > On Thu, Mar 1, 2018 at 6:46 AM, Jean-Baptiste Onofré <jb...@nanthrax.net>
>>>> wrote:
>>>> >> About BEAM-3409, I did a review yesterday and it looks good to me.
>>>> We are
>>>> >> waiting for Thomas' feedback.
>>>> >>
>>>> >> Regards
>>>> >> JB
>>>> >> Le 1 mars 2018, à 09:38, Robert Bradshaw <ro...@google.com> a
>>>> écrit:
>>>> >>>
>>>> >>> Looking at the burn-down list, we have 5 remaining issues. None of
>>>> these
>>>> >>> are blockers, but all look like they're really close (reviewed,
>>>> review
>>>> >>> comments were addressed, waiting for a final LGTM). Specifically:
>>>> >>>
>>>> >>> BEAM-3409 (teardown issues): Thomas Groh had some concerns, could
>>>> you
>>>> >>> verify they have all been addressed?
>>>> >>> BEAM-3479 (DoFn classloader regression test): Kenn Knowles had
>>>> minor
>>>> >>> comments, looks like they were addressed, could you confirm?
>>>> >>> BEAM-3735 (Missing gaming release archetypes): Lukasz Cwik had minor
>>>> >>> comments, looks like they were addressed, could you confirm?
>>>> >>> BEAM-3611 (KafkaIO.java splitting): Looks like this was resolved.
>>>> >>> BEAM-3762 (unlimited JCE for Dataflow Worker): LGTM pending
>>>> (currently
>>>> >>> running) tests passing.
>>>> >>>
>>>> >>> Let's see how many of these we can get in by, say, noon PST
>>>> tomorrow.
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> On Wed, Feb 28, 2018 at 9:26 PM Robert Bradshaw <
>>>> robertwb@google.com>
>>>> >>> wrote:
>>>> >>>>
>>>> >>>> I tend to fall into the "release early, release often" camp in
>>>> general,
>>>> >>>> but for this one I'm particularly anxious to get the faster Python
>>>> direct
>>>> >>>> runner out in the hands of TFT/TFX users (and in particular have
>>>> an eye on
>>>> >>>> https://www.tensorflow.org/dev-summit/ which I think can be a
>>>> healthy source
>>>> >>>> of Beam users).
>>>> >>>>
>>>> >>>>
>>>> >>>> On Wed, Feb 28, 2018 at 7:01 PM Jean-Baptiste Onofré <
>>>> jb@nanthrax.net>
>>>> >>>> wrote:
>>>> >>>>>
>>>> >>>>> Hi Gus,
>>>> >>>>>
>>>> >>>>> Thanks for the update, it makes sense.
>>>> >>>>>
>>>> >>>>> Regards
>>>> >>>>> JB
>>>> >>>>>
>>>> >>>>> On 03/01/2018 02:59 AM, Konstantinos Katsiapis wrote:
>>>> >>>>>> Hi Jean-Baptiste,
>>>> >>>>>>
>>>> >>>>>> I can speak from the perspective of tf.transform
>>>> >>>>>> < https://github.com/tensorflow/transform> (TFT) in particular
>>>> and TFX
>>>> >>>>>> < https://research.google.com/pubs/pub46484.html> libs in
>>>> general, in
>>>> >>>>>> case it is
>>>> >>>>>> useful.
>>>> >>>>>>
>>>> >>>>>> TFX distributed computation has 2 "large" dependencies, namely
>>>> >>>>>> TensorFlow and
>>>> >>>>>> Apache Beam, each on their own release schedule.
>>>> >>>>>> As such, releasing of new TFX functionality often (but not
>>>> always)
>>>> >>>>>> depends on
>>>> >>>>>> (and is blocked by) releases of *both* TensorFlow *and* Apache
>>>> Beam.
>>>> >>>>>>
>>>> >>>>>> Synchronizing releases across such large projects and
>>>> organizations is
>>>> >>>>>> likely
>>>> >>>>>> hard, so from our perspective having *frequent* releases of
>>>> Tensorflow
>>>> >>>>>> or Apache
>>>> >>>>>> Beam (and better yet both) decreases the time for which we are
>>>> blocked
>>>> >>>>>> on
>>>> >>>>>> releasing our features.
>>>> >>>>>>
>>>> >>>>>> In light of this, I would vote for more frequent releases in
>>>> general,
>>>> >>>>>> and for a
>>>> >>>>>> Beam 2.4 release soon in particular (as TFT 0.6 depends on it).
>>>> >>>>>>
>>>> >>>>>> Thanks,
>>>> >>>>>> Gus
>>>> >>>>>>
>>>> >>>>>> On Tue, Feb 27, 2018 at 10:29 PM, Jean-Baptiste Onofré <
>>>> >>>>>> jb@nanthrax.net
>>>> >>>>>> <mailto: jb@nanthrax.net>> wrote:
>>>> >>>>>>
>>>> >>>>>> By the way, if third party projects based on Beam (Google
>>>> >>>>>> Dataflow, Talend
>>>> >>>>>> DataStream, and others) need a release (including some
>>>> features),
>>>> >>>>>> it's better to
>>>> >>>>>> clearly state this on the mailing list.
>>>> >>>>>>
>>>> >>>>>> At Apache Karaf, I have lot of projects based on it
>>>> (OpenDaylight,
>>>> >>>>>> OpenHAB,
>>>> >>>>>> Websphere, ...). They just ask for the release schedule and
>>>> they
>>>> >>>>>> align with
>>>> >>>>>> these release. As a best effort, I'm always trying to move
>>>> fast
>>>> >>>>>> when a release
>>>> >>>>>> is asked.
>>>> >>>>>>
>>>> >>>>>> So, if 2.4.0 is required by third party, no problem to "ask
>>>> for a
>>>> >>>>>> release".
>>>> >>>>>>
>>>> >>>>>> Regards
>>>> >>>>>> JB
>>>> >>>>>>
>>>> >>>>>> On 02/28/2018 04:17 AM, Reuven Lax wrote:
>>>> >>>>>> > It's been six weeks since you proposed beam 2.3.0. so
>>>> assuming
>>>> >>>>>> the same time
>>>> >>>>>> > scale for this release, that's 1.5 months between releases.
>>>> >>>>>> Slightly faster than
>>>> >>>>>> > 2 months, but not by much.
>>>> >>>>>> >
>>>> >>>>>> > I do seem to remember that the original goal for beam was
>>>> >>>>>> monthly releases though.
>>>> >>>>>> >
>>>> >>>>>> > Reuven
>>>> >>>>>> >
>>>> >>>>>> > On Tue, Feb 27, 2018, 9:12 PM Jean-Baptiste Onofré <
>>>> >>>>>> jb@nanthrax.net <mailto: jb@nanthrax.net>
>>>> >>>>>> > <mailto: jb@nanthrax.net <mailto: jb@nanthrax.net>>>
>>>> wrote:
>>>> >>>>>> >
>>>> >>>>>> > Hi Reuven,
>>>> >>>>>> >
>>>> >>>>>> > In a previous thread (about Beam project execution), I
>>>> >>>>>> proposed a release every
>>>> >>>>>> > two months (as a best effort), I will find the e-mail.
>>>> >>>>>> >
>>>> >>>>>> > Beam 2.3.0 has been released "officially" on February
>>>> 16th,
>>>> >>>>>> so two week ago
>>>> >>>>>> > roughly. I would have expected 2.4.0 not before end of
>>>> >>>>>> March.
>>>> >>>>>> >
>>>> >>>>>> > If we have issue we want to fix fast, then 2.3.1 is
>>>> good. If
>>>> >>>>>> it's a new release
>>>> >>>>>> > in the pace, it's pretty fast and might "confuse" our
>>>> users.
>>>> >>>>>> >
>>>> >>>>>> > That's why I'm curious ;)
>>>> >>>>>> >
>>>> >>>>>> > Regards
>>>> >>>>>> > JB
>>>> >>>>>> >
>>>> >>>>>> > On 02/28/2018 03:50 AM, Reuven Lax wrote:
>>>> >>>>>> > > Wasn't the original statement monthly releases? We've
>>>> >>>>>> never realistically
>>>> >>>>>> > > managed that, but Robert's proposed cut will be on a
>>>> >>>>>> 6-week pace.
>>>> >>>>>> > >
>>>> >>>>>> > > On Tue, Feb 27, 2018, 8:48 PM Jean-Baptiste Onofré <
>>>> >>>>>> jb@nanthrax.net <mailto: jb@nanthrax.net>
>>>> >>>>>> > <mailto: jb@nanthrax.net <mailto: jb@nanthrax.net>>
>>>> >>>>>> > > <mailto: jb@nanthrax.net <mailto: jb@nanthrax.net>
>>>> >>>>>> <mailto: jb@nanthrax.net
>>>> >>>>>> <mailto: jb@nanthrax.net>>>> wrote:
>>>> >>>>>> > >
>>>> >>>>>> > > Hi Robert,
>>>> >>>>>> > >
>>>> >>>>>> > > I'm just curious: it's pretty fast compared to
>>>> the
>>>> >>>>>> original plan of a
>>>> >>>>>> > release
>>>> >>>>>> > > every two months. What's the reason to cut 2.4.0
>>>> now
>>>> >>>>>> instead of end of
>>>> >>>>>> > March ?
>>>> >>>>>> > >
>>>> >>>>>> > > I will do the Jira triage and update today.
>>>> >>>>>> > >
>>>> >>>>>> > > Regards
>>>> >>>>>> > > JB
>>>> >>>>>> > >
>>>> >>>>>> > > On 02/27/2018 09:21 PM, Robert Bradshaw wrote:
>>>> >>>>>> > > > I'm planning on cutting the 2.4.0 release
>>>> branch
>>>> >>>>>> soon (tomorrow?). I
>>>> >>>>>> > see 13
>>>> >>>>>> > > > open issues on JIRA [1], none of which are
>>>> labeled
>>>> >>>>>> as blockers. If there
>>>> >>>>>> > > > are any that cannot be bumped to the next
>>>> release,
>>>> >>>>>> let me know soon.
>>>> >>>>>> > > >
>>>> >>>>>> > > > - Robert
>>>> >>>>>> > > >
>>>> >>>>>> > > >
>>>> >>>>>> > > > [1]
>>>> >>>>>> > > >
>>>> >>>>>> > >
>>>> >>>>>> >
>>>> >>>>>>
>>>> https://issues.apache.org/jira/browse/BEAM-3749?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%202.4.0
>>>> >>>>>> <
>>>> >>>>>>
>>>> https://issues.apache.org/jira/browse/BEAM-3749?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%202.4.0
>>>> >
>>>> >>>>>> > > >
>>>> >>>>>> > >
>>>> >>>>>> > > --
>>>> >>>>>> > > Jean-Baptiste Onofré
>>>> >>>>>> > > jbonofre@apache.org <mailto:
>>>> jbonofre@apache.org>
>>>> >>>>>> <mailto: jbonofre@apache.org
>>>> >>>>>> <mailto: jbonofre@apache.org>>
>>>> >>>>>> > <mailto: jbonofre@apache.org <mailto:
>>>> jbonofre@apache.org>
>>>> >>>>>> <mailto: jbonofre@apache.org <mailto: jbonofre@apache.org>>>
>>>> >>>>>> > > http://blog.nanthrax.net
>>>> >>>>>> > > Talend - http://www.talend.com
>>>> >>>>>> > >
>>>> >>>>>> >
>>>> >>>>>> > --
>>>> >>>>>> > Jean-Baptiste Onofré
>>>> >>>>>> > jbonofre@apache.org <mailto: jbonofre@apache.org>
>>>> >>>>>> <mailto: jbonofre@apache.org <mailto: jbonofre@apache.org>>
>>>> >>>>>> > http://blog.nanthrax.net
>>>> >>>>>> > Talend - http://www.talend.com
>>>> >>>>>> >
>>>> >>>>>>
>>>> >>>>>> --
>>>> >>>>>> Jean-Baptiste Onofré
>>>> >>>>>> jbonofre@apache.org <mailto: jbonofre@apache.org>
>>>> >>>>>> http://blog.nanthrax.net
>>>> >>>>>> Talend - http://www.talend.com
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>>> --
>>>> >>>>>> Gus Katsiapis | Software Engineer | katsiapis@google.com
>>>> >>>>>> <mailto: katsiapis@google.com> | 650-918-7487
>>>> >>>>>
>>>> >>>>> --
>>>> >>>>> Jean-Baptiste Onofré
>>>> >>>>> jbonofre@apache.org
>>>> >>>>> http://blog.nanthrax.net
>>>> >>>>> Talend - http://www.talend.com
>>>>
>>>> --
>>>> Jean-Baptiste Onofré
>>>> jbonofre@apache.org
>>>> http://blog.nanthrax.net
>>>> Talend - http://www.talend.com
>>>>
>>>
>>
>
Re: Beam 2.4.0
Posted by Romain Manni-Bucau <rm...@gmail.com>.
up, know it missed the 2.4 but can https://github.com/apache/beam/pull/4790
have some love? it really makes beam pretty unusable with direct runner,
I start to have "// workaround for BEAM-3409" everywhere in my codebase
which is quite bothering after 3 months :(
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<https://rmannibucau.metawerx.net/> | Old Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
<https://www.packtpub.com/application-development/java-ee-8-high-performance>
2018-03-06 14:44 GMT+01:00 Romain Manni-Bucau <rm...@gmail.com>:
> Hi guys,
>
> tried to reapply the waitUntilFinish fix - without breaking the
> compilation this time ;) - in https://github.com/apache/beam/pull/4790,
> anyone able to help to review and move that forward?
>
>
> Romain Manni-Bucau
> @rmannibucau <https://twitter.com/rmannibucau> | Blog
> <https://rmannibucau.metawerx.net/> | Old Blog
> <http://rmannibucau.wordpress.com> | Github
> <https://github.com/rmannibucau> | LinkedIn
> <https://www.linkedin.com/in/rmannibucau> | Book
> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>
> 2018-03-02 9:28 GMT+01:00 Robert Bradshaw <ro...@google.com>:
>
>> On Fri, Mar 2, 2018 at 12:01 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
>> wrote:
>>
>>> Hi Ismaël,
>>>
>>> that's a good idea to show history.
>>>
>>> For me, the vote duration doesn't matter as we are in the release
>>> process already.
>>>
>>
>> A more relevant duration to track would probably be cut to final release.
>> This both measures our investment in the release process, as well as how
>> behind head is when we finally get the release out.
>>
>>
>>> The gap between two releases is more significant.
>>
>>
>> +1, this is what users see. (To clarify terminology, the "time between
>> release" is the time between actually releasing x.y and x.y+1 that is most
>> visible to end users, regardless of intermediate process like cutting and
>> voting that we have.) Of course this gets thrown off if our
>> time-to-prepare-a-release becomes a significant fraction of the desired
>> time-between-releases.
>>
>> And clearly with an average of
>>> 80 days (~ 3 months) it's two long. The idea is to reduce this clearly. I
>>> propose two months previously (including the vote period), so meaning 6
>>> weeks
>>> between releases.
>>>
>>
>> It seems there have been proposals for monthly, every 6 weeks
>> (sesquimonthly?), and bimonthly.
>>
>>
>>> Regarding the time we take for the PR review and merge, I think it's a
>>> fair time
>>> to give us time to include new improvements and features, but also to
>>> fix bugs.
>>>
>>
>> Concrete deadlines can provide good motivation to get around to doing
>> reviews, cleaning up PRs, fixing bugs, etc. that you've been meaning to do
>> but for whatever reason keep putting off. So I think it's still good
>> practice to have some lead time that a release is coming for a chance for
>> folks to get stuff in, while still being clear that we're not holding
>> things back for new features and if you don't make the cut another one is
>> close behind.
>>
>> Regards
>>> JB
>>>
>>> On 03/01/2018 06:17 PM, Ismaël Mejía wrote:
>>> > The average time just in the vote process for Beam since we are out of
>>> the
>>> > incubator is 17.5 days with an average of 75 days between versions.
>>> >
>>> > Version Vote Period No. days
>>> > 2.3.0 30/01-16/02 17 days (83 days since last)
>>> > 2.2.0 27/10-25/11 29 days (101 days since last)
>>> > 2.1.0 11/07-16/08 36 days (92 days since last)
>>> > 2.0.0 08/05-16/05 8 days (62 days since last)
>>> > 0.6.0 10/03-15/03 5 days (37 days since last)
>>> > 0.5.0 27/01-06/02 10 days
>>> >
>>> > I think we should have these numbers into account to refine the
>>> distance between
>>> > releases. If we want to follow strict time-based releases, what we can
>>> probably
>>> > refine is how we cut the release so we try to reduce release overlaps
>>> and avoid
>>> > rushing unnecessarily.
>>> >
>>> > Maybe we should follow the proposed 6 weeks for the next release like
>>> this:
>>> >
>>> > - 4 weeks let’s say just after succesful vote and then cut the release
>>> branch.
>>> > - 1 week to burn the blocker list (good to have ones that don’t make
>>> will be
>>> > moved to the next release).
>>> > - 1 week for the vote + RCs (in case the vote takes longer at least
>>> the overlap
>>> > between vote + next dev cycle will be smaller).
>>> >
>>> > If we do this for the next cycle we will have a 6 week ‘dev’ period
>>> and then we
>>> > will have optimistically an average of 2 weeks of ‘releasing’ and 6
>>> weeks ‘dev’
>>> > cycles.
>>> >
>>> > On Thu, Mar 1, 2018 at 6:46 AM, Jean-Baptiste Onofré <jb...@nanthrax.net>
>>> wrote:
>>> >> About BEAM-3409, I did a review yesterday and it looks good to me. We
>>> are
>>> >> waiting for Thomas' feedback.
>>> >>
>>> >> Regards
>>> >> JB
>>> >> Le 1 mars 2018, à 09:38, Robert Bradshaw <ro...@google.com> a
>>> écrit:
>>> >>>
>>> >>> Looking at the burn-down list, we have 5 remaining issues. None of
>>> these
>>> >>> are blockers, but all look like they're really close (reviewed,
>>> review
>>> >>> comments were addressed, waiting for a final LGTM). Specifically:
>>> >>>
>>> >>> BEAM-3409 (teardown issues): Thomas Groh had some concerns, could you
>>> >>> verify they have all been addressed?
>>> >>> BEAM-3479 (DoFn classloader regression test): Kenn Knowles had minor
>>> >>> comments, looks like they were addressed, could you confirm?
>>> >>> BEAM-3735 (Missing gaming release archetypes): Lukasz Cwik had minor
>>> >>> comments, looks like they were addressed, could you confirm?
>>> >>> BEAM-3611 (KafkaIO.java splitting): Looks like this was resolved.
>>> >>> BEAM-3762 (unlimited JCE for Dataflow Worker): LGTM pending
>>> (currently
>>> >>> running) tests passing.
>>> >>>
>>> >>> Let's see how many of these we can get in by, say, noon PST tomorrow.
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> On Wed, Feb 28, 2018 at 9:26 PM Robert Bradshaw <
>>> robertwb@google.com>
>>> >>> wrote:
>>> >>>>
>>> >>>> I tend to fall into the "release early, release often" camp in
>>> general,
>>> >>>> but for this one I'm particularly anxious to get the faster Python
>>> direct
>>> >>>> runner out in the hands of TFT/TFX users (and in particular have an
>>> eye on
>>> >>>> https://www.tensorflow.org/dev-summit/ which I think can be a
>>> healthy source
>>> >>>> of Beam users).
>>> >>>>
>>> >>>>
>>> >>>> On Wed, Feb 28, 2018 at 7:01 PM Jean-Baptiste Onofré <
>>> jb@nanthrax.net>
>>> >>>> wrote:
>>> >>>>>
>>> >>>>> Hi Gus,
>>> >>>>>
>>> >>>>> Thanks for the update, it makes sense.
>>> >>>>>
>>> >>>>> Regards
>>> >>>>> JB
>>> >>>>>
>>> >>>>> On 03/01/2018 02:59 AM, Konstantinos Katsiapis wrote:
>>> >>>>>> Hi Jean-Baptiste,
>>> >>>>>>
>>> >>>>>> I can speak from the perspective of tf.transform
>>> >>>>>> < https://github.com/tensorflow/transform> (TFT) in particular
>>> and TFX
>>> >>>>>> < https://research.google.com/pubs/pub46484.html> libs in
>>> general, in
>>> >>>>>> case it is
>>> >>>>>> useful.
>>> >>>>>>
>>> >>>>>> TFX distributed computation has 2 "large" dependencies, namely
>>> >>>>>> TensorFlow and
>>> >>>>>> Apache Beam, each on their own release schedule.
>>> >>>>>> As such, releasing of new TFX functionality often (but not always)
>>> >>>>>> depends on
>>> >>>>>> (and is blocked by) releases of *both* TensorFlow *and* Apache
>>> Beam.
>>> >>>>>>
>>> >>>>>> Synchronizing releases across such large projects and
>>> organizations is
>>> >>>>>> likely
>>> >>>>>> hard, so from our perspective having *frequent* releases of
>>> Tensorflow
>>> >>>>>> or Apache
>>> >>>>>> Beam (and better yet both) decreases the time for which we are
>>> blocked
>>> >>>>>> on
>>> >>>>>> releasing our features.
>>> >>>>>>
>>> >>>>>> In light of this, I would vote for more frequent releases in
>>> general,
>>> >>>>>> and for a
>>> >>>>>> Beam 2.4 release soon in particular (as TFT 0.6 depends on it).
>>> >>>>>>
>>> >>>>>> Thanks,
>>> >>>>>> Gus
>>> >>>>>>
>>> >>>>>> On Tue, Feb 27, 2018 at 10:29 PM, Jean-Baptiste Onofré <
>>> >>>>>> jb@nanthrax.net
>>> >>>>>> <mailto: jb@nanthrax.net>> wrote:
>>> >>>>>>
>>> >>>>>> By the way, if third party projects based on Beam (Google
>>> >>>>>> Dataflow, Talend
>>> >>>>>> DataStream, and others) need a release (including some
>>> features),
>>> >>>>>> it's better to
>>> >>>>>> clearly state this on the mailing list.
>>> >>>>>>
>>> >>>>>> At Apache Karaf, I have lot of projects based on it
>>> (OpenDaylight,
>>> >>>>>> OpenHAB,
>>> >>>>>> Websphere, ...). They just ask for the release schedule and
>>> they
>>> >>>>>> align with
>>> >>>>>> these release. As a best effort, I'm always trying to move
>>> fast
>>> >>>>>> when a release
>>> >>>>>> is asked.
>>> >>>>>>
>>> >>>>>> So, if 2.4.0 is required by third party, no problem to "ask
>>> for a
>>> >>>>>> release".
>>> >>>>>>
>>> >>>>>> Regards
>>> >>>>>> JB
>>> >>>>>>
>>> >>>>>> On 02/28/2018 04:17 AM, Reuven Lax wrote:
>>> >>>>>> > It's been six weeks since you proposed beam 2.3.0. so
>>> assuming
>>> >>>>>> the same time
>>> >>>>>> > scale for this release, that's 1.5 months between releases.
>>> >>>>>> Slightly faster than
>>> >>>>>> > 2 months, but not by much.
>>> >>>>>> >
>>> >>>>>> > I do seem to remember that the original goal for beam was
>>> >>>>>> monthly releases though.
>>> >>>>>> >
>>> >>>>>> > Reuven
>>> >>>>>> >
>>> >>>>>> > On Tue, Feb 27, 2018, 9:12 PM Jean-Baptiste Onofré <
>>> >>>>>> jb@nanthrax.net <mailto: jb@nanthrax.net>
>>> >>>>>> > <mailto: jb@nanthrax.net <mailto: jb@nanthrax.net>>> wrote:
>>> >>>>>> >
>>> >>>>>> > Hi Reuven,
>>> >>>>>> >
>>> >>>>>> > In a previous thread (about Beam project execution), I
>>> >>>>>> proposed a release every
>>> >>>>>> > two months (as a best effort), I will find the e-mail.
>>> >>>>>> >
>>> >>>>>> > Beam 2.3.0 has been released "officially" on February
>>> 16th,
>>> >>>>>> so two week ago
>>> >>>>>> > roughly. I would have expected 2.4.0 not before end of
>>> >>>>>> March.
>>> >>>>>> >
>>> >>>>>> > If we have issue we want to fix fast, then 2.3.1 is
>>> good. If
>>> >>>>>> it's a new release
>>> >>>>>> > in the pace, it's pretty fast and might "confuse" our
>>> users.
>>> >>>>>> >
>>> >>>>>> > That's why I'm curious ;)
>>> >>>>>> >
>>> >>>>>> > Regards
>>> >>>>>> > JB
>>> >>>>>> >
>>> >>>>>> > On 02/28/2018 03:50 AM, Reuven Lax wrote:
>>> >>>>>> > > Wasn't the original statement monthly releases? We've
>>> >>>>>> never realistically
>>> >>>>>> > > managed that, but Robert's proposed cut will be on a
>>> >>>>>> 6-week pace.
>>> >>>>>> > >
>>> >>>>>> > > On Tue, Feb 27, 2018, 8:48 PM Jean-Baptiste Onofré <
>>> >>>>>> jb@nanthrax.net <mailto: jb@nanthrax.net>
>>> >>>>>> > <mailto: jb@nanthrax.net <mailto: jb@nanthrax.net>>
>>> >>>>>> > > <mailto: jb@nanthrax.net <mailto: jb@nanthrax.net>
>>> >>>>>> <mailto: jb@nanthrax.net
>>> >>>>>> <mailto: jb@nanthrax.net>>>> wrote:
>>> >>>>>> > >
>>> >>>>>> > > Hi Robert,
>>> >>>>>> > >
>>> >>>>>> > > I'm just curious: it's pretty fast compared to the
>>> >>>>>> original plan of a
>>> >>>>>> > release
>>> >>>>>> > > every two months. What's the reason to cut 2.4.0
>>> now
>>> >>>>>> instead of end of
>>> >>>>>> > March ?
>>> >>>>>> > >
>>> >>>>>> > > I will do the Jira triage and update today.
>>> >>>>>> > >
>>> >>>>>> > > Regards
>>> >>>>>> > > JB
>>> >>>>>> > >
>>> >>>>>> > > On 02/27/2018 09:21 PM, Robert Bradshaw wrote:
>>> >>>>>> > > > I'm planning on cutting the 2.4.0 release branch
>>> >>>>>> soon (tomorrow?). I
>>> >>>>>> > see 13
>>> >>>>>> > > > open issues on JIRA [1], none of which are
>>> labeled
>>> >>>>>> as blockers. If there
>>> >>>>>> > > > are any that cannot be bumped to the next
>>> release,
>>> >>>>>> let me know soon.
>>> >>>>>> > > >
>>> >>>>>> > > > - Robert
>>> >>>>>> > > >
>>> >>>>>> > > >
>>> >>>>>> > > > [1]
>>> >>>>>> > > >
>>> >>>>>> > >
>>> >>>>>> >
>>> >>>>>> https://issues.apache.org/jira/browse/BEAM-3749?jql=project%
>>> 20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%
>>> 20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%202.4.0
>>> >>>>>> <
>>> >>>>>> https://issues.apache.org/jira/browse/BEAM-3749?jql=project%
>>> 20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%
>>> 20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%202.4.0>
>>> >>>>>> > > >
>>> >>>>>> > >
>>> >>>>>> > > --
>>> >>>>>> > > Jean-Baptiste Onofré
>>> >>>>>> > > jbonofre@apache.org <mailto: jbonofre@apache.org
>>> >
>>> >>>>>> <mailto: jbonofre@apache.org
>>> >>>>>> <mailto: jbonofre@apache.org>>
>>> >>>>>> > <mailto: jbonofre@apache.org <mailto:
>>> jbonofre@apache.org>
>>> >>>>>> <mailto: jbonofre@apache.org <mailto: jbonofre@apache.org>>>
>>> >>>>>> > > http://blog.nanthrax.net
>>> >>>>>> > > Talend - http://www.talend.com
>>> >>>>>> > >
>>> >>>>>> >
>>> >>>>>> > --
>>> >>>>>> > Jean-Baptiste Onofré
>>> >>>>>> > jbonofre@apache.org <mailto: jbonofre@apache.org>
>>> >>>>>> <mailto: jbonofre@apache.org <mailto: jbonofre@apache.org>>
>>> >>>>>> > http://blog.nanthrax.net
>>> >>>>>> > Talend - http://www.talend.com
>>> >>>>>> >
>>> >>>>>>
>>> >>>>>> --
>>> >>>>>> Jean-Baptiste Onofré
>>> >>>>>> jbonofre@apache.org <mailto: jbonofre@apache.org>
>>> >>>>>> http://blog.nanthrax.net
>>> >>>>>> Talend - http://www.talend.com
>>> >>>>>>
>>> >>>>>>
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> --
>>> >>>>>> Gus Katsiapis | Software Engineer | katsiapis@google.com
>>> >>>>>> <mailto: katsiapis@google.com> | 650-918-7487
>>> >>>>>
>>> >>>>> --
>>> >>>>> Jean-Baptiste Onofré
>>> >>>>> jbonofre@apache.org
>>> >>>>> http://blog.nanthrax.net
>>> >>>>> Talend - http://www.talend.com
>>>
>>> --
>>> Jean-Baptiste Onofré
>>> jbonofre@apache.org
>>> http://blog.nanthrax.net
>>> Talend - http://www.talend.com
>>>
>>
>
Re: Beam 2.4.0
Posted by Romain Manni-Bucau <rm...@gmail.com>.
Hi guys,
tried to reapply the waitUntilFinish fix - without breaking the compilation
this time ;) - in https://github.com/apache/beam/pull/4790, anyone able to
help to review and move that forward?
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<https://rmannibucau.metawerx.net/> | Old Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
<https://www.packtpub.com/application-development/java-ee-8-high-performance>
2018-03-02 9:28 GMT+01:00 Robert Bradshaw <ro...@google.com>:
> On Fri, Mar 2, 2018 at 12:01 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
>> Hi Ismaël,
>>
>> that's a good idea to show history.
>>
>> For me, the vote duration doesn't matter as we are in the release process
>> already.
>>
>
> A more relevant duration to track would probably be cut to final release.
> This both measures our investment in the release process, as well as how
> behind head is when we finally get the release out.
>
>
>> The gap between two releases is more significant.
>
>
> +1, this is what users see. (To clarify terminology, the "time between
> release" is the time between actually releasing x.y and x.y+1 that is most
> visible to end users, regardless of intermediate process like cutting and
> voting that we have.) Of course this gets thrown off if our
> time-to-prepare-a-release becomes a significant fraction of the desired
> time-between-releases.
>
> And clearly with an average of
>> 80 days (~ 3 months) it's two long. The idea is to reduce this clearly. I
>> propose two months previously (including the vote period), so meaning 6
>> weeks
>> between releases.
>>
>
> It seems there have been proposals for monthly, every 6 weeks
> (sesquimonthly?), and bimonthly.
>
>
>> Regarding the time we take for the PR review and merge, I think it's a
>> fair time
>> to give us time to include new improvements and features, but also to fix
>> bugs.
>>
>
> Concrete deadlines can provide good motivation to get around to doing
> reviews, cleaning up PRs, fixing bugs, etc. that you've been meaning to do
> but for whatever reason keep putting off. So I think it's still good
> practice to have some lead time that a release is coming for a chance for
> folks to get stuff in, while still being clear that we're not holding
> things back for new features and if you don't make the cut another one is
> close behind.
>
> Regards
>> JB
>>
>> On 03/01/2018 06:17 PM, Ismaël Mejía wrote:
>> > The average time just in the vote process for Beam since we are out of
>> the
>> > incubator is 17.5 days with an average of 75 days between versions.
>> >
>> > Version Vote Period No. days
>> > 2.3.0 30/01-16/02 17 days (83 days since last)
>> > 2.2.0 27/10-25/11 29 days (101 days since last)
>> > 2.1.0 11/07-16/08 36 days (92 days since last)
>> > 2.0.0 08/05-16/05 8 days (62 days since last)
>> > 0.6.0 10/03-15/03 5 days (37 days since last)
>> > 0.5.0 27/01-06/02 10 days
>> >
>> > I think we should have these numbers into account to refine the
>> distance between
>> > releases. If we want to follow strict time-based releases, what we can
>> probably
>> > refine is how we cut the release so we try to reduce release overlaps
>> and avoid
>> > rushing unnecessarily.
>> >
>> > Maybe we should follow the proposed 6 weeks for the next release like
>> this:
>> >
>> > - 4 weeks let’s say just after succesful vote and then cut the release
>> branch.
>> > - 1 week to burn the blocker list (good to have ones that don’t make
>> will be
>> > moved to the next release).
>> > - 1 week for the vote + RCs (in case the vote takes longer at least the
>> overlap
>> > between vote + next dev cycle will be smaller).
>> >
>> > If we do this for the next cycle we will have a 6 week ‘dev’ period and
>> then we
>> > will have optimistically an average of 2 weeks of ‘releasing’ and 6
>> weeks ‘dev’
>> > cycles.
>> >
>> > On Thu, Mar 1, 2018 at 6:46 AM, Jean-Baptiste Onofré <jb...@nanthrax.net>
>> wrote:
>> >> About BEAM-3409, I did a review yesterday and it looks good to me. We
>> are
>> >> waiting for Thomas' feedback.
>> >>
>> >> Regards
>> >> JB
>> >> Le 1 mars 2018, à 09:38, Robert Bradshaw <ro...@google.com> a
>> écrit:
>> >>>
>> >>> Looking at the burn-down list, we have 5 remaining issues. None of
>> these
>> >>> are blockers, but all look like they're really close (reviewed, review
>> >>> comments were addressed, waiting for a final LGTM). Specifically:
>> >>>
>> >>> BEAM-3409 (teardown issues): Thomas Groh had some concerns, could you
>> >>> verify they have all been addressed?
>> >>> BEAM-3479 (DoFn classloader regression test): Kenn Knowles had minor
>> >>> comments, looks like they were addressed, could you confirm?
>> >>> BEAM-3735 (Missing gaming release archetypes): Lukasz Cwik had minor
>> >>> comments, looks like they were addressed, could you confirm?
>> >>> BEAM-3611 (KafkaIO.java splitting): Looks like this was resolved.
>> >>> BEAM-3762 (unlimited JCE for Dataflow Worker): LGTM pending (currently
>> >>> running) tests passing.
>> >>>
>> >>> Let's see how many of these we can get in by, say, noon PST tomorrow.
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> On Wed, Feb 28, 2018 at 9:26 PM Robert Bradshaw < robertwb@google.com
>> >
>> >>> wrote:
>> >>>>
>> >>>> I tend to fall into the "release early, release often" camp in
>> general,
>> >>>> but for this one I'm particularly anxious to get the faster Python
>> direct
>> >>>> runner out in the hands of TFT/TFX users (and in particular have an
>> eye on
>> >>>> https://www.tensorflow.org/dev-summit/ which I think can be a
>> healthy source
>> >>>> of Beam users).
>> >>>>
>> >>>>
>> >>>> On Wed, Feb 28, 2018 at 7:01 PM Jean-Baptiste Onofré <
>> jb@nanthrax.net>
>> >>>> wrote:
>> >>>>>
>> >>>>> Hi Gus,
>> >>>>>
>> >>>>> Thanks for the update, it makes sense.
>> >>>>>
>> >>>>> Regards
>> >>>>> JB
>> >>>>>
>> >>>>> On 03/01/2018 02:59 AM, Konstantinos Katsiapis wrote:
>> >>>>>> Hi Jean-Baptiste,
>> >>>>>>
>> >>>>>> I can speak from the perspective of tf.transform
>> >>>>>> < https://github.com/tensorflow/transform> (TFT) in particular
>> and TFX
>> >>>>>> < https://research.google.com/pubs/pub46484.html> libs in
>> general, in
>> >>>>>> case it is
>> >>>>>> useful.
>> >>>>>>
>> >>>>>> TFX distributed computation has 2 "large" dependencies, namely
>> >>>>>> TensorFlow and
>> >>>>>> Apache Beam, each on their own release schedule.
>> >>>>>> As such, releasing of new TFX functionality often (but not always)
>> >>>>>> depends on
>> >>>>>> (and is blocked by) releases of *both* TensorFlow *and* Apache
>> Beam.
>> >>>>>>
>> >>>>>> Synchronizing releases across such large projects and
>> organizations is
>> >>>>>> likely
>> >>>>>> hard, so from our perspective having *frequent* releases of
>> Tensorflow
>> >>>>>> or Apache
>> >>>>>> Beam (and better yet both) decreases the time for which we are
>> blocked
>> >>>>>> on
>> >>>>>> releasing our features.
>> >>>>>>
>> >>>>>> In light of this, I would vote for more frequent releases in
>> general,
>> >>>>>> and for a
>> >>>>>> Beam 2.4 release soon in particular (as TFT 0.6 depends on it).
>> >>>>>>
>> >>>>>> Thanks,
>> >>>>>> Gus
>> >>>>>>
>> >>>>>> On Tue, Feb 27, 2018 at 10:29 PM, Jean-Baptiste Onofré <
>> >>>>>> jb@nanthrax.net
>> >>>>>> <mailto: jb@nanthrax.net>> wrote:
>> >>>>>>
>> >>>>>> By the way, if third party projects based on Beam (Google
>> >>>>>> Dataflow, Talend
>> >>>>>> DataStream, and others) need a release (including some
>> features),
>> >>>>>> it's better to
>> >>>>>> clearly state this on the mailing list.
>> >>>>>>
>> >>>>>> At Apache Karaf, I have lot of projects based on it
>> (OpenDaylight,
>> >>>>>> OpenHAB,
>> >>>>>> Websphere, ...). They just ask for the release schedule and
>> they
>> >>>>>> align with
>> >>>>>> these release. As a best effort, I'm always trying to move fast
>> >>>>>> when a release
>> >>>>>> is asked.
>> >>>>>>
>> >>>>>> So, if 2.4.0 is required by third party, no problem to "ask
>> for a
>> >>>>>> release".
>> >>>>>>
>> >>>>>> Regards
>> >>>>>> JB
>> >>>>>>
>> >>>>>> On 02/28/2018 04:17 AM, Reuven Lax wrote:
>> >>>>>> > It's been six weeks since you proposed beam 2.3.0. so
>> assuming
>> >>>>>> the same time
>> >>>>>> > scale for this release, that's 1.5 months between releases.
>> >>>>>> Slightly faster than
>> >>>>>> > 2 months, but not by much.
>> >>>>>> >
>> >>>>>> > I do seem to remember that the original goal for beam was
>> >>>>>> monthly releases though.
>> >>>>>> >
>> >>>>>> > Reuven
>> >>>>>> >
>> >>>>>> > On Tue, Feb 27, 2018, 9:12 PM Jean-Baptiste Onofré <
>> >>>>>> jb@nanthrax.net <mailto: jb@nanthrax.net>
>> >>>>>> > <mailto: jb@nanthrax.net <mailto: jb@nanthrax.net>>> wrote:
>> >>>>>> >
>> >>>>>> > Hi Reuven,
>> >>>>>> >
>> >>>>>> > In a previous thread (about Beam project execution), I
>> >>>>>> proposed a release every
>> >>>>>> > two months (as a best effort), I will find the e-mail.
>> >>>>>> >
>> >>>>>> > Beam 2.3.0 has been released "officially" on February
>> 16th,
>> >>>>>> so two week ago
>> >>>>>> > roughly. I would have expected 2.4.0 not before end of
>> >>>>>> March.
>> >>>>>> >
>> >>>>>> > If we have issue we want to fix fast, then 2.3.1 is
>> good. If
>> >>>>>> it's a new release
>> >>>>>> > in the pace, it's pretty fast and might "confuse" our
>> users.
>> >>>>>> >
>> >>>>>> > That's why I'm curious ;)
>> >>>>>> >
>> >>>>>> > Regards
>> >>>>>> > JB
>> >>>>>> >
>> >>>>>> > On 02/28/2018 03:50 AM, Reuven Lax wrote:
>> >>>>>> > > Wasn't the original statement monthly releases? We've
>> >>>>>> never realistically
>> >>>>>> > > managed that, but Robert's proposed cut will be on a
>> >>>>>> 6-week pace.
>> >>>>>> > >
>> >>>>>> > > On Tue, Feb 27, 2018, 8:48 PM Jean-Baptiste Onofré <
>> >>>>>> jb@nanthrax.net <mailto: jb@nanthrax.net>
>> >>>>>> > <mailto: jb@nanthrax.net <mailto: jb@nanthrax.net>>
>> >>>>>> > > <mailto: jb@nanthrax.net <mailto: jb@nanthrax.net>
>> >>>>>> <mailto: jb@nanthrax.net
>> >>>>>> <mailto: jb@nanthrax.net>>>> wrote:
>> >>>>>> > >
>> >>>>>> > > Hi Robert,
>> >>>>>> > >
>> >>>>>> > > I'm just curious: it's pretty fast compared to the
>> >>>>>> original plan of a
>> >>>>>> > release
>> >>>>>> > > every two months. What's the reason to cut 2.4.0
>> now
>> >>>>>> instead of end of
>> >>>>>> > March ?
>> >>>>>> > >
>> >>>>>> > > I will do the Jira triage and update today.
>> >>>>>> > >
>> >>>>>> > > Regards
>> >>>>>> > > JB
>> >>>>>> > >
>> >>>>>> > > On 02/27/2018 09:21 PM, Robert Bradshaw wrote:
>> >>>>>> > > > I'm planning on cutting the 2.4.0 release branch
>> >>>>>> soon (tomorrow?). I
>> >>>>>> > see 13
>> >>>>>> > > > open issues on JIRA [1], none of which are
>> labeled
>> >>>>>> as blockers. If there
>> >>>>>> > > > are any that cannot be bumped to the next
>> release,
>> >>>>>> let me know soon.
>> >>>>>> > > >
>> >>>>>> > > > - Robert
>> >>>>>> > > >
>> >>>>>> > > >
>> >>>>>> > > > [1]
>> >>>>>> > > >
>> >>>>>> > >
>> >>>>>> >
>> >>>>>> https://issues.apache.org/jira/browse/BEAM-3749?jql=
>> project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%
>> 22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%202.4.0
>> >>>>>> <
>> >>>>>> https://issues.apache.org/jira/browse/BEAM-3749?jql=
>> project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%
>> 22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%202.4.0>
>> >>>>>> > > >
>> >>>>>> > >
>> >>>>>> > > --
>> >>>>>> > > Jean-Baptiste Onofré
>> >>>>>> > > jbonofre@apache.org <mailto: jbonofre@apache.org>
>> >>>>>> <mailto: jbonofre@apache.org
>> >>>>>> <mailto: jbonofre@apache.org>>
>> >>>>>> > <mailto: jbonofre@apache.org <mailto:
>> jbonofre@apache.org>
>> >>>>>> <mailto: jbonofre@apache.org <mailto: jbonofre@apache.org>>>
>> >>>>>> > > http://blog.nanthrax.net
>> >>>>>> > > Talend - http://www.talend.com
>> >>>>>> > >
>> >>>>>> >
>> >>>>>> > --
>> >>>>>> > Jean-Baptiste Onofré
>> >>>>>> > jbonofre@apache.org <mailto: jbonofre@apache.org>
>> >>>>>> <mailto: jbonofre@apache.org <mailto: jbonofre@apache.org>>
>> >>>>>> > http://blog.nanthrax.net
>> >>>>>> > Talend - http://www.talend.com
>> >>>>>> >
>> >>>>>>
>> >>>>>> --
>> >>>>>> Jean-Baptiste Onofré
>> >>>>>> jbonofre@apache.org <mailto: jbonofre@apache.org>
>> >>>>>> http://blog.nanthrax.net
>> >>>>>> Talend - http://www.talend.com
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> --
>> >>>>>> Gus Katsiapis | Software Engineer | katsiapis@google.com
>> >>>>>> <mailto: katsiapis@google.com> | 650-918-7487
>> >>>>>
>> >>>>> --
>> >>>>> Jean-Baptiste Onofré
>> >>>>> jbonofre@apache.org
>> >>>>> http://blog.nanthrax.net
>> >>>>> Talend - http://www.talend.com
>>
>> --
>> Jean-Baptiste Onofré
>> jbonofre@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>>
>
Re: Beam 2.4.0
Posted by Robert Bradshaw <ro...@google.com>.
On Fri, Mar 2, 2018 at 12:01 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> Hi Ismaël,
>
> that's a good idea to show history.
>
> For me, the vote duration doesn't matter as we are in the release process
> already.
>
A more relevant duration to track would probably be cut to final release.
This both measures our investment in the release process, as well as how
behind head is when we finally get the release out.
> The gap between two releases is more significant.
+1, this is what users see. (To clarify terminology, the "time between
release" is the time between actually releasing x.y and x.y+1 that is most
visible to end users, regardless of intermediate process like cutting and
voting that we have.) Of course this gets thrown off if our
time-to-prepare-a-release becomes a significant fraction of the desired
time-between-releases.
And clearly with an average of
> 80 days (~ 3 months) it's two long. The idea is to reduce this clearly. I
> propose two months previously (including the vote period), so meaning 6
> weeks
> between releases.
>
It seems there have been proposals for monthly, every 6 weeks
(sesquimonthly?), and bimonthly.
> Regarding the time we take for the PR review and merge, I think it's a
> fair time
> to give us time to include new improvements and features, but also to fix
> bugs.
>
Concrete deadlines can provide good motivation to get around to doing
reviews, cleaning up PRs, fixing bugs, etc. that you've been meaning to do
but for whatever reason keep putting off. So I think it's still good
practice to have some lead time that a release is coming for a chance for
folks to get stuff in, while still being clear that we're not holding
things back for new features and if you don't make the cut another one is
close behind.
Regards
> JB
>
> On 03/01/2018 06:17 PM, Ismaël Mejía wrote:
> > The average time just in the vote process for Beam since we are out of
> the
> > incubator is 17.5 days with an average of 75 days between versions.
> >
> > Version Vote Period No. days
> > 2.3.0 30/01-16/02 17 days (83 days since last)
> > 2.2.0 27/10-25/11 29 days (101 days since last)
> > 2.1.0 11/07-16/08 36 days (92 days since last)
> > 2.0.0 08/05-16/05 8 days (62 days since last)
> > 0.6.0 10/03-15/03 5 days (37 days since last)
> > 0.5.0 27/01-06/02 10 days
> >
> > I think we should have these numbers into account to refine the distance
> between
> > releases. If we want to follow strict time-based releases, what we can
> probably
> > refine is how we cut the release so we try to reduce release overlaps
> and avoid
> > rushing unnecessarily.
> >
> > Maybe we should follow the proposed 6 weeks for the next release like
> this:
> >
> > - 4 weeks let’s say just after succesful vote and then cut the release
> branch.
> > - 1 week to burn the blocker list (good to have ones that don’t make
> will be
> > moved to the next release).
> > - 1 week for the vote + RCs (in case the vote takes longer at least the
> overlap
> > between vote + next dev cycle will be smaller).
> >
> > If we do this for the next cycle we will have a 6 week ‘dev’ period and
> then we
> > will have optimistically an average of 2 weeks of ‘releasing’ and 6
> weeks ‘dev’
> > cycles.
> >
> > On Thu, Mar 1, 2018 at 6:46 AM, Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
> >> About BEAM-3409, I did a review yesterday and it looks good to me. We
> are
> >> waiting for Thomas' feedback.
> >>
> >> Regards
> >> JB
> >> Le 1 mars 2018, à 09:38, Robert Bradshaw <ro...@google.com> a écrit:
> >>>
> >>> Looking at the burn-down list, we have 5 remaining issues. None of
> these
> >>> are blockers, but all look like they're really close (reviewed, review
> >>> comments were addressed, waiting for a final LGTM). Specifically:
> >>>
> >>> BEAM-3409 (teardown issues): Thomas Groh had some concerns, could you
> >>> verify they have all been addressed?
> >>> BEAM-3479 (DoFn classloader regression test): Kenn Knowles had minor
> >>> comments, looks like they were addressed, could you confirm?
> >>> BEAM-3735 (Missing gaming release archetypes): Lukasz Cwik had minor
> >>> comments, looks like they were addressed, could you confirm?
> >>> BEAM-3611 (KafkaIO.java splitting): Looks like this was resolved.
> >>> BEAM-3762 (unlimited JCE for Dataflow Worker): LGTM pending (currently
> >>> running) tests passing.
> >>>
> >>> Let's see how many of these we can get in by, say, noon PST tomorrow.
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On Wed, Feb 28, 2018 at 9:26 PM Robert Bradshaw < robertwb@google.com>
> >>> wrote:
> >>>>
> >>>> I tend to fall into the "release early, release often" camp in
> general,
> >>>> but for this one I'm particularly anxious to get the faster Python
> direct
> >>>> runner out in the hands of TFT/TFX users (and in particular have an
> eye on
> >>>> https://www.tensorflow.org/dev-summit/ which I think can be a
> healthy source
> >>>> of Beam users).
> >>>>
> >>>>
> >>>> On Wed, Feb 28, 2018 at 7:01 PM Jean-Baptiste Onofré <
> jb@nanthrax.net>
> >>>> wrote:
> >>>>>
> >>>>> Hi Gus,
> >>>>>
> >>>>> Thanks for the update, it makes sense.
> >>>>>
> >>>>> Regards
> >>>>> JB
> >>>>>
> >>>>> On 03/01/2018 02:59 AM, Konstantinos Katsiapis wrote:
> >>>>>> Hi Jean-Baptiste,
> >>>>>>
> >>>>>> I can speak from the perspective of tf.transform
> >>>>>> < https://github.com/tensorflow/transform> (TFT) in particular and
> TFX
> >>>>>> < https://research.google.com/pubs/pub46484.html> libs in general,
> in
> >>>>>> case it is
> >>>>>> useful.
> >>>>>>
> >>>>>> TFX distributed computation has 2 "large" dependencies, namely
> >>>>>> TensorFlow and
> >>>>>> Apache Beam, each on their own release schedule.
> >>>>>> As such, releasing of new TFX functionality often (but not always)
> >>>>>> depends on
> >>>>>> (and is blocked by) releases of *both* TensorFlow *and* Apache Beam.
> >>>>>>
> >>>>>> Synchronizing releases across such large projects and organizations
> is
> >>>>>> likely
> >>>>>> hard, so from our perspective having *frequent* releases of
> Tensorflow
> >>>>>> or Apache
> >>>>>> Beam (and better yet both) decreases the time for which we are
> blocked
> >>>>>> on
> >>>>>> releasing our features.
> >>>>>>
> >>>>>> In light of this, I would vote for more frequent releases in
> general,
> >>>>>> and for a
> >>>>>> Beam 2.4 release soon in particular (as TFT 0.6 depends on it).
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Gus
> >>>>>>
> >>>>>> On Tue, Feb 27, 2018 at 10:29 PM, Jean-Baptiste Onofré <
> >>>>>> jb@nanthrax.net
> >>>>>> <mailto: jb@nanthrax.net>> wrote:
> >>>>>>
> >>>>>> By the way, if third party projects based on Beam (Google
> >>>>>> Dataflow, Talend
> >>>>>> DataStream, and others) need a release (including some
> features),
> >>>>>> it's better to
> >>>>>> clearly state this on the mailing list.
> >>>>>>
> >>>>>> At Apache Karaf, I have lot of projects based on it
> (OpenDaylight,
> >>>>>> OpenHAB,
> >>>>>> Websphere, ...). They just ask for the release schedule and
> they
> >>>>>> align with
> >>>>>> these release. As a best effort, I'm always trying to move fast
> >>>>>> when a release
> >>>>>> is asked.
> >>>>>>
> >>>>>> So, if 2.4.0 is required by third party, no problem to "ask for
> a
> >>>>>> release".
> >>>>>>
> >>>>>> Regards
> >>>>>> JB
> >>>>>>
> >>>>>> On 02/28/2018 04:17 AM, Reuven Lax wrote:
> >>>>>> > It's been six weeks since you proposed beam 2.3.0. so assuming
> >>>>>> the same time
> >>>>>> > scale for this release, that's 1.5 months between releases.
> >>>>>> Slightly faster than
> >>>>>> > 2 months, but not by much.
> >>>>>> >
> >>>>>> > I do seem to remember that the original goal for beam was
> >>>>>> monthly releases though.
> >>>>>> >
> >>>>>> > Reuven
> >>>>>> >
> >>>>>> > On Tue, Feb 27, 2018, 9:12 PM Jean-Baptiste Onofré <
> >>>>>> jb@nanthrax.net <mailto: jb@nanthrax.net>
> >>>>>> > <mailto: jb@nanthrax.net <mailto: jb@nanthrax.net>>> wrote:
> >>>>>> >
> >>>>>> > Hi Reuven,
> >>>>>> >
> >>>>>> > In a previous thread (about Beam project execution), I
> >>>>>> proposed a release every
> >>>>>> > two months (as a best effort), I will find the e-mail.
> >>>>>> >
> >>>>>> > Beam 2.3.0 has been released "officially" on February
> 16th,
> >>>>>> so two week ago
> >>>>>> > roughly. I would have expected 2.4.0 not before end of
> >>>>>> March.
> >>>>>> >
> >>>>>> > If we have issue we want to fix fast, then 2.3.1 is good.
> If
> >>>>>> it's a new release
> >>>>>> > in the pace, it's pretty fast and might "confuse" our
> users.
> >>>>>> >
> >>>>>> > That's why I'm curious ;)
> >>>>>> >
> >>>>>> > Regards
> >>>>>> > JB
> >>>>>> >
> >>>>>> > On 02/28/2018 03:50 AM, Reuven Lax wrote:
> >>>>>> > > Wasn't the original statement monthly releases? We've
> >>>>>> never realistically
> >>>>>> > > managed that, but Robert's proposed cut will be on a
> >>>>>> 6-week pace.
> >>>>>> > >
> >>>>>> > > On Tue, Feb 27, 2018, 8:48 PM Jean-Baptiste Onofré <
> >>>>>> jb@nanthrax.net <mailto: jb@nanthrax.net>
> >>>>>> > <mailto: jb@nanthrax.net <mailto: jb@nanthrax.net>>
> >>>>>> > > <mailto: jb@nanthrax.net <mailto: jb@nanthrax.net>
> >>>>>> <mailto: jb@nanthrax.net
> >>>>>> <mailto: jb@nanthrax.net>>>> wrote:
> >>>>>> > >
> >>>>>> > > Hi Robert,
> >>>>>> > >
> >>>>>> > > I'm just curious: it's pretty fast compared to the
> >>>>>> original plan of a
> >>>>>> > release
> >>>>>> > > every two months. What's the reason to cut 2.4.0 now
> >>>>>> instead of end of
> >>>>>> > March ?
> >>>>>> > >
> >>>>>> > > I will do the Jira triage and update today.
> >>>>>> > >
> >>>>>> > > Regards
> >>>>>> > > JB
> >>>>>> > >
> >>>>>> > > On 02/27/2018 09:21 PM, Robert Bradshaw wrote:
> >>>>>> > > > I'm planning on cutting the 2.4.0 release branch
> >>>>>> soon (tomorrow?). I
> >>>>>> > see 13
> >>>>>> > > > open issues on JIRA [1], none of which are labeled
> >>>>>> as blockers. If there
> >>>>>> > > > are any that cannot be bumped to the next release,
> >>>>>> let me know soon.
> >>>>>> > > >
> >>>>>> > > > - Robert
> >>>>>> > > >
> >>>>>> > > >
> >>>>>> > > > [1]
> >>>>>> > > >
> >>>>>> > >
> >>>>>> >
> >>>>>>
> https://issues.apache.org/jira/browse/BEAM-3749?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%202.4.0
> >>>>>> <
> >>>>>>
> https://issues.apache.org/jira/browse/BEAM-3749?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%202.4.0
> >
> >>>>>> > > >
> >>>>>> > >
> >>>>>> > > --
> >>>>>> > > Jean-Baptiste Onofré
> >>>>>> > > jbonofre@apache.org <mailto: jbonofre@apache.org>
> >>>>>> <mailto: jbonofre@apache.org
> >>>>>> <mailto: jbonofre@apache.org>>
> >>>>>> > <mailto: jbonofre@apache.org <mailto: jbonofre@apache.org
> >
> >>>>>> <mailto: jbonofre@apache.org <mailto: jbonofre@apache.org>>>
> >>>>>> > > http://blog.nanthrax.net
> >>>>>> > > Talend - http://www.talend.com
> >>>>>> > >
> >>>>>> >
> >>>>>> > --
> >>>>>> > Jean-Baptiste Onofré
> >>>>>> > jbonofre@apache.org <mailto: jbonofre@apache.org>
> >>>>>> <mailto: jbonofre@apache.org <mailto: jbonofre@apache.org>>
> >>>>>> > http://blog.nanthrax.net
> >>>>>> > Talend - http://www.talend.com
> >>>>>> >
> >>>>>>
> >>>>>> --
> >>>>>> Jean-Baptiste Onofré
> >>>>>> jbonofre@apache.org <mailto: jbonofre@apache.org>
> >>>>>> http://blog.nanthrax.net
> >>>>>> Talend - http://www.talend.com
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Gus Katsiapis | Software Engineer | katsiapis@google.com
> >>>>>> <mailto: katsiapis@google.com> | 650-918-7487
> >>>>>
> >>>>> --
> >>>>> Jean-Baptiste Onofré
> >>>>> jbonofre@apache.org
> >>>>> http://blog.nanthrax.net
> >>>>> Talend - http://www.talend.com
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
Re: Beam 2.4.0
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi Ismaël,
that's a good idea to show history.
For me, the vote duration doesn't matter as we are in the release process already.
The gap between two releases is more significant. And clearly with an average of
80 days (~ 3 months) it's two long. The idea is to reduce this clearly. I
propose two months previously (including the vote period), so meaning 6 weeks
between releases.
Regarding the time we take for the PR review and merge, I think it's a fair time
to give us time to include new improvements and features, but also to fix bugs.
Regards
JB
On 03/01/2018 06:17 PM, Ismaël Mejía wrote:
> The average time just in the vote process for Beam since we are out of the
> incubator is 17.5 days with an average of 75 days between versions.
>
> Version Vote Period No. days
> 2.3.0 30/01-16/02 17 days (83 days since last)
> 2.2.0 27/10-25/11 29 days (101 days since last)
> 2.1.0 11/07-16/08 36 days (92 days since last)
> 2.0.0 08/05-16/05 8 days (62 days since last)
> 0.6.0 10/03-15/03 5 days (37 days since last)
> 0.5.0 27/01-06/02 10 days
>
> I think we should have these numbers into account to refine the distance between
> releases. If we want to follow strict time-based releases, what we can probably
> refine is how we cut the release so we try to reduce release overlaps and avoid
> rushing unnecessarily.
>
> Maybe we should follow the proposed 6 weeks for the next release like this:
>
> - 4 weeks let’s say just after succesful vote and then cut the release branch.
> - 1 week to burn the blocker list (good to have ones that don’t make will be
> moved to the next release).
> - 1 week for the vote + RCs (in case the vote takes longer at least the overlap
> between vote + next dev cycle will be smaller).
>
> If we do this for the next cycle we will have a 6 week ‘dev’ period and then we
> will have optimistically an average of 2 weeks of ‘releasing’ and 6 weeks ‘dev’
> cycles.
>
> On Thu, Mar 1, 2018 at 6:46 AM, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>> About BEAM-3409, I did a review yesterday and it looks good to me. We are
>> waiting for Thomas' feedback.
>>
>> Regards
>> JB
>> Le 1 mars 2018, à 09:38, Robert Bradshaw <ro...@google.com> a écrit:
>>>
>>> Looking at the burn-down list, we have 5 remaining issues. None of these
>>> are blockers, but all look like they're really close (reviewed, review
>>> comments were addressed, waiting for a final LGTM). Specifically:
>>>
>>> BEAM-3409 (teardown issues): Thomas Groh had some concerns, could you
>>> verify they have all been addressed?
>>> BEAM-3479 (DoFn classloader regression test): Kenn Knowles had minor
>>> comments, looks like they were addressed, could you confirm?
>>> BEAM-3735 (Missing gaming release archetypes): Lukasz Cwik had minor
>>> comments, looks like they were addressed, could you confirm?
>>> BEAM-3611 (KafkaIO.java splitting): Looks like this was resolved.
>>> BEAM-3762 (unlimited JCE for Dataflow Worker): LGTM pending (currently
>>> running) tests passing.
>>>
>>> Let's see how many of these we can get in by, say, noon PST tomorrow.
>>>
>>>
>>>
>>>
>>>
>>> On Wed, Feb 28, 2018 at 9:26 PM Robert Bradshaw < robertwb@google.com>
>>> wrote:
>>>>
>>>> I tend to fall into the "release early, release often" camp in general,
>>>> but for this one I'm particularly anxious to get the faster Python direct
>>>> runner out in the hands of TFT/TFX users (and in particular have an eye on
>>>> https://www.tensorflow.org/dev-summit/ which I think can be a healthy source
>>>> of Beam users).
>>>>
>>>>
>>>> On Wed, Feb 28, 2018 at 7:01 PM Jean-Baptiste Onofré < jb@nanthrax.net>
>>>> wrote:
>>>>>
>>>>> Hi Gus,
>>>>>
>>>>> Thanks for the update, it makes sense.
>>>>>
>>>>> Regards
>>>>> JB
>>>>>
>>>>> On 03/01/2018 02:59 AM, Konstantinos Katsiapis wrote:
>>>>>> Hi Jean-Baptiste,
>>>>>>
>>>>>> I can speak from the perspective of tf.transform
>>>>>> < https://github.com/tensorflow/transform> (TFT) in particular and TFX
>>>>>> < https://research.google.com/pubs/pub46484.html> libs in general, in
>>>>>> case it is
>>>>>> useful.
>>>>>>
>>>>>> TFX distributed computation has 2 "large" dependencies, namely
>>>>>> TensorFlow and
>>>>>> Apache Beam, each on their own release schedule.
>>>>>> As such, releasing of new TFX functionality often (but not always)
>>>>>> depends on
>>>>>> (and is blocked by) releases of *both* TensorFlow *and* Apache Beam.
>>>>>>
>>>>>> Synchronizing releases across such large projects and organizations is
>>>>>> likely
>>>>>> hard, so from our perspective having *frequent* releases of Tensorflow
>>>>>> or Apache
>>>>>> Beam (and better yet both) decreases the time for which we are blocked
>>>>>> on
>>>>>> releasing our features.
>>>>>>
>>>>>> In light of this, I would vote for more frequent releases in general,
>>>>>> and for a
>>>>>> Beam 2.4 release soon in particular (as TFT 0.6 depends on it).
>>>>>>
>>>>>> Thanks,
>>>>>> Gus
>>>>>>
>>>>>> On Tue, Feb 27, 2018 at 10:29 PM, Jean-Baptiste Onofré <
>>>>>> jb@nanthrax.net
>>>>>> <mailto: jb@nanthrax.net>> wrote:
>>>>>>
>>>>>> By the way, if third party projects based on Beam (Google
>>>>>> Dataflow, Talend
>>>>>> DataStream, and others) need a release (including some features),
>>>>>> it's better to
>>>>>> clearly state this on the mailing list.
>>>>>>
>>>>>> At Apache Karaf, I have lot of projects based on it (OpenDaylight,
>>>>>> OpenHAB,
>>>>>> Websphere, ...). They just ask for the release schedule and they
>>>>>> align with
>>>>>> these release. As a best effort, I'm always trying to move fast
>>>>>> when a release
>>>>>> is asked.
>>>>>>
>>>>>> So, if 2.4.0 is required by third party, no problem to "ask for a
>>>>>> release".
>>>>>>
>>>>>> Regards
>>>>>> JB
>>>>>>
>>>>>> On 02/28/2018 04:17 AM, Reuven Lax wrote:
>>>>>> > It's been six weeks since you proposed beam 2.3.0. so assuming
>>>>>> the same time
>>>>>> > scale for this release, that's 1.5 months between releases.
>>>>>> Slightly faster than
>>>>>> > 2 months, but not by much.
>>>>>> >
>>>>>> > I do seem to remember that the original goal for beam was
>>>>>> monthly releases though.
>>>>>> >
>>>>>> > Reuven
>>>>>> >
>>>>>> > On Tue, Feb 27, 2018, 9:12 PM Jean-Baptiste Onofré <
>>>>>> jb@nanthrax.net <mailto: jb@nanthrax.net>
>>>>>> > <mailto: jb@nanthrax.net <mailto: jb@nanthrax.net>>> wrote:
>>>>>> >
>>>>>> > Hi Reuven,
>>>>>> >
>>>>>> > In a previous thread (about Beam project execution), I
>>>>>> proposed a release every
>>>>>> > two months (as a best effort), I will find the e-mail.
>>>>>> >
>>>>>> > Beam 2.3.0 has been released "officially" on February 16th,
>>>>>> so two week ago
>>>>>> > roughly. I would have expected 2.4.0 not before end of
>>>>>> March.
>>>>>> >
>>>>>> > If we have issue we want to fix fast, then 2.3.1 is good. If
>>>>>> it's a new release
>>>>>> > in the pace, it's pretty fast and might "confuse" our users.
>>>>>> >
>>>>>> > That's why I'm curious ;)
>>>>>> >
>>>>>> > Regards
>>>>>> > JB
>>>>>> >
>>>>>> > On 02/28/2018 03:50 AM, Reuven Lax wrote:
>>>>>> > > Wasn't the original statement monthly releases? We've
>>>>>> never realistically
>>>>>> > > managed that, but Robert's proposed cut will be on a
>>>>>> 6-week pace.
>>>>>> > >
>>>>>> > > On Tue, Feb 27, 2018, 8:48 PM Jean-Baptiste Onofré <
>>>>>> jb@nanthrax.net <mailto: jb@nanthrax.net>
>>>>>> > <mailto: jb@nanthrax.net <mailto: jb@nanthrax.net>>
>>>>>> > > <mailto: jb@nanthrax.net <mailto: jb@nanthrax.net>
>>>>>> <mailto: jb@nanthrax.net
>>>>>> <mailto: jb@nanthrax.net>>>> wrote:
>>>>>> > >
>>>>>> > > Hi Robert,
>>>>>> > >
>>>>>> > > I'm just curious: it's pretty fast compared to the
>>>>>> original plan of a
>>>>>> > release
>>>>>> > > every two months. What's the reason to cut 2.4.0 now
>>>>>> instead of end of
>>>>>> > March ?
>>>>>> > >
>>>>>> > > I will do the Jira triage and update today.
>>>>>> > >
>>>>>> > > Regards
>>>>>> > > JB
>>>>>> > >
>>>>>> > > On 02/27/2018 09:21 PM, Robert Bradshaw wrote:
>>>>>> > > > I'm planning on cutting the 2.4.0 release branch
>>>>>> soon (tomorrow?). I
>>>>>> > see 13
>>>>>> > > > open issues on JIRA [1], none of which are labeled
>>>>>> as blockers. If there
>>>>>> > > > are any that cannot be bumped to the next release,
>>>>>> let me know soon.
>>>>>> > > >
>>>>>> > > > - Robert
>>>>>> > > >
>>>>>> > > >
>>>>>> > > > [1]
>>>>>> > > >
>>>>>> > >
>>>>>> >
>>>>>> https://issues.apache.org/jira/browse/BEAM-3749?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%202.4.0
>>>>>> <
>>>>>> https://issues.apache.org/jira/browse/BEAM-3749?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%202.4.0>
>>>>>> > > >
>>>>>> > >
>>>>>> > > --
>>>>>> > > Jean-Baptiste Onofré
>>>>>> > > jbonofre@apache.org <mailto: jbonofre@apache.org>
>>>>>> <mailto: jbonofre@apache.org
>>>>>> <mailto: jbonofre@apache.org>>
>>>>>> > <mailto: jbonofre@apache.org <mailto: jbonofre@apache.org>
>>>>>> <mailto: jbonofre@apache.org <mailto: jbonofre@apache.org>>>
>>>>>> > > http://blog.nanthrax.net
>>>>>> > > Talend - http://www.talend.com
>>>>>> > >
>>>>>> >
>>>>>> > --
>>>>>> > Jean-Baptiste Onofré
>>>>>> > jbonofre@apache.org <mailto: jbonofre@apache.org>
>>>>>> <mailto: jbonofre@apache.org <mailto: jbonofre@apache.org>>
>>>>>> > http://blog.nanthrax.net
>>>>>> > Talend - http://www.talend.com
>>>>>> >
>>>>>>
>>>>>> --
>>>>>> Jean-Baptiste Onofré
>>>>>> jbonofre@apache.org <mailto: jbonofre@apache.org>
>>>>>> http://blog.nanthrax.net
>>>>>> Talend - http://www.talend.com
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Gus Katsiapis | Software Engineer | katsiapis@google.com
>>>>>> <mailto: katsiapis@google.com> | 650-918-7487
>>>>>
>>>>> --
>>>>> Jean-Baptiste Onofré
>>>>> jbonofre@apache.org
>>>>> http://blog.nanthrax.net
>>>>> Talend - http://www.talend.com
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: Beam 2.4.0
Posted by Kenneth Knowles <kl...@google.com>.
The features/bugfixes included in a release are determined by the time
between cutting release branches. So I'd focus on that cadence (outside of
special requests).
Kenn
On Thu, Mar 1, 2018 at 12:33 PM, Robert Bradshaw <ro...@google.com>
wrote:
> On Thu, Mar 1, 2018 at 9:17 AM Ismaël Mejía <ie...@gmail.com> wrote:
>
>> The average time just in the vote process for Beam since we are out of the
>> incubator is 17.5 days with an average of 75 days between versions.
>>
>
> Good thought to look at history. I think there's general consensus that
> this is longer than we would like.
>
>
>> Version Vote Period No. days
>> 2.3.0 30/01-16/02 17 days (83 days since last)
>> 2.2.0 27/10-25/11 29 days (101 days since last)
>> 2.1.0 11/07-16/08 36 days (92 days since last)
>> 2.0.0 08/05-16/05 8 days (62 days since last)
>> 0.6.0 10/03-15/03 5 days (37 days since last)
>> 0.5.0 27/01-06/02 10 days
>>
>> I think we should have these numbers into account to refine the distance
>> between
>> releases. If we want to follow strict time-based releases, what we can
>> probably
>> refine is how we cut the release so we try to reduce release overlaps and
>> avoid
>> rushing unnecessarily.
>>
>> Maybe we should follow the proposed 6 weeks for the next release like
>> this:
>>
>> - 4 weeks let’s say just after succesful vote and then cut the release
>> branch.
>> - 1 week to burn the blocker list (good to have ones that don’t make will
>> be
>> moved to the next release).
>> - 1 week for the vote + RCs (in case the vote takes longer at least the
>> overlap
>> between vote + next dev cycle will be smaller).
>>
>> If we do this for the next cycle we will have a 6 week ‘dev’ period and
>> then we
>> will have optimistically an average of 2 weeks of ‘releasing’ and 6 weeks
>> ‘dev’
>> cycles.
>>
>
> 1 week vote seems optimistic. On the other hand, the reason to have a
> release branch is so that dev work can continue during an ongoing release.
>
>
>> On Thu, Mar 1, 2018 at 6:46 AM, Jean-Baptiste Onofré <jb...@nanthrax.net>
>> wrote:
>> > About BEAM-3409, I did a review yesterday and it looks good to me. We
>> are
>> > waiting for Thomas' feedback.
>> >
>> > Regards
>> > JB
>> > Le 1 mars 2018, à 09:38, Robert Bradshaw <ro...@google.com> a écrit:
>> >>
>> >> Looking at the burn-down list, we have 5 remaining issues. None of
>> these
>> >> are blockers, but all look like they're really close (reviewed, review
>> >> comments were addressed, waiting for a final LGTM). Specifically:
>> >>
>> >> BEAM-3409 (teardown issues): Thomas Groh had some concerns, could you
>> >> verify they have all been addressed?
>> >> BEAM-3479 (DoFn classloader regression test): Kenn Knowles had minor
>> >> comments, looks like they were addressed, could you confirm?
>> >> BEAM-3735 (Missing gaming release archetypes): Lukasz Cwik had minor
>> >> comments, looks like they were addressed, could you confirm?
>> >> BEAM-3611 (KafkaIO.java splitting): Looks like this was resolved.
>> >> BEAM-3762 (unlimited JCE for Dataflow Worker): LGTM pending (currently
>> >> running) tests passing.
>> >>
>> >> Let's see how many of these we can get in by, say, noon PST tomorrow.
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> On Wed, Feb 28, 2018 at 9:26 PM Robert Bradshaw < robertwb@google.com>
>> >> wrote:
>> >>>
>> >>> I tend to fall into the "release early, release often" camp in
>> general,
>> >>> but for this one I'm particularly anxious to get the faster Python
>> direct
>> >>> runner out in the hands of TFT/TFX users (and in particular have an
>> eye on
>> >>> https://www.tensorflow.org/dev-summit/ which I think can be a
>> healthy source
>> >>> of Beam users).
>> >>>
>> >>>
>> >>> On Wed, Feb 28, 2018 at 7:01 PM Jean-Baptiste Onofré <
>> jb@nanthrax.net>
>> >>> wrote:
>> >>>>
>> >>>> Hi Gus,
>> >>>>
>> >>>> Thanks for the update, it makes sense.
>> >>>>
>> >>>> Regards
>> >>>> JB
>> >>>>
>> >>>> On 03/01/2018 02:59 AM, Konstantinos Katsiapis wrote:
>> >>>> > Hi Jean-Baptiste,
>> >>>> >
>> >>>> > I can speak from the perspective of tf.transform
>> >>>> > < https://github.com/tensorflow/transform> (TFT) in particular
>> and TFX
>> >>>> > < https://research.google.com/pubs/pub46484.html> libs in
>> general, in
>> >>>> > case it is
>> >>>> > useful.
>> >>>> >
>> >>>> > TFX distributed computation has 2 "large" dependencies, namely
>> >>>> > TensorFlow and
>> >>>> > Apache Beam, each on their own release schedule.
>> >>>> > As such, releasing of new TFX functionality often (but not always)
>> >>>> > depends on
>> >>>> > (and is blocked by) releases of *both* TensorFlow *and* Apache
>> Beam.
>> >>>> >
>> >>>> > Synchronizing releases across such large projects and
>> organizations is
>> >>>> > likely
>> >>>> > hard, so from our perspective having *frequent* releases of
>> Tensorflow
>> >>>> > or Apache
>> >>>> > Beam (and better yet both) decreases the time for which we are
>> blocked
>> >>>> > on
>> >>>> > releasing our features.
>> >>>> >
>> >>>> > In light of this, I would vote for more frequent releases in
>> general,
>> >>>> > and for a
>> >>>> > Beam 2.4 release soon in particular (as TFT 0.6 depends on it).
>> >>>> >
>> >>>> > Thanks,
>> >>>> > Gus
>> >>>> >
>> >>>> > On Tue, Feb 27, 2018 at 10:29 PM, Jean-Baptiste Onofré <
>> >>>> > jb@nanthrax.net
>> >>>> > <mailto: jb@nanthrax.net>> wrote:
>> >>>> >
>> >>>> > By the way, if third party projects based on Beam (Google
>> >>>> > Dataflow, Talend
>> >>>> > DataStream, and others) need a release (including some
>> features),
>> >>>> > it's better to
>> >>>> > clearly state this on the mailing list.
>> >>>> >
>> >>>> > At Apache Karaf, I have lot of projects based on it
>> (OpenDaylight,
>> >>>> > OpenHAB,
>> >>>> > Websphere, ...). They just ask for the release schedule and
>> they
>> >>>> > align with
>> >>>> > these release. As a best effort, I'm always trying to move fast
>> >>>> > when a release
>> >>>> > is asked.
>> >>>> >
>> >>>> > So, if 2.4.0 is required by third party, no problem to "ask
>> for a
>> >>>> > release".
>> >>>> >
>> >>>> > Regards
>> >>>> > JB
>> >>>> >
>> >>>> > On 02/28/2018 04:17 AM, Reuven Lax wrote:
>> >>>> > > It's been six weeks since you proposed beam 2.3.0. so
>> assuming
>> >>>> > the same time
>> >>>> > > scale for this release, that's 1.5 months between releases.
>> >>>> > Slightly faster than
>> >>>> > > 2 months, but not by much.
>> >>>> > >
>> >>>> > > I do seem to remember that the original goal for beam was
>> >>>> > monthly releases though.
>> >>>> > >
>> >>>> > > Reuven
>> >>>> > >
>> >>>> > > On Tue, Feb 27, 2018, 9:12 PM Jean-Baptiste Onofré <
>> >>>> > jb@nanthrax.net <mailto: jb@nanthrax.net>
>> >>>> > > <mailto: jb@nanthrax.net <mailto: jb@nanthrax.net>>> wrote:
>> >>>> > >
>> >>>> > > Hi Reuven,
>> >>>> > >
>> >>>> > > In a previous thread (about Beam project execution), I
>> >>>> > proposed a release every
>> >>>> > > two months (as a best effort), I will find the e-mail.
>> >>>> > >
>> >>>> > > Beam 2.3.0 has been released "officially" on February
>> 16th,
>> >>>> > so two week ago
>> >>>> > > roughly. I would have expected 2.4.0 not before end of
>> >>>> > March.
>> >>>> > >
>> >>>> > > If we have issue we want to fix fast, then 2.3.1 is
>> good. If
>> >>>> > it's a new release
>> >>>> > > in the pace, it's pretty fast and might "confuse" our
>> users.
>> >>>> > >
>> >>>> > > That's why I'm curious ;)
>> >>>> > >
>> >>>> > > Regards
>> >>>> > > JB
>> >>>> > >
>> >>>> > > On 02/28/2018 03:50 AM, Reuven Lax wrote:
>> >>>> > > > Wasn't the original statement monthly releases? We've
>> >>>> > never realistically
>> >>>> > > > managed that, but Robert's proposed cut will be on a
>> >>>> > 6-week pace.
>> >>>> > > >
>> >>>> > > > On Tue, Feb 27, 2018, 8:48 PM Jean-Baptiste Onofré <
>> >>>> > jb@nanthrax.net <mailto: jb@nanthrax.net>
>> >>>> > > <mailto: jb@nanthrax.net <mailto: jb@nanthrax.net>>
>> >>>> > > > <mailto: jb@nanthrax.net <mailto: jb@nanthrax.net>
>> >>>> > <mailto: jb@nanthrax.net
>> >>>> > <mailto: jb@nanthrax.net>>>> wrote:
>> >>>> > > >
>> >>>> > > > Hi Robert,
>> >>>> > > >
>> >>>> > > > I'm just curious: it's pretty fast compared to the
>> >>>> > original plan of a
>> >>>> > > release
>> >>>> > > > every two months. What's the reason to cut 2.4.0
>> now
>> >>>> > instead of end of
>> >>>> > > March ?
>> >>>> > > >
>> >>>> > > > I will do the Jira triage and update today.
>> >>>> > > >
>> >>>> > > > Regards
>> >>>> > > > JB
>> >>>> > > >
>> >>>> > > > On 02/27/2018 09:21 PM, Robert Bradshaw wrote:
>> >>>> > > > > I'm planning on cutting the 2.4.0 release branch
>> >>>> > soon (tomorrow?). I
>> >>>> > > see 13
>> >>>> > > > > open issues on JIRA [1], none of which are
>> labeled
>> >>>> > as blockers. If there
>> >>>> > > > > are any that cannot be bumped to the next
>> release,
>> >>>> > let me know soon.
>> >>>> > > > >
>> >>>> > > > > - Robert
>> >>>> > > > >
>> >>>> > > > >
>> >>>> > > > > [1]
>> >>>> > > > >
>> >>>> > > >
>> >>>> > >
>> >>>> > https://issues.apache.org/jira/browse/BEAM-3749?jql=
>> project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%
>> 22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%202.4.0
>> >>>> > <
>> >>>> > https://issues.apache.org/jira/browse/BEAM-3749?jql=
>> project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%
>> 22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%202.4.0>
>> >>>> > > > >
>> >>>> > > >
>> >>>> > > > --
>> >>>> > > > Jean-Baptiste Onofré
>> >>>> > > > jbonofre@apache.org <mailto: jbonofre@apache.org>
>> >>>> > <mailto: jbonofre@apache.org
>> >>>> > <mailto: jbonofre@apache.org>>
>> >>>> > > <mailto: jbonofre@apache.org <mailto:
>> jbonofre@apache.org>
>> >>>> > <mailto: jbonofre@apache.org <mailto: jbonofre@apache.org>>>
>> >>>> > > > http://blog.nanthrax.net
>> >>>> > > > Talend - http://www.talend.com
>> >>>> > > >
>> >>>> > >
>> >>>> > > --
>> >>>> > > Jean-Baptiste Onofré
>> >>>> > > jbonofre@apache.org <mailto: jbonofre@apache.org>
>> >>>> > <mailto: jbonofre@apache.org <mailto: jbonofre@apache.org>>
>> >>>> > > http://blog.nanthrax.net
>> >>>> > > Talend - http://www.talend.com
>> >>>> > >
>> >>>> >
>> >>>> > --
>> >>>> > Jean-Baptiste Onofré
>> >>>> > jbonofre@apache.org <mailto: jbonofre@apache.org>
>> >>>> > http://blog.nanthrax.net
>> >>>> > Talend - http://www.talend.com
>> >>>> >
>> >>>> >
>> >>>> >
>> >>>> >
>> >>>> > --
>> >>>> > Gus Katsiapis | Software Engineer | katsiapis@google.com
>> >>>> > <mailto: katsiapis@google.com> | 650-918-7487 <(650)%20918-7487>
>> >>>>
>> >>>> --
>> >>>> Jean-Baptiste Onofré
>> >>>> jbonofre@apache.org
>> >>>> http://blog.nanthrax.net
>> >>>> Talend - http://www.talend.com
>>
>
Re: Beam 2.4.0
Posted by Robert Bradshaw <ro...@google.com>.
On Thu, Mar 1, 2018 at 9:17 AM Ismaël Mejía <ie...@gmail.com> wrote:
> The average time just in the vote process for Beam since we are out of the
> incubator is 17.5 days with an average of 75 days between versions.
>
Good thought to look at history. I think there's general consensus that
this is longer than we would like.
> Version Vote Period No. days
> 2.3.0 30/01-16/02 17 days (83 days since last)
> 2.2.0 27/10-25/11 29 days (101 days since last)
> 2.1.0 11/07-16/08 36 days (92 days since last)
> 2.0.0 08/05-16/05 8 days (62 days since last)
> 0.6.0 10/03-15/03 5 days (37 days since last)
> 0.5.0 27/01-06/02 10 days
>
> I think we should have these numbers into account to refine the distance
> between
> releases. If we want to follow strict time-based releases, what we can
> probably
> refine is how we cut the release so we try to reduce release overlaps and
> avoid
> rushing unnecessarily.
>
> Maybe we should follow the proposed 6 weeks for the next release like this:
>
> - 4 weeks let’s say just after succesful vote and then cut the release
> branch.
> - 1 week to burn the blocker list (good to have ones that don’t make will
> be
> moved to the next release).
> - 1 week for the vote + RCs (in case the vote takes longer at least the
> overlap
> between vote + next dev cycle will be smaller).
>
> If we do this for the next cycle we will have a 6 week ‘dev’ period and
> then we
> will have optimistically an average of 2 weeks of ‘releasing’ and 6 weeks
> ‘dev’
> cycles.
>
1 week vote seems optimistic. On the other hand, the reason to have a
release branch is so that dev work can continue during an ongoing release.
> On Thu, Mar 1, 2018 at 6:46 AM, Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
> > About BEAM-3409, I did a review yesterday and it looks good to me. We are
> > waiting for Thomas' feedback.
> >
> > Regards
> > JB
> > Le 1 mars 2018, à 09:38, Robert Bradshaw <ro...@google.com> a écrit:
> >>
> >> Looking at the burn-down list, we have 5 remaining issues. None of these
> >> are blockers, but all look like they're really close (reviewed, review
> >> comments were addressed, waiting for a final LGTM). Specifically:
> >>
> >> BEAM-3409 (teardown issues): Thomas Groh had some concerns, could you
> >> verify they have all been addressed?
> >> BEAM-3479 (DoFn classloader regression test): Kenn Knowles had minor
> >> comments, looks like they were addressed, could you confirm?
> >> BEAM-3735 (Missing gaming release archetypes): Lukasz Cwik had minor
> >> comments, looks like they were addressed, could you confirm?
> >> BEAM-3611 (KafkaIO.java splitting): Looks like this was resolved.
> >> BEAM-3762 (unlimited JCE for Dataflow Worker): LGTM pending (currently
> >> running) tests passing.
> >>
> >> Let's see how many of these we can get in by, say, noon PST tomorrow.
> >>
> >>
> >>
> >>
> >>
> >> On Wed, Feb 28, 2018 at 9:26 PM Robert Bradshaw < robertwb@google.com>
> >> wrote:
> >>>
> >>> I tend to fall into the "release early, release often" camp in general,
> >>> but for this one I'm particularly anxious to get the faster Python
> direct
> >>> runner out in the hands of TFT/TFX users (and in particular have an
> eye on
> >>> https://www.tensorflow.org/dev-summit/ which I think can be a healthy
> source
> >>> of Beam users).
> >>>
> >>>
> >>> On Wed, Feb 28, 2018 at 7:01 PM Jean-Baptiste Onofré < jb@nanthrax.net
> >
> >>> wrote:
> >>>>
> >>>> Hi Gus,
> >>>>
> >>>> Thanks for the update, it makes sense.
> >>>>
> >>>> Regards
> >>>> JB
> >>>>
> >>>> On 03/01/2018 02:59 AM, Konstantinos Katsiapis wrote:
> >>>> > Hi Jean-Baptiste,
> >>>> >
> >>>> > I can speak from the perspective of tf.transform
> >>>> > < https://github.com/tensorflow/transform> (TFT) in particular and
> TFX
> >>>> > < https://research.google.com/pubs/pub46484.html> libs in general,
> in
> >>>> > case it is
> >>>> > useful.
> >>>> >
> >>>> > TFX distributed computation has 2 "large" dependencies, namely
> >>>> > TensorFlow and
> >>>> > Apache Beam, each on their own release schedule.
> >>>> > As such, releasing of new TFX functionality often (but not always)
> >>>> > depends on
> >>>> > (and is blocked by) releases of *both* TensorFlow *and* Apache Beam.
> >>>> >
> >>>> > Synchronizing releases across such large projects and organizations
> is
> >>>> > likely
> >>>> > hard, so from our perspective having *frequent* releases of
> Tensorflow
> >>>> > or Apache
> >>>> > Beam (and better yet both) decreases the time for which we are
> blocked
> >>>> > on
> >>>> > releasing our features.
> >>>> >
> >>>> > In light of this, I would vote for more frequent releases in
> general,
> >>>> > and for a
> >>>> > Beam 2.4 release soon in particular (as TFT 0.6 depends on it).
> >>>> >
> >>>> > Thanks,
> >>>> > Gus
> >>>> >
> >>>> > On Tue, Feb 27, 2018 at 10:29 PM, Jean-Baptiste Onofré <
> >>>> > jb@nanthrax.net
> >>>> > <mailto: jb@nanthrax.net>> wrote:
> >>>> >
> >>>> > By the way, if third party projects based on Beam (Google
> >>>> > Dataflow, Talend
> >>>> > DataStream, and others) need a release (including some
> features),
> >>>> > it's better to
> >>>> > clearly state this on the mailing list.
> >>>> >
> >>>> > At Apache Karaf, I have lot of projects based on it
> (OpenDaylight,
> >>>> > OpenHAB,
> >>>> > Websphere, ...). They just ask for the release schedule and
> they
> >>>> > align with
> >>>> > these release. As a best effort, I'm always trying to move fast
> >>>> > when a release
> >>>> > is asked.
> >>>> >
> >>>> > So, if 2.4.0 is required by third party, no problem to "ask for
> a
> >>>> > release".
> >>>> >
> >>>> > Regards
> >>>> > JB
> >>>> >
> >>>> > On 02/28/2018 04:17 AM, Reuven Lax wrote:
> >>>> > > It's been six weeks since you proposed beam 2.3.0. so assuming
> >>>> > the same time
> >>>> > > scale for this release, that's 1.5 months between releases.
> >>>> > Slightly faster than
> >>>> > > 2 months, but not by much.
> >>>> > >
> >>>> > > I do seem to remember that the original goal for beam was
> >>>> > monthly releases though.
> >>>> > >
> >>>> > > Reuven
> >>>> > >
> >>>> > > On Tue, Feb 27, 2018, 9:12 PM Jean-Baptiste Onofré <
> >>>> > jb@nanthrax.net <mailto: jb@nanthrax.net>
> >>>> > > <mailto: jb@nanthrax.net <mailto: jb@nanthrax.net>>> wrote:
> >>>> > >
> >>>> > > Hi Reuven,
> >>>> > >
> >>>> > > In a previous thread (about Beam project execution), I
> >>>> > proposed a release every
> >>>> > > two months (as a best effort), I will find the e-mail.
> >>>> > >
> >>>> > > Beam 2.3.0 has been released "officially" on February
> 16th,
> >>>> > so two week ago
> >>>> > > roughly. I would have expected 2.4.0 not before end of
> >>>> > March.
> >>>> > >
> >>>> > > If we have issue we want to fix fast, then 2.3.1 is good.
> If
> >>>> > it's a new release
> >>>> > > in the pace, it's pretty fast and might "confuse" our
> users.
> >>>> > >
> >>>> > > That's why I'm curious ;)
> >>>> > >
> >>>> > > Regards
> >>>> > > JB
> >>>> > >
> >>>> > > On 02/28/2018 03:50 AM, Reuven Lax wrote:
> >>>> > > > Wasn't the original statement monthly releases? We've
> >>>> > never realistically
> >>>> > > > managed that, but Robert's proposed cut will be on a
> >>>> > 6-week pace.
> >>>> > > >
> >>>> > > > On Tue, Feb 27, 2018, 8:48 PM Jean-Baptiste Onofré <
> >>>> > jb@nanthrax.net <mailto: jb@nanthrax.net>
> >>>> > > <mailto: jb@nanthrax.net <mailto: jb@nanthrax.net>>
> >>>> > > > <mailto: jb@nanthrax.net <mailto: jb@nanthrax.net>
> >>>> > <mailto: jb@nanthrax.net
> >>>> > <mailto: jb@nanthrax.net>>>> wrote:
> >>>> > > >
> >>>> > > > Hi Robert,
> >>>> > > >
> >>>> > > > I'm just curious: it's pretty fast compared to the
> >>>> > original plan of a
> >>>> > > release
> >>>> > > > every two months. What's the reason to cut 2.4.0 now
> >>>> > instead of end of
> >>>> > > March ?
> >>>> > > >
> >>>> > > > I will do the Jira triage and update today.
> >>>> > > >
> >>>> > > > Regards
> >>>> > > > JB
> >>>> > > >
> >>>> > > > On 02/27/2018 09:21 PM, Robert Bradshaw wrote:
> >>>> > > > > I'm planning on cutting the 2.4.0 release branch
> >>>> > soon (tomorrow?). I
> >>>> > > see 13
> >>>> > > > > open issues on JIRA [1], none of which are labeled
> >>>> > as blockers. If there
> >>>> > > > > are any that cannot be bumped to the next release,
> >>>> > let me know soon.
> >>>> > > > >
> >>>> > > > > - Robert
> >>>> > > > >
> >>>> > > > >
> >>>> > > > > [1]
> >>>> > > > >
> >>>> > > >
> >>>> > >
> >>>> >
> https://issues.apache.org/jira/browse/BEAM-3749?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%202.4.0
> >>>> > <
> >>>> >
> https://issues.apache.org/jira/browse/BEAM-3749?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%202.4.0
> >
> >>>> > > > >
> >>>> > > >
> >>>> > > > --
> >>>> > > > Jean-Baptiste Onofré
> >>>> > > > jbonofre@apache.org <mailto: jbonofre@apache.org>
> >>>> > <mailto: jbonofre@apache.org
> >>>> > <mailto: jbonofre@apache.org>>
> >>>> > > <mailto: jbonofre@apache.org <mailto: jbonofre@apache.org
> >
> >>>> > <mailto: jbonofre@apache.org <mailto: jbonofre@apache.org>>>
> >>>> > > > http://blog.nanthrax.net
> >>>> > > > Talend - http://www.talend.com
> >>>> > > >
> >>>> > >
> >>>> > > --
> >>>> > > Jean-Baptiste Onofré
> >>>> > > jbonofre@apache.org <mailto: jbonofre@apache.org>
> >>>> > <mailto: jbonofre@apache.org <mailto: jbonofre@apache.org>>
> >>>> > > http://blog.nanthrax.net
> >>>> > > Talend - http://www.talend.com
> >>>> > >
> >>>> >
> >>>> > --
> >>>> > Jean-Baptiste Onofré
> >>>> > jbonofre@apache.org <mailto: jbonofre@apache.org>
> >>>> > http://blog.nanthrax.net
> >>>> > Talend - http://www.talend.com
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> > --
> >>>> > Gus Katsiapis | Software Engineer | katsiapis@google.com
> >>>> > <mailto: katsiapis@google.com> | 650-918-7487
> >>>>
> >>>> --
> >>>> Jean-Baptiste Onofré
> >>>> jbonofre@apache.org
> >>>> http://blog.nanthrax.net
> >>>> Talend - http://www.talend.com
>
Re: Beam 2.4.0
Posted by Ismaël Mejía <ie...@gmail.com>.
The average time just in the vote process for Beam since we are out of the
incubator is 17.5 days with an average of 75 days between versions.
Version Vote Period No. days
2.3.0 30/01-16/02 17 days (83 days since last)
2.2.0 27/10-25/11 29 days (101 days since last)
2.1.0 11/07-16/08 36 days (92 days since last)
2.0.0 08/05-16/05 8 days (62 days since last)
0.6.0 10/03-15/03 5 days (37 days since last)
0.5.0 27/01-06/02 10 days
I think we should have these numbers into account to refine the distance between
releases. If we want to follow strict time-based releases, what we can probably
refine is how we cut the release so we try to reduce release overlaps and avoid
rushing unnecessarily.
Maybe we should follow the proposed 6 weeks for the next release like this:
- 4 weeks let’s say just after succesful vote and then cut the release branch.
- 1 week to burn the blocker list (good to have ones that don’t make will be
moved to the next release).
- 1 week for the vote + RCs (in case the vote takes longer at least the overlap
between vote + next dev cycle will be smaller).
If we do this for the next cycle we will have a 6 week ‘dev’ period and then we
will have optimistically an average of 2 weeks of ‘releasing’ and 6 weeks ‘dev’
cycles.
On Thu, Mar 1, 2018 at 6:46 AM, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
> About BEAM-3409, I did a review yesterday and it looks good to me. We are
> waiting for Thomas' feedback.
>
> Regards
> JB
> Le 1 mars 2018, à 09:38, Robert Bradshaw <ro...@google.com> a écrit:
>>
>> Looking at the burn-down list, we have 5 remaining issues. None of these
>> are blockers, but all look like they're really close (reviewed, review
>> comments were addressed, waiting for a final LGTM). Specifically:
>>
>> BEAM-3409 (teardown issues): Thomas Groh had some concerns, could you
>> verify they have all been addressed?
>> BEAM-3479 (DoFn classloader regression test): Kenn Knowles had minor
>> comments, looks like they were addressed, could you confirm?
>> BEAM-3735 (Missing gaming release archetypes): Lukasz Cwik had minor
>> comments, looks like they were addressed, could you confirm?
>> BEAM-3611 (KafkaIO.java splitting): Looks like this was resolved.
>> BEAM-3762 (unlimited JCE for Dataflow Worker): LGTM pending (currently
>> running) tests passing.
>>
>> Let's see how many of these we can get in by, say, noon PST tomorrow.
>>
>>
>>
>>
>>
>> On Wed, Feb 28, 2018 at 9:26 PM Robert Bradshaw < robertwb@google.com>
>> wrote:
>>>
>>> I tend to fall into the "release early, release often" camp in general,
>>> but for this one I'm particularly anxious to get the faster Python direct
>>> runner out in the hands of TFT/TFX users (and in particular have an eye on
>>> https://www.tensorflow.org/dev-summit/ which I think can be a healthy source
>>> of Beam users).
>>>
>>>
>>> On Wed, Feb 28, 2018 at 7:01 PM Jean-Baptiste Onofré < jb@nanthrax.net>
>>> wrote:
>>>>
>>>> Hi Gus,
>>>>
>>>> Thanks for the update, it makes sense.
>>>>
>>>> Regards
>>>> JB
>>>>
>>>> On 03/01/2018 02:59 AM, Konstantinos Katsiapis wrote:
>>>> > Hi Jean-Baptiste,
>>>> >
>>>> > I can speak from the perspective of tf.transform
>>>> > < https://github.com/tensorflow/transform> (TFT) in particular and TFX
>>>> > < https://research.google.com/pubs/pub46484.html> libs in general, in
>>>> > case it is
>>>> > useful.
>>>> >
>>>> > TFX distributed computation has 2 "large" dependencies, namely
>>>> > TensorFlow and
>>>> > Apache Beam, each on their own release schedule.
>>>> > As such, releasing of new TFX functionality often (but not always)
>>>> > depends on
>>>> > (and is blocked by) releases of *both* TensorFlow *and* Apache Beam.
>>>> >
>>>> > Synchronizing releases across such large projects and organizations is
>>>> > likely
>>>> > hard, so from our perspective having *frequent* releases of Tensorflow
>>>> > or Apache
>>>> > Beam (and better yet both) decreases the time for which we are blocked
>>>> > on
>>>> > releasing our features.
>>>> >
>>>> > In light of this, I would vote for more frequent releases in general,
>>>> > and for a
>>>> > Beam 2.4 release soon in particular (as TFT 0.6 depends on it).
>>>> >
>>>> > Thanks,
>>>> > Gus
>>>> >
>>>> > On Tue, Feb 27, 2018 at 10:29 PM, Jean-Baptiste Onofré <
>>>> > jb@nanthrax.net
>>>> > <mailto: jb@nanthrax.net>> wrote:
>>>> >
>>>> > By the way, if third party projects based on Beam (Google
>>>> > Dataflow, Talend
>>>> > DataStream, and others) need a release (including some features),
>>>> > it's better to
>>>> > clearly state this on the mailing list.
>>>> >
>>>> > At Apache Karaf, I have lot of projects based on it (OpenDaylight,
>>>> > OpenHAB,
>>>> > Websphere, ...). They just ask for the release schedule and they
>>>> > align with
>>>> > these release. As a best effort, I'm always trying to move fast
>>>> > when a release
>>>> > is asked.
>>>> >
>>>> > So, if 2.4.0 is required by third party, no problem to "ask for a
>>>> > release".
>>>> >
>>>> > Regards
>>>> > JB
>>>> >
>>>> > On 02/28/2018 04:17 AM, Reuven Lax wrote:
>>>> > > It's been six weeks since you proposed beam 2.3.0. so assuming
>>>> > the same time
>>>> > > scale for this release, that's 1.5 months between releases.
>>>> > Slightly faster than
>>>> > > 2 months, but not by much.
>>>> > >
>>>> > > I do seem to remember that the original goal for beam was
>>>> > monthly releases though.
>>>> > >
>>>> > > Reuven
>>>> > >
>>>> > > On Tue, Feb 27, 2018, 9:12 PM Jean-Baptiste Onofré <
>>>> > jb@nanthrax.net <mailto: jb@nanthrax.net>
>>>> > > <mailto: jb@nanthrax.net <mailto: jb@nanthrax.net>>> wrote:
>>>> > >
>>>> > > Hi Reuven,
>>>> > >
>>>> > > In a previous thread (about Beam project execution), I
>>>> > proposed a release every
>>>> > > two months (as a best effort), I will find the e-mail.
>>>> > >
>>>> > > Beam 2.3.0 has been released "officially" on February 16th,
>>>> > so two week ago
>>>> > > roughly. I would have expected 2.4.0 not before end of
>>>> > March.
>>>> > >
>>>> > > If we have issue we want to fix fast, then 2.3.1 is good. If
>>>> > it's a new release
>>>> > > in the pace, it's pretty fast and might "confuse" our users.
>>>> > >
>>>> > > That's why I'm curious ;)
>>>> > >
>>>> > > Regards
>>>> > > JB
>>>> > >
>>>> > > On 02/28/2018 03:50 AM, Reuven Lax wrote:
>>>> > > > Wasn't the original statement monthly releases? We've
>>>> > never realistically
>>>> > > > managed that, but Robert's proposed cut will be on a
>>>> > 6-week pace.
>>>> > > >
>>>> > > > On Tue, Feb 27, 2018, 8:48 PM Jean-Baptiste Onofré <
>>>> > jb@nanthrax.net <mailto: jb@nanthrax.net>
>>>> > > <mailto: jb@nanthrax.net <mailto: jb@nanthrax.net>>
>>>> > > > <mailto: jb@nanthrax.net <mailto: jb@nanthrax.net>
>>>> > <mailto: jb@nanthrax.net
>>>> > <mailto: jb@nanthrax.net>>>> wrote:
>>>> > > >
>>>> > > > Hi Robert,
>>>> > > >
>>>> > > > I'm just curious: it's pretty fast compared to the
>>>> > original plan of a
>>>> > > release
>>>> > > > every two months. What's the reason to cut 2.4.0 now
>>>> > instead of end of
>>>> > > March ?
>>>> > > >
>>>> > > > I will do the Jira triage and update today.
>>>> > > >
>>>> > > > Regards
>>>> > > > JB
>>>> > > >
>>>> > > > On 02/27/2018 09:21 PM, Robert Bradshaw wrote:
>>>> > > > > I'm planning on cutting the 2.4.0 release branch
>>>> > soon (tomorrow?). I
>>>> > > see 13
>>>> > > > > open issues on JIRA [1], none of which are labeled
>>>> > as blockers. If there
>>>> > > > > are any that cannot be bumped to the next release,
>>>> > let me know soon.
>>>> > > > >
>>>> > > > > - Robert
>>>> > > > >
>>>> > > > >
>>>> > > > > [1]
>>>> > > > >
>>>> > > >
>>>> > >
>>>> > https://issues.apache.org/jira/browse/BEAM-3749?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%202.4.0
>>>> > <
>>>> > https://issues.apache.org/jira/browse/BEAM-3749?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%202.4.0>
>>>> > > > >
>>>> > > >
>>>> > > > --
>>>> > > > Jean-Baptiste Onofré
>>>> > > > jbonofre@apache.org <mailto: jbonofre@apache.org>
>>>> > <mailto: jbonofre@apache.org
>>>> > <mailto: jbonofre@apache.org>>
>>>> > > <mailto: jbonofre@apache.org <mailto: jbonofre@apache.org>
>>>> > <mailto: jbonofre@apache.org <mailto: jbonofre@apache.org>>>
>>>> > > > http://blog.nanthrax.net
>>>> > > > Talend - http://www.talend.com
>>>> > > >
>>>> > >
>>>> > > --
>>>> > > Jean-Baptiste Onofré
>>>> > > jbonofre@apache.org <mailto: jbonofre@apache.org>
>>>> > <mailto: jbonofre@apache.org <mailto: jbonofre@apache.org>>
>>>> > > http://blog.nanthrax.net
>>>> > > Talend - http://www.talend.com
>>>> > >
>>>> >
>>>> > --
>>>> > Jean-Baptiste Onofré
>>>> > jbonofre@apache.org <mailto: jbonofre@apache.org>
>>>> > http://blog.nanthrax.net
>>>> > Talend - http://www.talend.com
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > Gus Katsiapis | Software Engineer | katsiapis@google.com
>>>> > <mailto: katsiapis@google.com> | 650-918-7487
>>>>
>>>> --
>>>> Jean-Baptiste Onofré
>>>> jbonofre@apache.org
>>>> http://blog.nanthrax.net
>>>> Talend - http://www.talend.com
Re: Beam 2.4.0
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
About BEAM-3409, I did a review yesterday and it looks good to me. We are waiting for Thomas' feedback.
Regards
JB
Le 1 mars 2018 à 09:38, à 09:38, Robert Bradshaw <ro...@google.com> a écrit:
>Looking at the burn-down list, we have 5 remaining issues. None of
>these
>are blockers, but all look like they're really close (reviewed, review
>comments were addressed, waiting for a final LGTM). Specifically:
>
>BEAM-3409 (teardown issues): Thomas Groh had some concerns, could you
>verify they have all been addressed?
>BEAM-3479 (DoFn classloader regression test): Kenn Knowles had minor
>comments, looks like they were addressed, could you confirm?
>BEAM-3735 (Missing gaming release archetypes): Lukasz Cwik had minor
>comments, looks like they were addressed, could you confirm?
>BEAM-3611 (KafkaIO.java splitting): Looks like this was resolved.
>BEAM-3762 (unlimited JCE for Dataflow Worker): LGTM pending (currently
>running) tests passing.
>
>Let's see how many of these we can get in by, say, noon PST tomorrow.
>
>
>
>
>
>On Wed, Feb 28, 2018 at 9:26 PM Robert Bradshaw <ro...@google.com>
>wrote:
>
>> I tend to fall into the "release early, release often" camp in
>general,
>> but for this one I'm particularly anxious to get the faster Python
>direct
>> runner out in the hands of TFT/TFX users (and in particular have an
>eye on
>> https://www.tensorflow.org/dev-summit/ which I think can be a healthy
>> source of Beam users).
>>
>>
>> On Wed, Feb 28, 2018 at 7:01 PM Jean-Baptiste Onofré
><jb...@nanthrax.net>
>> wrote:
>>
>>> Hi Gus,
>>>
>>> Thanks for the update, it makes sense.
>>>
>>> Regards
>>> JB
>>>
>>> On 03/01/2018 02:59 AM, Konstantinos Katsiapis wrote:
>>> > Hi Jean-Baptiste,
>>> >
>>> > I can speak from the perspective of tf.transform
>>> > <https://github.com/tensorflow/transform> (TFT) in particular and
>TFX
>>> > <https://research.google.com/pubs/pub46484.html> libs in general,
>in
>>> case it is
>>> > useful.
>>> >
>>> > TFX distributed computation has 2 "large" dependencies, namely
>>> TensorFlow and
>>> > Apache Beam, each on their own release schedule.
>>> > As such, releasing of new TFX functionality often (but not always)
>>> depends on
>>> > (and is blocked by) releases of *both* TensorFlow *and* Apache
>Beam.
>>> >
>>> > Synchronizing releases across such large projects and
>organizations is
>>> likely
>>> > hard, so from our perspective having *frequent* releases of
>Tensorflow
>>> or Apache
>>> > Beam (and better yet both) decreases the time for which we are
>blocked
>>> on
>>> > releasing our features.
>>> >
>>> > In light of this, I would vote for more frequent releases in
>general,
>>> and for a
>>> > Beam 2.4 release soon in particular (as TFT 0.6 depends on it).
>>> >
>>> > Thanks,
>>> > Gus
>>> >
>>> > On Tue, Feb 27, 2018 at 10:29 PM, Jean-Baptiste Onofré
><jb@nanthrax.net
>>> > <ma...@nanthrax.net>> wrote:
>>> >
>>> > By the way, if third party projects based on Beam (Google
>Dataflow,
>>> Talend
>>> > DataStream, and others) need a release (including some
>features),
>>> it's better to
>>> > clearly state this on the mailing list.
>>> >
>>> > At Apache Karaf, I have lot of projects based on it
>(OpenDaylight,
>>> OpenHAB,
>>> > Websphere, ...). They just ask for the release schedule and
>they
>>> align with
>>> > these release. As a best effort, I'm always trying to move
>fast
>>> when a release
>>> > is asked.
>>> >
>>> > So, if 2.4.0 is required by third party, no problem to "ask
>for a
>>> release".
>>> >
>>> > Regards
>>> > JB
>>> >
>>> > On 02/28/2018 04:17 AM, Reuven Lax wrote:
>>> > > It's been six weeks since you proposed beam 2.3.0. so
>assuming
>>> the same time
>>> > > scale for this release, that's 1.5 months between releases.
>>> Slightly faster than
>>> > > 2 months, but not by much.
>>> > >
>>> > > I do seem to remember that the original goal for beam was
>monthly
>>> releases though.
>>> > >
>>> > > Reuven
>>> > >
>>> > > On Tue, Feb 27, 2018, 9:12 PM Jean-Baptiste Onofré <
>>> jb@nanthrax.net <ma...@nanthrax.net>
>>> > > <mailto:jb@nanthrax.net <ma...@nanthrax.net>>> wrote:
>>> > >
>>> > > Hi Reuven,
>>> > >
>>> > > In a previous thread (about Beam project execution), I
>>> proposed a release every
>>> > > two months (as a best effort), I will find the e-mail.
>>> > >
>>> > > Beam 2.3.0 has been released "officially" on February
>16th,
>>> so two week ago
>>> > > roughly. I would have expected 2.4.0 not before end of
>March.
>>> > >
>>> > > If we have issue we want to fix fast, then 2.3.1 is
>good. If
>>> it's a new release
>>> > > in the pace, it's pretty fast and might "confuse" our
>users.
>>> > >
>>> > > That's why I'm curious ;)
>>> > >
>>> > > Regards
>>> > > JB
>>> > >
>>> > > On 02/28/2018 03:50 AM, Reuven Lax wrote:
>>> > > > Wasn't the original statement monthly releases? We've
>never
>>> realistically
>>> > > > managed that, but Robert's proposed cut will be on a
>6-week
>>> pace.
>>> > > >
>>> > > > On Tue, Feb 27, 2018, 8:48 PM Jean-Baptiste Onofré <
>>> jb@nanthrax.net <ma...@nanthrax.net>
>>> > > <mailto:jb@nanthrax.net <ma...@nanthrax.net>>
>>> > > > <mailto:jb@nanthrax.net <ma...@nanthrax.net>
><mailto:
>>> jb@nanthrax.net
>>> > <ma...@nanthrax.net>>>> wrote:
>>> > > >
>>> > > > Hi Robert,
>>> > > >
>>> > > > I'm just curious: it's pretty fast compared to the
>>> original plan of a
>>> > > release
>>> > > > every two months. What's the reason to cut 2.4.0
>now
>>> instead of end of
>>> > > March ?
>>> > > >
>>> > > > I will do the Jira triage and update today.
>>> > > >
>>> > > > Regards
>>> > > > JB
>>> > > >
>>> > > > On 02/27/2018 09:21 PM, Robert Bradshaw wrote:
>>> > > > > I'm planning on cutting the 2.4.0 release branch
>soon
>>> (tomorrow?). I
>>> > > see 13
>>> > > > > open issues on JIRA [1], none of which are
>labeled as
>>> blockers. If there
>>> > > > > are any that cannot be bumped to the next
>release,
>>> let me know soon.
>>> > > > >
>>> > > > > - Robert
>>> > > > >
>>> > > > >
>>> > > > > [1]
>>> > > > >
>>> > > >
>>> > >
>>>
>https://issues.apache.org/jira/browse/BEAM-3749?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%202.4.0
>>> > <
>>>
>https://issues.apache.org/jira/browse/BEAM-3749?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%202.4.0
>>> >
>>> > > > >
>>> > > >
>>> > > > --
>>> > > > Jean-Baptiste Onofré
>>> > > > jbonofre@apache.org <ma...@apache.org>
>>> <mailto:jbonofre@apache.org
>>> > <ma...@apache.org>>
>>> > > <mailto:jbonofre@apache.org <ma...@apache.org>
>>> > <mailto:jbonofre@apache.org <ma...@apache.org>>>
>>> > > > http://blog.nanthrax.net
>>> > > > Talend - http://www.talend.com
>>> > > >
>>> > >
>>> > > --
>>> > > Jean-Baptiste Onofré
>>> > > jbonofre@apache.org <ma...@apache.org>
>>> > <mailto:jbonofre@apache.org <ma...@apache.org>>
>>> > > http://blog.nanthrax.net
>>> > > Talend - http://www.talend.com
>>> > >
>>> >
>>> > --
>>> > Jean-Baptiste Onofré
>>> > jbonofre@apache.org <ma...@apache.org>
>>> > http://blog.nanthrax.net
>>> > Talend - http://www.talend.com
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> > Gus Katsiapis | Software Engineer | katsiapis@google.com
>>> > <ma...@google.com> | 650-918-7487
>>>
>>> --
>>> Jean-Baptiste Onofré
>>> jbonofre@apache.org
>>> http://blog.nanthrax.net
>>> Talend - http://www.talend.com
>>>
>>
Re: Beam 2.4.0
Posted by Robert Bradshaw <ro...@google.com>.
Looking at the burn-down list, we have 5 remaining issues. None of these
are blockers, but all look like they're really close (reviewed, review
comments were addressed, waiting for a final LGTM). Specifically:
BEAM-3409 (teardown issues): Thomas Groh had some concerns, could you
verify they have all been addressed?
BEAM-3479 (DoFn classloader regression test): Kenn Knowles had minor
comments, looks like they were addressed, could you confirm?
BEAM-3735 (Missing gaming release archetypes): Lukasz Cwik had minor
comments, looks like they were addressed, could you confirm?
BEAM-3611 (KafkaIO.java splitting): Looks like this was resolved.
BEAM-3762 (unlimited JCE for Dataflow Worker): LGTM pending (currently
running) tests passing.
Let's see how many of these we can get in by, say, noon PST tomorrow.
On Wed, Feb 28, 2018 at 9:26 PM Robert Bradshaw <ro...@google.com> wrote:
> I tend to fall into the "release early, release often" camp in general,
> but for this one I'm particularly anxious to get the faster Python direct
> runner out in the hands of TFT/TFX users (and in particular have an eye on
> https://www.tensorflow.org/dev-summit/ which I think can be a healthy
> source of Beam users).
>
>
> On Wed, Feb 28, 2018 at 7:01 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
>> Hi Gus,
>>
>> Thanks for the update, it makes sense.
>>
>> Regards
>> JB
>>
>> On 03/01/2018 02:59 AM, Konstantinos Katsiapis wrote:
>> > Hi Jean-Baptiste,
>> >
>> > I can speak from the perspective of tf.transform
>> > <https://github.com/tensorflow/transform> (TFT) in particular and TFX
>> > <https://research.google.com/pubs/pub46484.html> libs in general, in
>> case it is
>> > useful.
>> >
>> > TFX distributed computation has 2 "large" dependencies, namely
>> TensorFlow and
>> > Apache Beam, each on their own release schedule.
>> > As such, releasing of new TFX functionality often (but not always)
>> depends on
>> > (and is blocked by) releases of *both* TensorFlow *and* Apache Beam.
>> >
>> > Synchronizing releases across such large projects and organizations is
>> likely
>> > hard, so from our perspective having *frequent* releases of Tensorflow
>> or Apache
>> > Beam (and better yet both) decreases the time for which we are blocked
>> on
>> > releasing our features.
>> >
>> > In light of this, I would vote for more frequent releases in general,
>> and for a
>> > Beam 2.4 release soon in particular (as TFT 0.6 depends on it).
>> >
>> > Thanks,
>> > Gus
>> >
>> > On Tue, Feb 27, 2018 at 10:29 PM, Jean-Baptiste Onofré <jb@nanthrax.net
>> > <ma...@nanthrax.net>> wrote:
>> >
>> > By the way, if third party projects based on Beam (Google Dataflow,
>> Talend
>> > DataStream, and others) need a release (including some features),
>> it's better to
>> > clearly state this on the mailing list.
>> >
>> > At Apache Karaf, I have lot of projects based on it (OpenDaylight,
>> OpenHAB,
>> > Websphere, ...). They just ask for the release schedule and they
>> align with
>> > these release. As a best effort, I'm always trying to move fast
>> when a release
>> > is asked.
>> >
>> > So, if 2.4.0 is required by third party, no problem to "ask for a
>> release".
>> >
>> > Regards
>> > JB
>> >
>> > On 02/28/2018 04:17 AM, Reuven Lax wrote:
>> > > It's been six weeks since you proposed beam 2.3.0. so assuming
>> the same time
>> > > scale for this release, that's 1.5 months between releases.
>> Slightly faster than
>> > > 2 months, but not by much.
>> > >
>> > > I do seem to remember that the original goal for beam was monthly
>> releases though.
>> > >
>> > > Reuven
>> > >
>> > > On Tue, Feb 27, 2018, 9:12 PM Jean-Baptiste Onofré <
>> jb@nanthrax.net <ma...@nanthrax.net>
>> > > <mailto:jb@nanthrax.net <ma...@nanthrax.net>>> wrote:
>> > >
>> > > Hi Reuven,
>> > >
>> > > In a previous thread (about Beam project execution), I
>> proposed a release every
>> > > two months (as a best effort), I will find the e-mail.
>> > >
>> > > Beam 2.3.0 has been released "officially" on February 16th,
>> so two week ago
>> > > roughly. I would have expected 2.4.0 not before end of March.
>> > >
>> > > If we have issue we want to fix fast, then 2.3.1 is good. If
>> it's a new release
>> > > in the pace, it's pretty fast and might "confuse" our users.
>> > >
>> > > That's why I'm curious ;)
>> > >
>> > > Regards
>> > > JB
>> > >
>> > > On 02/28/2018 03:50 AM, Reuven Lax wrote:
>> > > > Wasn't the original statement monthly releases? We've never
>> realistically
>> > > > managed that, but Robert's proposed cut will be on a 6-week
>> pace.
>> > > >
>> > > > On Tue, Feb 27, 2018, 8:48 PM Jean-Baptiste Onofré <
>> jb@nanthrax.net <ma...@nanthrax.net>
>> > > <mailto:jb@nanthrax.net <ma...@nanthrax.net>>
>> > > > <mailto:jb@nanthrax.net <ma...@nanthrax.net> <mailto:
>> jb@nanthrax.net
>> > <ma...@nanthrax.net>>>> wrote:
>> > > >
>> > > > Hi Robert,
>> > > >
>> > > > I'm just curious: it's pretty fast compared to the
>> original plan of a
>> > > release
>> > > > every two months. What's the reason to cut 2.4.0 now
>> instead of end of
>> > > March ?
>> > > >
>> > > > I will do the Jira triage and update today.
>> > > >
>> > > > Regards
>> > > > JB
>> > > >
>> > > > On 02/27/2018 09:21 PM, Robert Bradshaw wrote:
>> > > > > I'm planning on cutting the 2.4.0 release branch soon
>> (tomorrow?). I
>> > > see 13
>> > > > > open issues on JIRA [1], none of which are labeled as
>> blockers. If there
>> > > > > are any that cannot be bumped to the next release,
>> let me know soon.
>> > > > >
>> > > > > - Robert
>> > > > >
>> > > > >
>> > > > > [1]
>> > > > >
>> > > >
>> > >
>> https://issues.apache.org/jira/browse/BEAM-3749?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%202.4.0
>> > <
>> https://issues.apache.org/jira/browse/BEAM-3749?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%202.4.0
>> >
>> > > > >
>> > > >
>> > > > --
>> > > > Jean-Baptiste Onofré
>> > > > jbonofre@apache.org <ma...@apache.org>
>> <mailto:jbonofre@apache.org
>> > <ma...@apache.org>>
>> > > <mailto:jbonofre@apache.org <ma...@apache.org>
>> > <mailto:jbonofre@apache.org <ma...@apache.org>>>
>> > > > http://blog.nanthrax.net
>> > > > Talend - http://www.talend.com
>> > > >
>> > >
>> > > --
>> > > Jean-Baptiste Onofré
>> > > jbonofre@apache.org <ma...@apache.org>
>> > <mailto:jbonofre@apache.org <ma...@apache.org>>
>> > > http://blog.nanthrax.net
>> > > Talend - http://www.talend.com
>> > >
>> >
>> > --
>> > Jean-Baptiste Onofré
>> > jbonofre@apache.org <ma...@apache.org>
>> > http://blog.nanthrax.net
>> > Talend - http://www.talend.com
>> >
>> >
>> >
>> >
>> > --
>> > Gus Katsiapis | Software Engineer | katsiapis@google.com
>> > <ma...@google.com> | 650-918-7487
>>
>> --
>> Jean-Baptiste Onofré
>> jbonofre@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>>
>
Re: Beam 2.4.0
Posted by Robert Bradshaw <ro...@google.com>.
I tend to fall into the "release early, release often" camp in general, but
for this one I'm particularly anxious to get the faster Python direct
runner out in the hands of TFT/TFX users (and in particular have an eye on
https://www.tensorflow.org/dev-summit/ which I think can be a healthy
source of Beam users).
On Wed, Feb 28, 2018 at 7:01 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> Hi Gus,
>
> Thanks for the update, it makes sense.
>
> Regards
> JB
>
> On 03/01/2018 02:59 AM, Konstantinos Katsiapis wrote:
> > Hi Jean-Baptiste,
> >
> > I can speak from the perspective of tf.transform
> > <https://github.com/tensorflow/transform> (TFT) in particular and TFX
> > <https://research.google.com/pubs/pub46484.html> libs in general, in
> case it is
> > useful.
> >
> > TFX distributed computation has 2 "large" dependencies, namely
> TensorFlow and
> > Apache Beam, each on their own release schedule.
> > As such, releasing of new TFX functionality often (but not always)
> depends on
> > (and is blocked by) releases of *both* TensorFlow *and* Apache Beam.
> >
> > Synchronizing releases across such large projects and organizations is
> likely
> > hard, so from our perspective having *frequent* releases of Tensorflow
> or Apache
> > Beam (and better yet both) decreases the time for which we are blocked on
> > releasing our features.
> >
> > In light of this, I would vote for more frequent releases in general,
> and for a
> > Beam 2.4 release soon in particular (as TFT 0.6 depends on it).
> >
> > Thanks,
> > Gus
> >
> > On Tue, Feb 27, 2018 at 10:29 PM, Jean-Baptiste Onofré <jb@nanthrax.net
> > <ma...@nanthrax.net>> wrote:
> >
> > By the way, if third party projects based on Beam (Google Dataflow,
> Talend
> > DataStream, and others) need a release (including some features),
> it's better to
> > clearly state this on the mailing list.
> >
> > At Apache Karaf, I have lot of projects based on it (OpenDaylight,
> OpenHAB,
> > Websphere, ...). They just ask for the release schedule and they
> align with
> > these release. As a best effort, I'm always trying to move fast when
> a release
> > is asked.
> >
> > So, if 2.4.0 is required by third party, no problem to "ask for a
> release".
> >
> > Regards
> > JB
> >
> > On 02/28/2018 04:17 AM, Reuven Lax wrote:
> > > It's been six weeks since you proposed beam 2.3.0. so assuming the
> same time
> > > scale for this release, that's 1.5 months between releases.
> Slightly faster than
> > > 2 months, but not by much.
> > >
> > > I do seem to remember that the original goal for beam was monthly
> releases though.
> > >
> > > Reuven
> > >
> > > On Tue, Feb 27, 2018, 9:12 PM Jean-Baptiste Onofré <
> jb@nanthrax.net <ma...@nanthrax.net>
> > > <mailto:jb@nanthrax.net <ma...@nanthrax.net>>> wrote:
> > >
> > > Hi Reuven,
> > >
> > > In a previous thread (about Beam project execution), I
> proposed a release every
> > > two months (as a best effort), I will find the e-mail.
> > >
> > > Beam 2.3.0 has been released "officially" on February 16th, so
> two week ago
> > > roughly. I would have expected 2.4.0 not before end of March.
> > >
> > > If we have issue we want to fix fast, then 2.3.1 is good. If
> it's a new release
> > > in the pace, it's pretty fast and might "confuse" our users.
> > >
> > > That's why I'm curious ;)
> > >
> > > Regards
> > > JB
> > >
> > > On 02/28/2018 03:50 AM, Reuven Lax wrote:
> > > > Wasn't the original statement monthly releases? We've never
> realistically
> > > > managed that, but Robert's proposed cut will be on a 6-week
> pace.
> > > >
> > > > On Tue, Feb 27, 2018, 8:48 PM Jean-Baptiste Onofré <
> jb@nanthrax.net <ma...@nanthrax.net>
> > > <mailto:jb@nanthrax.net <ma...@nanthrax.net>>
> > > > <mailto:jb@nanthrax.net <ma...@nanthrax.net> <mailto:
> jb@nanthrax.net
> > <ma...@nanthrax.net>>>> wrote:
> > > >
> > > > Hi Robert,
> > > >
> > > > I'm just curious: it's pretty fast compared to the
> original plan of a
> > > release
> > > > every two months. What's the reason to cut 2.4.0 now
> instead of end of
> > > March ?
> > > >
> > > > I will do the Jira triage and update today.
> > > >
> > > > Regards
> > > > JB
> > > >
> > > > On 02/27/2018 09:21 PM, Robert Bradshaw wrote:
> > > > > I'm planning on cutting the 2.4.0 release branch soon
> (tomorrow?). I
> > > see 13
> > > > > open issues on JIRA [1], none of which are labeled as
> blockers. If there
> > > > > are any that cannot be bumped to the next release, let
> me know soon.
> > > > >
> > > > > - Robert
> > > > >
> > > > >
> > > > > [1]
> > > > >
> > > >
> > >
> https://issues.apache.org/jira/browse/BEAM-3749?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%202.4.0
> > <
> https://issues.apache.org/jira/browse/BEAM-3749?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%202.4.0
> >
> > > > >
> > > >
> > > > --
> > > > Jean-Baptiste Onofré
> > > > jbonofre@apache.org <ma...@apache.org>
> <mailto:jbonofre@apache.org
> > <ma...@apache.org>>
> > > <mailto:jbonofre@apache.org <ma...@apache.org>
> > <mailto:jbonofre@apache.org <ma...@apache.org>>>
> > > > http://blog.nanthrax.net
> > > > Talend - http://www.talend.com
> > > >
> > >
> > > --
> > > Jean-Baptiste Onofré
> > > jbonofre@apache.org <ma...@apache.org>
> > <mailto:jbonofre@apache.org <ma...@apache.org>>
> > > http://blog.nanthrax.net
> > > Talend - http://www.talend.com
> > >
> >
> > --
> > Jean-Baptiste Onofré
> > jbonofre@apache.org <ma...@apache.org>
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
> >
> >
> >
> >
> > --
> > Gus Katsiapis | Software Engineer | katsiapis@google.com
> > <ma...@google.com> | 650-918-7487
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
Re: Beam 2.4.0
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi Gus,
Thanks for the update, it makes sense.
Regards
JB
On 03/01/2018 02:59 AM, Konstantinos Katsiapis wrote:
> Hi Jean-Baptiste,
>
> I can speak from the perspective of tf.transform
> <https://github.com/tensorflow/transform> (TFT) in particular and TFX
> <https://research.google.com/pubs/pub46484.html> libs in general, in case it is
> useful.
>
> TFX distributed computation has 2 "large" dependencies, namely TensorFlow and
> Apache Beam, each on their own release schedule.
> As such, releasing of new TFX functionality often (but not always) depends on
> (and is blocked by) releases of *both* TensorFlow *and* Apache Beam.
>
> Synchronizing releases across such large projects and organizations is likely
> hard, so from our perspective having *frequent* releases of Tensorflow or Apache
> Beam (and better yet both) decreases the time for which we are blocked on
> releasing our features.
>
> In light of this, I would vote for more frequent releases in general, and for a
> Beam 2.4 release soon in particular (as TFT 0.6 depends on it).
>
> Thanks,
> Gus
>
> On Tue, Feb 27, 2018 at 10:29 PM, Jean-Baptiste Onofré <jb@nanthrax.net
> <ma...@nanthrax.net>> wrote:
>
> By the way, if third party projects based on Beam (Google Dataflow, Talend
> DataStream, and others) need a release (including some features), it's better to
> clearly state this on the mailing list.
>
> At Apache Karaf, I have lot of projects based on it (OpenDaylight, OpenHAB,
> Websphere, ...). They just ask for the release schedule and they align with
> these release. As a best effort, I'm always trying to move fast when a release
> is asked.
>
> So, if 2.4.0 is required by third party, no problem to "ask for a release".
>
> Regards
> JB
>
> On 02/28/2018 04:17 AM, Reuven Lax wrote:
> > It's been six weeks since you proposed beam 2.3.0. so assuming the same time
> > scale for this release, that's 1.5 months between releases. Slightly faster than
> > 2 months, but not by much.
> >
> > I do seem to remember that the original goal for beam was monthly releases though.
> >
> > Reuven
> >
> > On Tue, Feb 27, 2018, 9:12 PM Jean-Baptiste Onofré <jb@nanthrax.net <ma...@nanthrax.net>
> > <mailto:jb@nanthrax.net <ma...@nanthrax.net>>> wrote:
> >
> > Hi Reuven,
> >
> > In a previous thread (about Beam project execution), I proposed a release every
> > two months (as a best effort), I will find the e-mail.
> >
> > Beam 2.3.0 has been released "officially" on February 16th, so two week ago
> > roughly. I would have expected 2.4.0 not before end of March.
> >
> > If we have issue we want to fix fast, then 2.3.1 is good. If it's a new release
> > in the pace, it's pretty fast and might "confuse" our users.
> >
> > That's why I'm curious ;)
> >
> > Regards
> > JB
> >
> > On 02/28/2018 03:50 AM, Reuven Lax wrote:
> > > Wasn't the original statement monthly releases? We've never realistically
> > > managed that, but Robert's proposed cut will be on a 6-week pace.
> > >
> > > On Tue, Feb 27, 2018, 8:48 PM Jean-Baptiste Onofré <jb@nanthrax.net <ma...@nanthrax.net>
> > <mailto:jb@nanthrax.net <ma...@nanthrax.net>>
> > > <mailto:jb@nanthrax.net <ma...@nanthrax.net> <mailto:jb@nanthrax.net
> <ma...@nanthrax.net>>>> wrote:
> > >
> > > Hi Robert,
> > >
> > > I'm just curious: it's pretty fast compared to the original plan of a
> > release
> > > every two months. What's the reason to cut 2.4.0 now instead of end of
> > March ?
> > >
> > > I will do the Jira triage and update today.
> > >
> > > Regards
> > > JB
> > >
> > > On 02/27/2018 09:21 PM, Robert Bradshaw wrote:
> > > > I'm planning on cutting the 2.4.0 release branch soon (tomorrow?). I
> > see 13
> > > > open issues on JIRA [1], none of which are labeled as blockers. If there
> > > > are any that cannot be bumped to the next release, let me know soon.
> > > >
> > > > - Robert
> > > >
> > > >
> > > > [1]
> > > >
> > >
> > https://issues.apache.org/jira/browse/BEAM-3749?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%202.4.0
> <https://issues.apache.org/jira/browse/BEAM-3749?jql=project%20%3D%20BEAM%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%202.4.0>
> > > >
> > >
> > > --
> > > Jean-Baptiste Onofré
> > > jbonofre@apache.org <ma...@apache.org> <mailto:jbonofre@apache.org
> <ma...@apache.org>>
> > <mailto:jbonofre@apache.org <ma...@apache.org>
> <mailto:jbonofre@apache.org <ma...@apache.org>>>
> > > http://blog.nanthrax.net
> > > Talend - http://www.talend.com
> > >
> >
> > --
> > Jean-Baptiste Onofré
> > jbonofre@apache.org <ma...@apache.org>
> <mailto:jbonofre@apache.org <ma...@apache.org>>
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
> >
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org <ma...@apache.org>
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
>
>
>
> --
> Gus Katsiapis | Software Engineer | katsiapis@google.com
> <ma...@google.com> | 650-918-7487
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com