You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Reuven Lax <re...@google.com.INVALID> on 2017/08/30 18:27:46 UTC
Beam 2.2.0 release
Now that Beam 2.1.0 has finally completed, I think we should cut Beam 2.2.0
soon. I volunteer to coordinate this release.
Are there any pending pull requests that people think should be merged
before we cut 2.2.0? If so, please let me know soon, as I would like to cut
by Wednesday of next week.
Thanks,
Reuven
Re: Beam 2.2.0 release
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi Kenn,
thanks !
I think we can target 2.2.0 release for October. Thoughts ?
I'm also volunteer to manage this next release.
Regards
JB
On 08/31/2017 05:57 AM, Kenneth Knowles wrote:
> I went ahead and set up https://s.apache.org/beam-2.2.0-burndown
>
> On Wed, Aug 30, 2017 at 2:27 PM, Eugene Kirpichov <
> kirpichov@google.com.invalid> wrote:
>
>> RedisIO in 2.2.0 is very unlikely. There's still a lot of review remaining
>> last time I checked on the PR.
>>
>> On Wed, Aug 30, 2017 at 2:24 PM Vilhelm von Ehrenheim <
>> vonehrenheim@gmail.com> wrote:
>>
>>> Any chance to get the RedisIO in this release?
>>> [BEAM-1017] Add RedisIO #1687
>>>
>>> Its not my PR but ll be happy to assist if there is anything I can do to
>>> help.
>>>
>>> On 30 Aug 2017 22:46, "Daniel Ribeiro" <da...@squareup.com.invalid>
>>> wrote:
>>>
>>>> It would be great to get a bump on pubsub
>>>> <https://github.com/apache/beam/blob/master/pom.xml#L145> dependency.
>> It
>>>> is
>>>> currently very outdated (v1-rev10-1.22.0, which was released over a
>> year
>>>> ago
>>>> <http://repo1.maven.org/maven2/com/google/apis/google-
>>>> api-services-pubsub/v1-rev10-1.22.0/>
>>>> ).
>>>>
>>>>
>>>> On Wed, Aug 30, 2017 at 1:27 PM, Eugene Kirpichov <
>>>> kirpichov@google.com.invalid> wrote:
>>>>
>>>>> Thanks Ismael. I've marked these two issues for fix in 2.2.0.
>>> Definitely
>>>>> agree that at least the first one must be fixed.
>>>>>
>>>>> Here's the current burndown list
>>>>> https://issues.apache.org/jira/projects/BEAM/versions/12341044 - we
>>>> should
>>>>> clean it up.
>>>>>
>>>>> On Wed, Aug 30, 2017 at 1:20 PM Ismaël Mejía <ie...@gmail.com>
>>> wrote:
>>>>>
>>>>>> The current master has accumulated a good amount of nice features
>>>>>> since 2.1.0 so a new release is welcomed. I have two JIRAs/PR that
>> I
>>>>>> think are important to check/solve before the cut:
>>>>>>
>>>>>> BEAM-2516 (this is a regression on the performance of Direct runner
>>> on
>>>>>> Java). We had never really defined if a performance regression is
>>>>>> critical to be a blocker. I executed WordCount with the
>> kinglear.txt
>>>>>> (170KB) file in version 2.1.0 vs the current 2.2.0-SNAPSHOT and I
>>>>>> found that the execution time passed from 5s to 126s. So maybe we
>>> need
>>>>>> to review this one before the release. I can understand if others
>>>>>> consider this a minor issue because the Direct runner is not
>> supposed
>>>>>> to be used for production, but this performance regression can
>> cause
>>> a
>>>>>> bad impression for a casual user starting with Beam.
>>>>>>
>>>>>> BEAM-2790 (fix reading from Amazon S3 via HadoopFileSystem). I
>> think
>>>>>> this one is a nice to have. I am not sure that I can tackle it for
>>> the
>>>>>> wednesday cut. I’m OOO until the beginning of next week, but maybe
>>>>>> someone else can take a look. In the worst case this is not a
>> release
>>>>>> blocker but definitely a really nice fix to include.
>>>>>>
>>>>>>
>>>>>> On Wed, Aug 30, 2017 at 8:49 PM, Eugene Kirpichov
>>>>>> <ki...@google.com.invalid> wrote:
>>>>>>> I'd like to get the following PRs into 2.2.0:
>>>>>>>
>>>>>>> #3765 <https://github.com/apache/beam/pull/3765> [BEAM-2753
>>>>>>> <https://issues.apache.org/jira/browse/BEAM-2753>] Fixes
>>> translation
>>>>> of
>>>>>>> WriteFiles side inputs (important bugfix for DynamicDestinations
>> in
>>>>>> files)
>>>>>>> #3725 <https://github.com/apache/beam/pull/3725> [BEAM-2827
>>>>>>> <https://issues.apache.org/jira/browse/BEAM-2827>] Introduces
>>>>>>> AvroIO.watchForNewFiles (parity for AvroIO with TextIO in a few
>>>>> important
>>>>>>> features)
>>>>>>> #3759 <https://github.com/apache/beam/pull/3759> [BEAM-2828
>>>>>>> <https://issues.apache.org/jira/browse/BEAM-2828>] Moves Match
>>> into
>>>>>>> FileIO.match()/matchAll() (to prevent releasing current
>>>>>>> Match.filepatterns() into 2.2.0 and then having to keep it under
>>> that
>>>>>> name)
>>>>>>>
>>>>>>> On Wed, Aug 30, 2017 at 11:31 AM Mingmin Xu <mi...@gmail.com>
>>>>> wrote:
>>>>>>>
>>>>>>>> Glad to see that 2.2.0 is coming. Can we include SQL feature in
>>> next
>>>>>>>> release? We're in the final stage and expect to merge back to
>>> master
>>>>>> this
>>>>>>>> week.
>>>>>>>>
>>>>>>>> On Wed, Aug 30, 2017 at 11:27 AM, Reuven Lax
>>>> <relax@google.com.invalid
>>>>>>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Now that Beam 2.1.0 has finally completed, I think we should
>> cut
>>>>> Beam
>>>>>>>> 2.2.0
>>>>>>>>> soon. I volunteer to coordinate this release.
>>>>>>>>>
>>>>>>>>> Are there any pending pull requests that people think should
>> be
>>>>> merged
>>>>>>>>> before we cut 2.2.0? If so, please let me know soon, as I
>> would
>>>> like
>>>>>> to
>>>>>>>> cut
>>>>>>>>> by Wednesday of next week.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> Reuven
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> ----
>>>>>>>> Mingmin
>>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: Beam 2.2.0 release
Posted by Kenneth Knowles <kl...@google.com.INVALID>.
I went ahead and set up https://s.apache.org/beam-2.2.0-burndown
On Wed, Aug 30, 2017 at 2:27 PM, Eugene Kirpichov <
kirpichov@google.com.invalid> wrote:
> RedisIO in 2.2.0 is very unlikely. There's still a lot of review remaining
> last time I checked on the PR.
>
> On Wed, Aug 30, 2017 at 2:24 PM Vilhelm von Ehrenheim <
> vonehrenheim@gmail.com> wrote:
>
> > Any chance to get the RedisIO in this release?
> > [BEAM-1017] Add RedisIO #1687
> >
> > Its not my PR but ll be happy to assist if there is anything I can do to
> > help.
> >
> > On 30 Aug 2017 22:46, "Daniel Ribeiro" <da...@squareup.com.invalid>
> > wrote:
> >
> > > It would be great to get a bump on pubsub
> > > <https://github.com/apache/beam/blob/master/pom.xml#L145> dependency.
> It
> > > is
> > > currently very outdated (v1-rev10-1.22.0, which was released over a
> year
> > > ago
> > > <http://repo1.maven.org/maven2/com/google/apis/google-
> > > api-services-pubsub/v1-rev10-1.22.0/>
> > > ).
> > >
> > >
> > > On Wed, Aug 30, 2017 at 1:27 PM, Eugene Kirpichov <
> > > kirpichov@google.com.invalid> wrote:
> > >
> > > > Thanks Ismael. I've marked these two issues for fix in 2.2.0.
> > Definitely
> > > > agree that at least the first one must be fixed.
> > > >
> > > > Here's the current burndown list
> > > > https://issues.apache.org/jira/projects/BEAM/versions/12341044 - we
> > > should
> > > > clean it up.
> > > >
> > > > On Wed, Aug 30, 2017 at 1:20 PM Ismaël Mejía <ie...@gmail.com>
> > wrote:
> > > >
> > > > > The current master has accumulated a good amount of nice features
> > > > > since 2.1.0 so a new release is welcomed. I have two JIRAs/PR that
> I
> > > > > think are important to check/solve before the cut:
> > > > >
> > > > > BEAM-2516 (this is a regression on the performance of Direct runner
> > on
> > > > > Java). We had never really defined if a performance regression is
> > > > > critical to be a blocker. I executed WordCount with the
> kinglear.txt
> > > > > (170KB) file in version 2.1.0 vs the current 2.2.0-SNAPSHOT and I
> > > > > found that the execution time passed from 5s to 126s. So maybe we
> > need
> > > > > to review this one before the release. I can understand if others
> > > > > consider this a minor issue because the Direct runner is not
> supposed
> > > > > to be used for production, but this performance regression can
> cause
> > a
> > > > > bad impression for a casual user starting with Beam.
> > > > >
> > > > > BEAM-2790 (fix reading from Amazon S3 via HadoopFileSystem). I
> think
> > > > > this one is a nice to have. I am not sure that I can tackle it for
> > the
> > > > > wednesday cut. I’m OOO until the beginning of next week, but maybe
> > > > > someone else can take a look. In the worst case this is not a
> release
> > > > > blocker but definitely a really nice fix to include.
> > > > >
> > > > >
> > > > > On Wed, Aug 30, 2017 at 8:49 PM, Eugene Kirpichov
> > > > > <ki...@google.com.invalid> wrote:
> > > > > > I'd like to get the following PRs into 2.2.0:
> > > > > >
> > > > > > #3765 <https://github.com/apache/beam/pull/3765> [BEAM-2753
> > > > > > <https://issues.apache.org/jira/browse/BEAM-2753>] Fixes
> > translation
> > > > of
> > > > > > WriteFiles side inputs (important bugfix for DynamicDestinations
> in
> > > > > files)
> > > > > > #3725 <https://github.com/apache/beam/pull/3725> [BEAM-2827
> > > > > > <https://issues.apache.org/jira/browse/BEAM-2827>] Introduces
> > > > > > AvroIO.watchForNewFiles (parity for AvroIO with TextIO in a few
> > > > important
> > > > > > features)
> > > > > > #3759 <https://github.com/apache/beam/pull/3759> [BEAM-2828
> > > > > > <https://issues.apache.org/jira/browse/BEAM-2828>] Moves Match
> > into
> > > > > > FileIO.match()/matchAll() (to prevent releasing current
> > > > > > Match.filepatterns() into 2.2.0 and then having to keep it under
> > that
> > > > > name)
> > > > > >
> > > > > > On Wed, Aug 30, 2017 at 11:31 AM Mingmin Xu <mi...@gmail.com>
> > > > wrote:
> > > > > >
> > > > > >> Glad to see that 2.2.0 is coming. Can we include SQL feature in
> > next
> > > > > >> release? We're in the final stage and expect to merge back to
> > master
> > > > > this
> > > > > >> week.
> > > > > >>
> > > > > >> On Wed, Aug 30, 2017 at 11:27 AM, Reuven Lax
> > > <relax@google.com.invalid
> > > > >
> > > > > >> wrote:
> > > > > >>
> > > > > >> > Now that Beam 2.1.0 has finally completed, I think we should
> cut
> > > > Beam
> > > > > >> 2.2.0
> > > > > >> > soon. I volunteer to coordinate this release.
> > > > > >> >
> > > > > >> > Are there any pending pull requests that people think should
> be
> > > > merged
> > > > > >> > before we cut 2.2.0? If so, please let me know soon, as I
> would
> > > like
> > > > > to
> > > > > >> cut
> > > > > >> > by Wednesday of next week.
> > > > > >> >
> > > > > >> > Thanks,
> > > > > >> >
> > > > > >> > Reuven
> > > > > >> >
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> --
> > > > > >> ----
> > > > > >> Mingmin
> > > > > >>
> > > > >
> > > >
> > >
> >
>
Re: Beam 2.2.0 release
Posted by Mingmin Xu <mi...@gmail.com>.
Regarding to BEAM-2833, a PR is there for review(
https://github.com/apache/beam/pull/3803). Once it's completed, I'll move
forward for SQL merge.
Mingmin
On Wed, Sep 6, 2017 at 4:00 PM, Reuven Lax <re...@google.com.invalid> wrote:
> It sounds like SQL is still not in, and there are a couple of other PRs
> that people have requested in 2.2.0. I am mostly out next week, so let's
> set September 18 as a target date for cutting the first RC. That should
> hopefully give plenty of time to get SQL and the remaining PRs merged into
> master.
>
> Reuven
>
> On Thu, Aug 31, 2017 at 3:04 PM, Mingmin Xu <mi...@gmail.com> wrote:
>
> > Add https://issues.apache.org/jira/browse/BEAM-2833 which is a blocker
> to
> > merge DSL_SQL. There may be something wrong in the back-end(maybe
> > RunnerApi) to handle parametered CustomCoder in TestPipeline.
> >
> > On Thu, Aug 31, 2017 at 10:38 AM, Jean-Baptiste Onofré <jb...@nanthrax.net>
> > wrote:
> >
> > > Fair enough.
> > >
> > > That's fine for me.
> > >
> > > Regards
> > > JB
> > >
> > > On Aug 31, 2017, 19:03, at 19:03, Steve Niemitz <sn...@apache.org>
> > > wrote:
> > > >I'll chime in as a user who would love to see 2.2.0 sooner than later,
> > > >specifically for the file IO Eugene mentioned. We're using the AvroIO
> > > >enhancements extensively, but I am hesitant to run from HEAD in master
> > > >in
> > > >production.
> > > >
> > > >On Thu, Aug 31, 2017 at 12:42 PM, Eugene Kirpichov <
> > > >kirpichov@google.com.invalid> wrote:
> > > >
> > > >> There are a lot of users including very large production customers
> > > >who have
> > > >> been asking specifically for the features that are in 2.2.0 (most of
> > > >them
> > > >> accumulated while 2.1.0 was being iterated on) - mostly I'm
> referring
> > > >to
> > > >> the vastly improved file IO - and they have been hesitant to use
> Beam
> > > >at
> > > >> HEAD in production. I think the slight unusualness of having a
> > > >release
> > > >> published soon after the previous release is a small price to pay
> for
> > > >> helping those users :)
> > > >>
> > > >> On Wed, Aug 30, 2017, 11:30 PM Jean-Baptiste Onofré <
> jb@nanthrax.net>
> > > >> wrote:
> > > >>
> > > >> > As we released 2.1.0 couple of weeks ago, it could sound weird to
> > > >the
> > > >> > users to
> > > >> > do a 2.2.0 so fast. If we have a blocking issue, we can do a 2.1.1
> > > >If
> > > >> it's
> > > >> > new
> > > >> > features, why not having a release pace in October (2.2.0) ?
> > > >> >
> > > >> > Thoughts ?
> > > >> >
> > > >> > Regards
> > > >> > JB
> > > >> >
> > > >> > On 08/31/2017 08:27 AM, Eugene Kirpichov wrote:
> > > >> > > I'd suggest to do 2.2.0 as quickly as possible, and target 2.3.0
> > > >for
> > > >> > > October. I don't see a reason to delay 2.2.0 until October:
> > > >there's a
> > > >> > huge
> > > >> > > amount of features worth releasing between when 2.1.0 was cut
> and
> > > >the
> > > >> > > current HEAD.
> > > >> > >
> > > >> > > On Wed, Aug 30, 2017 at 10:18 PM Jean-Baptiste Onofré
> > > ><jb@nanthrax.net
> > > >> >
> > > >> > > wrote:
> > > >> > >
> > > >> > >> With a 2.2.0 in October, I think we can try to move forward on
> > > >> RedisIO.
> > > >> > >>
> > > >> > >> I'm now back from vacation and I will resume the work on this
> > > >IO.
> > > >> > >>
> > > >> > >> Regards
> > > >> > >> JB
> > > >> > >>
> > > >> > >> On 08/30/2017 11:27 PM, Eugene Kirpichov wrote:
> > > >> > >>> RedisIO in 2.2.0 is very unlikely. There's still a lot of
> > > >review
> > > >> > >> remaining
> > > >> > >>> last time I checked on the PR.
> > > >> > >>>
> > > >> > >>> On Wed, Aug 30, 2017 at 2:24 PM Vilhelm von Ehrenheim <
> > > >> > >>> vonehrenheim@gmail.com> wrote:
> > > >> > >>>
> > > >> > >>>> Any chance to get the RedisIO in this release?
> > > >> > >>>> [BEAM-1017] Add RedisIO #1687
> > > >> > >>>>
> > > >> > >>>> Its not my PR but ll be happy to assist if there is anything
> I
> > > >can
> > > >> do
> > > >> > to
> > > >> > >>>> help.
> > > >> > >>>>
> > > >> > >>>> On 30 Aug 2017 22:46, "Daniel Ribeiro"
> > > >> <danielrb@squareup.com.invalid
> > > >> > >
> > > >> > >>>> wrote:
> > > >> > >>>>
> > > >> > >>>>> It would be great to get a bump on pubsub
> > > >> > >>>>> <https://github.com/apache/beam/blob/master/pom.xml#L145>
> > > >> > dependency.
> > > >> > >> It
> > > >> > >>>>> is
> > > >> > >>>>> currently very outdated (v1-rev10-1.22.0, which was released
> > > >over a
> > > >> > >> year
> > > >> > >>>>> ago
> > > >> > >>>>> <http://repo1.maven.org/maven2/com/google/apis/google-
> > > >> > >>>>> api-services-pubsub/v1-rev10-1.22.0/>
> > > >> > >>>>> ).
> > > >> > >>>>>
> > > >> > >>>>>
> > > >> > >>>>> On Wed, Aug 30, 2017 at 1:27 PM, Eugene Kirpichov <
> > > >> > >>>>> kirpichov@google.com.invalid> wrote:
> > > >> > >>>>>
> > > >> > >>>>>> Thanks Ismael. I've marked these two issues for fix in
> > > >2.2.0.
> > > >> > >>>> Definitely
> > > >> > >>>>>> agree that at least the first one must be fixed.
> > > >> > >>>>>>
> > > >> > >>>>>> Here's the current burndown list
> > > >> > >>>>>>
> > > >https://issues.apache.org/jira/projects/BEAM/versions/12341044 -
> > > >> we
> > > >> > >>>>> should
> > > >> > >>>>>> clean it up.
> > > >> > >>>>>>
> > > >> > >>>>>> On Wed, Aug 30, 2017 at 1:20 PM Ismaël Mejía
> > > ><ie...@gmail.com>
> > > >> > >>>> wrote:
> > > >> > >>>>>>
> > > >> > >>>>>>> The current master has accumulated a good amount of nice
> > > >features
> > > >> > >>>>>>> since 2.1.0 so a new release is welcomed. I have two
> > > >JIRAs/PR
> > > >> that
> > > >> > I
> > > >> > >>>>>>> think are important to check/solve before the cut:
> > > >> > >>>>>>>
> > > >> > >>>>>>> BEAM-2516 (this is a regression on the performance of
> > > >Direct
> > > >> runner
> > > >> > >>>> on
> > > >> > >>>>>>> Java). We had never really defined if a performance
> > > >regression is
> > > >> > >>>>>>> critical to be a blocker. I executed WordCount with the
> > > >> > kinglear.txt
> > > >> > >>>>>>> (170KB) file in version 2.1.0 vs the current
> 2.2.0-SNAPSHOT
> > > >and I
> > > >> > >>>>>>> found that the execution time passed from 5s to 126s. So
> > > >maybe we
> > > >> > >>>> need
> > > >> > >>>>>>> to review this one before the release. I can understand if
> > > >others
> > > >> > >>>>>>> consider this a minor issue because the Direct runner is
> > > >not
> > > >> > supposed
> > > >> > >>>>>>> to be used for production, but this performance regression
> > > >can
> > > >> > cause
> > > >> > >>>> a
> > > >> > >>>>>>> bad impression for a casual user starting with Beam.
> > > >> > >>>>>>>
> > > >> > >>>>>>> BEAM-2790 (fix reading from Amazon S3 via
> > > >HadoopFileSystem). I
> > > >> > think
> > > >> > >>>>>>> this one is a nice to have. I am not sure that I can
> tackle
> > > >it
> > > >> for
> > > >> > >>>> the
> > > >> > >>>>>>> wednesday cut. I’m OOO until the beginning of next week,
> > > >but
> > > >> maybe
> > > >> > >>>>>>> someone else can take a look. In the worst case this is
> not
> > > >a
> > > >> > release
> > > >> > >>>>>>> blocker but definitely a really nice fix to include.
> > > >> > >>>>>>>
> > > >> > >>>>>>>
> > > >> > >>>>>>> On Wed, Aug 30, 2017 at 8:49 PM, Eugene Kirpichov
> > > >> > >>>>>>> <ki...@google.com.invalid> wrote:
> > > >> > >>>>>>>> I'd like to get the following PRs into 2.2.0:
> > > >> > >>>>>>>>
> > > >> > >>>>>>>> #3765 <https://github.com/apache/beam/pull/3765>
> > > >[BEAM-2753
> > > >> > >>>>>>>> <https://issues.apache.org/jira/browse/BEAM-2753>] Fixes
> > > >> > >>>> translation
> > > >> > >>>>>> of
> > > >> > >>>>>>>> WriteFiles side inputs (important bugfix for
> > > >DynamicDestinations
> > > >> > in
> > > >> > >>>>>>> files)
> > > >> > >>>>>>>> #3725 <https://github.com/apache/beam/pull/3725>
> > > >[BEAM-2827
> > > >> > >>>>>>>> <https://issues.apache.org/jira/browse/BEAM-2827>]
> > > >Introduces
> > > >> > >>>>>>>> AvroIO.watchForNewFiles (parity for AvroIO with TextIO in
> > > >a few
> > > >> > >>>>>> important
> > > >> > >>>>>>>> features)
> > > >> > >>>>>>>> #3759 <https://github.com/apache/beam/pull/3759>
> > > >[BEAM-2828
> > > >> > >>>>>>>> <https://issues.apache.org/jira/browse/BEAM-2828>] Moves
> > > >Match
> > > >> > >>>> into
> > > >> > >>>>>>>> FileIO.match()/matchAll() (to prevent releasing current
> > > >> > >>>>>>>> Match.filepatterns() into 2.2.0 and then having to keep
> it
> > > >under
> > > >> > >>>> that
> > > >> > >>>>>>> name)
> > > >> > >>>>>>>>
> > > >> > >>>>>>>> On Wed, Aug 30, 2017 at 11:31 AM Mingmin Xu
> > > ><mingmxus@gmail.com
> > > >> >
> > > >> > >>>>>> wrote:
> > > >> > >>>>>>>>
> > > >> > >>>>>>>>> Glad to see that 2.2.0 is coming. Can we include SQL
> > > >feature in
> > > >> > >>>> next
> > > >> > >>>>>>>>> release? We're in the final stage and expect to merge
> > > >back to
> > > >> > >>>> master
> > > >> > >>>>>>> this
> > > >> > >>>>>>>>> week.
> > > >> > >>>>>>>>>
> > > >> > >>>>>>>>> On Wed, Aug 30, 2017 at 11:27 AM, Reuven Lax
> > > >> > >>>>> <relax@google.com.invalid
> > > >> > >>>>>>>
> > > >> > >>>>>>>>> wrote:
> > > >> > >>>>>>>>>
> > > >> > >>>>>>>>>> Now that Beam 2.1.0 has finally completed, I think we
> > > >should
> > > >> cut
> > > >> > >>>>>> Beam
> > > >> > >>>>>>>>> 2.2.0
> > > >> > >>>>>>>>>> soon. I volunteer to coordinate this release.
> > > >> > >>>>>>>>>>
> > > >> > >>>>>>>>>> Are there any pending pull requests that people think
> > > >should
> > > >> be
> > > >> > >>>>>> merged
> > > >> > >>>>>>>>>> before we cut 2.2.0? If so, please let me know soon, as
> > > >I
> > > >> would
> > > >> > >>>>> like
> > > >> > >>>>>>> to
> > > >> > >>>>>>>>> cut
> > > >> > >>>>>>>>>> by Wednesday of next week.
> > > >> > >>>>>>>>>>
> > > >> > >>>>>>>>>> Thanks,
> > > >> > >>>>>>>>>>
> > > >> > >>>>>>>>>> Reuven
> > > >> > >>>>>>>>>>
> > > >> > >>>>>>>>>
> > > >> > >>>>>>>>>
> > > >> > >>>>>>>>>
> > > >> > >>>>>>>>> --
> > > >> > >>>>>>>>> ----
> > > >> > >>>>>>>>> Mingmin
> > > >> > >>>>>>>>>
> > > >> > >>>>>>>
> > > >> > >>>>>>
> > > >> > >>>>>
> > > >> > >>>>
> > > >> > >>>
> > > >> > >>
> > > >> > >> --
> > > >> > >> Jean-Baptiste Onofré
> > > >> > >> jbonofre@apache.org
> > > >> > >> http://blog.nanthrax.net
> > > >> > >> Talend - http://www.talend.com
> > > >> > >>
> > > >> > >
> > > >> >
> > > >> > --
> > > >> > Jean-Baptiste Onofré
> > > >> > jbonofre@apache.org
> > > >> > http://blog.nanthrax.net
> > > >> > Talend - http://www.talend.com
> > > >> >
> > > >>
> > >
> >
> >
> >
> > --
> > ----
> > Mingmin
> >
>
--
----
Mingmin
Re: Beam 2.2.0 release
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi,
Any PMC member has the permissions (updating Jira, access to
reporter.apache.org, etc).
Regards
JB
On 09/11/2017 06:46 PM, Reuven Lax wrote:
> Are these permissions that only you have, or does anyone on the PMC have
> these permissions? I'm asking so that in the future if you are unavailable,
> we know who has these permissions. We should also make sure this is all
> documented on the Beam release guide.
>
> Reuven
>
> On Wed, Sep 6, 2017 at 9:25 PM, Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
>> It sounds good to me.
>>
>> By the way, you will need my help to complete the release process (as you
>> need some permissions that you don't have).
>>
>> Regards
>> JB
>>
>>
>> On 09/07/2017 01:00 AM, Reuven Lax wrote:
>>
>>> It sounds like SQL is still not in, and there are a couple of other PRs
>>> that people have requested in 2.2.0. I am mostly out next week, so let's
>>> set September 18 as a target date for cutting the first RC. That should
>>> hopefully give plenty of time to get SQL and the remaining PRs merged into
>>> master.
>>>
>>> Reuven
>>>
>>> On Thu, Aug 31, 2017 at 3:04 PM, Mingmin Xu <mi...@gmail.com> wrote:
>>>
>>> Add https://issues.apache.org/jira/browse/BEAM-2833 which is a blocker to
>>>> merge DSL_SQL. There may be something wrong in the back-end(maybe
>>>> RunnerApi) to handle parametered CustomCoder in TestPipeline.
>>>>
>>>> On Thu, Aug 31, 2017 at 10:38 AM, Jean-Baptiste Onofré <jb...@nanthrax.net>
>>>> wrote:
>>>>
>>>> Fair enough.
>>>>>
>>>>> That's fine for me.
>>>>>
>>>>> Regards
>>>>> JB
>>>>>
>>>>> On Aug 31, 2017, 19:03, at 19:03, Steve Niemitz <sn...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> I'll chime in as a user who would love to see 2.2.0 sooner than later,
>>>>>> specifically for the file IO Eugene mentioned. We're using the AvroIO
>>>>>> enhancements extensively, but I am hesitant to run from HEAD in master
>>>>>> in
>>>>>> production.
>>>>>>
>>>>>> On Thu, Aug 31, 2017 at 12:42 PM, Eugene Kirpichov <
>>>>>> kirpichov@google.com.invalid> wrote:
>>>>>>
>>>>>> There are a lot of users including very large production customers
>>>>>>>
>>>>>> who have
>>>>>>
>>>>>>> been asking specifically for the features that are in 2.2.0 (most of
>>>>>>>
>>>>>> them
>>>>>>
>>>>>>> accumulated while 2.1.0 was being iterated on) - mostly I'm referring
>>>>>>>
>>>>>> to
>>>>>>
>>>>>>> the vastly improved file IO - and they have been hesitant to use Beam
>>>>>>>
>>>>>> at
>>>>>>
>>>>>>> HEAD in production. I think the slight unusualness of having a
>>>>>>>
>>>>>> release
>>>>>>
>>>>>>> published soon after the previous release is a small price to pay for
>>>>>>> helping those users :)
>>>>>>>
>>>>>>> On Wed, Aug 30, 2017, 11:30 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
>>>>>>> wrote:
>>>>>>>
>>>>>>> As we released 2.1.0 couple of weeks ago, it could sound weird to
>>>>>>>>
>>>>>>> the
>>>>>>
>>>>>>> users to
>>>>>>>> do a 2.2.0 so fast. If we have a blocking issue, we can do a 2.1.1
>>>>>>>>
>>>>>>> If
>>>>>>
>>>>>>> it's
>>>>>>>
>>>>>>>> new
>>>>>>>> features, why not having a release pace in October (2.2.0) ?
>>>>>>>>
>>>>>>>> Thoughts ?
>>>>>>>>
>>>>>>>> Regards
>>>>>>>> JB
>>>>>>>>
>>>>>>>> On 08/31/2017 08:27 AM, Eugene Kirpichov wrote:
>>>>>>>>
>>>>>>>>> I'd suggest to do 2.2.0 as quickly as possible, and target 2.3.0
>>>>>>>>>
>>>>>>>> for
>>>>>>
>>>>>>> October. I don't see a reason to delay 2.2.0 until October:
>>>>>>>>>
>>>>>>>> there's a
>>>>>>
>>>>>>> huge
>>>>>>>>
>>>>>>>>> amount of features worth releasing between when 2.1.0 was cut and
>>>>>>>>>
>>>>>>>> the
>>>>>>
>>>>>>> current HEAD.
>>>>>>>>>
>>>>>>>>> On Wed, Aug 30, 2017 at 10:18 PM Jean-Baptiste Onofré
>>>>>>>>>
>>>>>>>> <jb@nanthrax.net
>>>>>>
>>>>>>>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> With a 2.2.0 in October, I think we can try to move forward on
>>>>>>>>>>
>>>>>>>>> RedisIO.
>>>>>>>
>>>>>>>>
>>>>>>>>>> I'm now back from vacation and I will resume the work on this
>>>>>>>>>>
>>>>>>>>> IO.
>>>>>>
>>>>>>>
>>>>>>>>>> Regards
>>>>>>>>>> JB
>>>>>>>>>>
>>>>>>>>>> On 08/30/2017 11:27 PM, Eugene Kirpichov wrote:
>>>>>>>>>>
>>>>>>>>>>> RedisIO in 2.2.0 is very unlikely. There's still a lot of
>>>>>>>>>>>
>>>>>>>>>> review
>>>>>>
>>>>>>> remaining
>>>>>>>>>>
>>>>>>>>>>> last time I checked on the PR.
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Aug 30, 2017 at 2:24 PM Vilhelm von Ehrenheim <
>>>>>>>>>>> vonehrenheim@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Any chance to get the RedisIO in this release?
>>>>>>>>>>>> [BEAM-1017] Add RedisIO #1687
>>>>>>>>>>>>
>>>>>>>>>>>> Its not my PR but ll be happy to assist if there is anything I
>>>>>>>>>>>>
>>>>>>>>>>> can
>>>>>>
>>>>>>> do
>>>>>>>
>>>>>>>> to
>>>>>>>>
>>>>>>>>> help.
>>>>>>>>>>>>
>>>>>>>>>>>> On 30 Aug 2017 22:46, "Daniel Ribeiro"
>>>>>>>>>>>>
>>>>>>>>>>> <danielrb@squareup.com.invalid
>>>>>>>
>>>>>>>>
>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> It would be great to get a bump on pubsub
>>>>>>>>>>>>> <https://github.com/apache/beam/blob/master/pom.xml#L145>
>>>>>>>>>>>>>
>>>>>>>>>>>> dependency.
>>>>>>>>
>>>>>>>>> It
>>>>>>>>>>
>>>>>>>>>>> is
>>>>>>>>>>>>> currently very outdated (v1-rev10-1.22.0, which was released
>>>>>>>>>>>>>
>>>>>>>>>>>> over a
>>>>>>
>>>>>>> year
>>>>>>>>>>
>>>>>>>>>>> ago
>>>>>>>>>>>>> <http://repo1.maven.org/maven2/com/google/apis/google-
>>>>>>>>>>>>> api-services-pubsub/v1-rev10-1.22.0/>
>>>>>>>>>>>>> ).
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Aug 30, 2017 at 1:27 PM, Eugene Kirpichov <
>>>>>>>>>>>>> kirpichov@google.com.invalid> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks Ismael. I've marked these two issues for fix in
>>>>>>>>>>>>>>
>>>>>>>>>>>>> 2.2.0.
>>>>>>
>>>>>>> Definitely
>>>>>>>>>>>>
>>>>>>>>>>>>> agree that at least the first one must be fixed.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Here's the current burndown list
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> https://issues.apache.org/jira/projects/BEAM/versions/12341044
>>>>>> -
>>>>>>
>>>>>>> we
>>>>>>>
>>>>>>>> should
>>>>>>>>>>>>>
>>>>>>>>>>>>>> clean it up.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Aug 30, 2017 at 1:20 PM Ismaël Mejía
>>>>>>>>>>>>>>
>>>>>>>>>>>>> <ie...@gmail.com>
>>>>>>
>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> The current master has accumulated a good amount of nice
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> features
>>>>>>
>>>>>>> since 2.1.0 so a new release is welcomed. I have two
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> JIRAs/PR
>>>>>>
>>>>>>> that
>>>>>>>
>>>>>>>> I
>>>>>>>>
>>>>>>>>> think are important to check/solve before the cut:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> BEAM-2516 (this is a regression on the performance of
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Direct
>>>>>>
>>>>>>> runner
>>>>>>>
>>>>>>>> on
>>>>>>>>>>>>
>>>>>>>>>>>>> Java). We had never really defined if a performance
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> regression is
>>>>>>
>>>>>>> critical to be a blocker. I executed WordCount with the
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> kinglear.txt
>>>>>>>>
>>>>>>>>> (170KB) file in version 2.1.0 vs the current 2.2.0-SNAPSHOT
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> and I
>>>>>>
>>>>>>> found that the execution time passed from 5s to 126s. So
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> maybe we
>>>>>>
>>>>>>> need
>>>>>>>>>>>>
>>>>>>>>>>>>> to review this one before the release. I can understand if
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> others
>>>>>>
>>>>>>> consider this a minor issue because the Direct runner is
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> not
>>>>>>
>>>>>>> supposed
>>>>>>>>
>>>>>>>>> to be used for production, but this performance regression
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> can
>>>>>>
>>>>>>> cause
>>>>>>>>
>>>>>>>>> a
>>>>>>>>>>>>
>>>>>>>>>>>>> bad impression for a casual user starting with Beam.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> BEAM-2790 (fix reading from Amazon S3 via
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> HadoopFileSystem). I
>>>>>>
>>>>>>> think
>>>>>>>>
>>>>>>>>> this one is a nice to have. I am not sure that I can tackle
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> it
>>>>>>
>>>>>>> for
>>>>>>>
>>>>>>>> the
>>>>>>>>>>>>
>>>>>>>>>>>>> wednesday cut. I’m OOO until the beginning of next week,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> but
>>>>>>
>>>>>>> maybe
>>>>>>>
>>>>>>>> someone else can take a look. In the worst case this is not
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> a
>>>>>>
>>>>>>> release
>>>>>>>>
>>>>>>>>> blocker but definitely a really nice fix to include.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Wed, Aug 30, 2017 at 8:49 PM, Eugene Kirpichov
>>>>>>>>>>>>>>> <ki...@google.com.invalid> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I'd like to get the following PRs into 2.2.0:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> #3765 <https://github.com/apache/beam/pull/3765>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [BEAM-2753
>>>>>>
>>>>>>> <https://issues.apache.org/jira/browse/BEAM-2753>] Fixes
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> translation
>>>>>>>>>>>>
>>>>>>>>>>>>> of
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> WriteFiles side inputs (important bugfix for
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> DynamicDestinations
>>>>>>
>>>>>>> in
>>>>>>>>
>>>>>>>>> files)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> #3725 <https://github.com/apache/beam/pull/3725>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [BEAM-2827
>>>>>>
>>>>>>> <https://issues.apache.org/jira/browse/BEAM-2827>]
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Introduces
>>>>>>
>>>>>>> AvroIO.watchForNewFiles (parity for AvroIO with TextIO in
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> a few
>>>>>>
>>>>>>> important
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> features)
>>>>>>>>>>>>>>>> #3759 <https://github.com/apache/beam/pull/3759>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [BEAM-2828
>>>>>>
>>>>>>> <https://issues.apache.org/jira/browse/BEAM-2828>] Moves
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Match
>>>>>>
>>>>>>> into
>>>>>>>>>>>>
>>>>>>>>>>>>> FileIO.match()/matchAll() (to prevent releasing current
>>>>>>>>>>>>>>>> Match.filepatterns() into 2.2.0 and then having to keep it
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> under
>>>>>>
>>>>>>> that
>>>>>>>>>>>>
>>>>>>>>>>>>> name)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Wed, Aug 30, 2017 at 11:31 AM Mingmin Xu
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> <mingmxus@gmail.com
>>>>>>
>>>>>>>
>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Glad to see that 2.2.0 is coming. Can we include SQL
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> feature in
>>>>>>
>>>>>>> next
>>>>>>>>>>>>
>>>>>>>>>>>>> release? We're in the final stage and expect to merge
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> back to
>>>>>>
>>>>>>> master
>>>>>>>>>>>>
>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> week.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Wed, Aug 30, 2017 at 11:27 AM, Reuven Lax
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> <relax@google.com.invalid
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Now that Beam 2.1.0 has finally completed, I think we
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> should
>>>>>>
>>>>>>> cut
>>>>>>>
>>>>>>>> Beam
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2.2.0
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> soon. I volunteer to coordinate this release.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Are there any pending pull requests that people think
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> should
>>>>>>
>>>>>>> be
>>>>>>>
>>>>>>>> merged
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> before we cut 2.2.0? If so, please let me know soon, as
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I
>>>>>>
>>>>>>> would
>>>>>>>
>>>>>>>> like
>>>>>>>>>>>>>
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> cut
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> by Wednesday of next week.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Reuven
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> ----
>>>>>>>>>>>>>>>>> Mingmin
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Jean-Baptiste Onofré
>>>>>>>>>> jbonofre@apache.org
>>>>>>>>>> http://blog.nanthrax.net
>>>>>>>>>> Talend - http://www.talend.com
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>> --
>>>>>>>> Jean-Baptiste Onofré
>>>>>>>> jbonofre@apache.org
>>>>>>>> http://blog.nanthrax.net
>>>>>>>> Talend - http://www.talend.com
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> ----
>>>> Mingmin
>>>>
>>>>
>>>
>> --
>> Jean-Baptiste Onofré
>> jbonofre@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>>
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: Beam 2.2.0 release
Posted by Mingmin Xu <mi...@gmail.com>.
Thanks @JB, add you and Tyler as reviewer and is waiting for jenkins job.
On Thu, Sep 7, 2017 at 12:25 PM, Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> It sounds good to me.
>
> By the way, you will need my help to complete the release process (as you
> need some permissions that you don't have).
>
> Regards
> JB
>
>
> On 09/07/2017 01:00 AM, Reuven Lax wrote:
>
>> It sounds like SQL is still not in, and there are a couple of other PRs
>> that people have requested in 2.2.0. I am mostly out next week, so let's
>> set September 18 as a target date for cutting the first RC. That should
>> hopefully give plenty of time to get SQL and the remaining PRs merged into
>> master.
>>
>> Reuven
>>
>> On Thu, Aug 31, 2017 at 3:04 PM, Mingmin Xu <mi...@gmail.com> wrote:
>>
>> Add https://issues.apache.org/jira/browse/BEAM-2833 which is a blocker to
>>> merge DSL_SQL. There may be something wrong in the back-end(maybe
>>> RunnerApi) to handle parametered CustomCoder in TestPipeline.
>>>
>>> On Thu, Aug 31, 2017 at 10:38 AM, Jean-Baptiste Onofré <jb...@nanthrax.net>
>>> wrote:
>>>
>>> Fair enough.
>>>>
>>>> That's fine for me.
>>>>
>>>> Regards
>>>> JB
>>>>
>>>> On Aug 31, 2017, 19:03, at 19:03, Steve Niemitz <sn...@apache.org>
>>>> wrote:
>>>>
>>>>> I'll chime in as a user who would love to see 2.2.0 sooner than later,
>>>>> specifically for the file IO Eugene mentioned. We're using the AvroIO
>>>>> enhancements extensively, but I am hesitant to run from HEAD in master
>>>>> in
>>>>> production.
>>>>>
>>>>> On Thu, Aug 31, 2017 at 12:42 PM, Eugene Kirpichov <
>>>>> kirpichov@google.com.invalid> wrote:
>>>>>
>>>>> There are a lot of users including very large production customers
>>>>>>
>>>>> who have
>>>>>
>>>>>> been asking specifically for the features that are in 2.2.0 (most of
>>>>>>
>>>>> them
>>>>>
>>>>>> accumulated while 2.1.0 was being iterated on) - mostly I'm referring
>>>>>>
>>>>> to
>>>>>
>>>>>> the vastly improved file IO - and they have been hesitant to use Beam
>>>>>>
>>>>> at
>>>>>
>>>>>> HEAD in production. I think the slight unusualness of having a
>>>>>>
>>>>> release
>>>>>
>>>>>> published soon after the previous release is a small price to pay for
>>>>>> helping those users :)
>>>>>>
>>>>>> On Wed, Aug 30, 2017, 11:30 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
>>>>>> wrote:
>>>>>>
>>>>>> As we released 2.1.0 couple of weeks ago, it could sound weird to
>>>>>>>
>>>>>> the
>>>>>
>>>>>> users to
>>>>>>> do a 2.2.0 so fast. If we have a blocking issue, we can do a 2.1.1
>>>>>>>
>>>>>> If
>>>>>
>>>>>> it's
>>>>>>
>>>>>>> new
>>>>>>> features, why not having a release pace in October (2.2.0) ?
>>>>>>>
>>>>>>> Thoughts ?
>>>>>>>
>>>>>>> Regards
>>>>>>> JB
>>>>>>>
>>>>>>> On 08/31/2017 08:27 AM, Eugene Kirpichov wrote:
>>>>>>>
>>>>>>>> I'd suggest to do 2.2.0 as quickly as possible, and target 2.3.0
>>>>>>>>
>>>>>>> for
>>>>>
>>>>>> October. I don't see a reason to delay 2.2.0 until October:
>>>>>>>>
>>>>>>> there's a
>>>>>
>>>>>> huge
>>>>>>>
>>>>>>>> amount of features worth releasing between when 2.1.0 was cut and
>>>>>>>>
>>>>>>> the
>>>>>
>>>>>> current HEAD.
>>>>>>>>
>>>>>>>> On Wed, Aug 30, 2017 at 10:18 PM Jean-Baptiste Onofré
>>>>>>>>
>>>>>>> <jb@nanthrax.net
>>>>>
>>>>>>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> With a 2.2.0 in October, I think we can try to move forward on
>>>>>>>>>
>>>>>>>> RedisIO.
>>>>>>
>>>>>>>
>>>>>>>>> I'm now back from vacation and I will resume the work on this
>>>>>>>>>
>>>>>>>> IO.
>>>>>
>>>>>>
>>>>>>>>> Regards
>>>>>>>>> JB
>>>>>>>>>
>>>>>>>>> On 08/30/2017 11:27 PM, Eugene Kirpichov wrote:
>>>>>>>>>
>>>>>>>>>> RedisIO in 2.2.0 is very unlikely. There's still a lot of
>>>>>>>>>>
>>>>>>>>> review
>>>>>
>>>>>> remaining
>>>>>>>>>
>>>>>>>>>> last time I checked on the PR.
>>>>>>>>>>
>>>>>>>>>> On Wed, Aug 30, 2017 at 2:24 PM Vilhelm von Ehrenheim <
>>>>>>>>>> vonehrenheim@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>> Any chance to get the RedisIO in this release?
>>>>>>>>>>> [BEAM-1017] Add RedisIO #1687
>>>>>>>>>>>
>>>>>>>>>>> Its not my PR but ll be happy to assist if there is anything I
>>>>>>>>>>>
>>>>>>>>>> can
>>>>>
>>>>>> do
>>>>>>
>>>>>>> to
>>>>>>>
>>>>>>>> help.
>>>>>>>>>>>
>>>>>>>>>>> On 30 Aug 2017 22:46, "Daniel Ribeiro"
>>>>>>>>>>>
>>>>>>>>>> <danielrb@squareup.com.invalid
>>>>>>
>>>>>>>
>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> It would be great to get a bump on pubsub
>>>>>>>>>>>> <https://github.com/apache/beam/blob/master/pom.xml#L145>
>>>>>>>>>>>>
>>>>>>>>>>> dependency.
>>>>>>>
>>>>>>>> It
>>>>>>>>>
>>>>>>>>>> is
>>>>>>>>>>>> currently very outdated (v1-rev10-1.22.0, which was released
>>>>>>>>>>>>
>>>>>>>>>>> over a
>>>>>
>>>>>> year
>>>>>>>>>
>>>>>>>>>> ago
>>>>>>>>>>>> <http://repo1.maven.org/maven2/com/google/apis/google-
>>>>>>>>>>>> api-services-pubsub/v1-rev10-1.22.0/>
>>>>>>>>>>>> ).
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Aug 30, 2017 at 1:27 PM, Eugene Kirpichov <
>>>>>>>>>>>> kirpichov@google.com.invalid> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks Ismael. I've marked these two issues for fix in
>>>>>>>>>>>>>
>>>>>>>>>>>> 2.2.0.
>>>>>
>>>>>> Definitely
>>>>>>>>>>>
>>>>>>>>>>>> agree that at least the first one must be fixed.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Here's the current burndown list
>>>>>>>>>>>>>
>>>>>>>>>>>>> https://issues.apache.org/jira/projects/BEAM/versions/12341044
>>>>> -
>>>>>
>>>>>> we
>>>>>>
>>>>>>> should
>>>>>>>>>>>>
>>>>>>>>>>>>> clean it up.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Aug 30, 2017 at 1:20 PM Ismaël Mejía
>>>>>>>>>>>>>
>>>>>>>>>>>> <ie...@gmail.com>
>>>>>
>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> The current master has accumulated a good amount of nice
>>>>>>>>>>>>>>
>>>>>>>>>>>>> features
>>>>>
>>>>>> since 2.1.0 so a new release is welcomed. I have two
>>>>>>>>>>>>>>
>>>>>>>>>>>>> JIRAs/PR
>>>>>
>>>>>> that
>>>>>>
>>>>>>> I
>>>>>>>
>>>>>>>> think are important to check/solve before the cut:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> BEAM-2516 (this is a regression on the performance of
>>>>>>>>>>>>>>
>>>>>>>>>>>>> Direct
>>>>>
>>>>>> runner
>>>>>>
>>>>>>> on
>>>>>>>>>>>
>>>>>>>>>>>> Java). We had never really defined if a performance
>>>>>>>>>>>>>>
>>>>>>>>>>>>> regression is
>>>>>
>>>>>> critical to be a blocker. I executed WordCount with the
>>>>>>>>>>>>>>
>>>>>>>>>>>>> kinglear.txt
>>>>>>>
>>>>>>>> (170KB) file in version 2.1.0 vs the current 2.2.0-SNAPSHOT
>>>>>>>>>>>>>>
>>>>>>>>>>>>> and I
>>>>>
>>>>>> found that the execution time passed from 5s to 126s. So
>>>>>>>>>>>>>>
>>>>>>>>>>>>> maybe we
>>>>>
>>>>>> need
>>>>>>>>>>>
>>>>>>>>>>>> to review this one before the release. I can understand if
>>>>>>>>>>>>>>
>>>>>>>>>>>>> others
>>>>>
>>>>>> consider this a minor issue because the Direct runner is
>>>>>>>>>>>>>>
>>>>>>>>>>>>> not
>>>>>
>>>>>> supposed
>>>>>>>
>>>>>>>> to be used for production, but this performance regression
>>>>>>>>>>>>>>
>>>>>>>>>>>>> can
>>>>>
>>>>>> cause
>>>>>>>
>>>>>>>> a
>>>>>>>>>>>
>>>>>>>>>>>> bad impression for a casual user starting with Beam.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> BEAM-2790 (fix reading from Amazon S3 via
>>>>>>>>>>>>>>
>>>>>>>>>>>>> HadoopFileSystem). I
>>>>>
>>>>>> think
>>>>>>>
>>>>>>>> this one is a nice to have. I am not sure that I can tackle
>>>>>>>>>>>>>>
>>>>>>>>>>>>> it
>>>>>
>>>>>> for
>>>>>>
>>>>>>> the
>>>>>>>>>>>
>>>>>>>>>>>> wednesday cut. I’m OOO until the beginning of next week,
>>>>>>>>>>>>>>
>>>>>>>>>>>>> but
>>>>>
>>>>>> maybe
>>>>>>
>>>>>>> someone else can take a look. In the worst case this is not
>>>>>>>>>>>>>>
>>>>>>>>>>>>> a
>>>>>
>>>>>> release
>>>>>>>
>>>>>>>> blocker but definitely a really nice fix to include.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Aug 30, 2017 at 8:49 PM, Eugene Kirpichov
>>>>>>>>>>>>>> <ki...@google.com.invalid> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I'd like to get the following PRs into 2.2.0:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> #3765 <https://github.com/apache/beam/pull/3765>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [BEAM-2753
>>>>>
>>>>>> <https://issues.apache.org/jira/browse/BEAM-2753>] Fixes
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> translation
>>>>>>>>>>>
>>>>>>>>>>>> of
>>>>>>>>>>>>>
>>>>>>>>>>>>>> WriteFiles side inputs (important bugfix for
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> DynamicDestinations
>>>>>
>>>>>> in
>>>>>>>
>>>>>>>> files)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> #3725 <https://github.com/apache/beam/pull/3725>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [BEAM-2827
>>>>>
>>>>>> <https://issues.apache.org/jira/browse/BEAM-2827>]
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Introduces
>>>>>
>>>>>> AvroIO.watchForNewFiles (parity for AvroIO with TextIO in
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> a few
>>>>>
>>>>>> important
>>>>>>>>>>>>>
>>>>>>>>>>>>>> features)
>>>>>>>>>>>>>>> #3759 <https://github.com/apache/beam/pull/3759>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [BEAM-2828
>>>>>
>>>>>> <https://issues.apache.org/jira/browse/BEAM-2828>] Moves
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Match
>>>>>
>>>>>> into
>>>>>>>>>>>
>>>>>>>>>>>> FileIO.match()/matchAll() (to prevent releasing current
>>>>>>>>>>>>>>> Match.filepatterns() into 2.2.0 and then having to keep it
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> under
>>>>>
>>>>>> that
>>>>>>>>>>>
>>>>>>>>>>>> name)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Wed, Aug 30, 2017 at 11:31 AM Mingmin Xu
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> <mingmxus@gmail.com
>>>>>
>>>>>>
>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Glad to see that 2.2.0 is coming. Can we include SQL
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> feature in
>>>>>
>>>>>> next
>>>>>>>>>>>
>>>>>>>>>>>> release? We're in the final stage and expect to merge
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> back to
>>>>>
>>>>>> master
>>>>>>>>>>>
>>>>>>>>>>>> this
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> week.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Wed, Aug 30, 2017 at 11:27 AM, Reuven Lax
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> <relax@google.com.invalid
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Now that Beam 2.1.0 has finally completed, I think we
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> should
>>>>>
>>>>>> cut
>>>>>>
>>>>>>> Beam
>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2.2.0
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> soon. I volunteer to coordinate this release.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Are there any pending pull requests that people think
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> should
>>>>>
>>>>>> be
>>>>>>
>>>>>>> merged
>>>>>>>>>>>>>
>>>>>>>>>>>>>> before we cut 2.2.0? If so, please let me know soon, as
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I
>>>>>
>>>>>> would
>>>>>>
>>>>>>> like
>>>>>>>>>>>>
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> cut
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> by Wednesday of next week.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Reuven
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> ----
>>>>>>>>>>>>>>>> Mingmin
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Jean-Baptiste Onofré
>>>>>>>>> jbonofre@apache.org
>>>>>>>>> http://blog.nanthrax.net
>>>>>>>>> Talend - http://www.talend.com
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>> --
>>>>>>> Jean-Baptiste Onofré
>>>>>>> jbonofre@apache.org
>>>>>>> http://blog.nanthrax.net
>>>>>>> Talend - http://www.talend.com
>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>
>>>
>>> --
>>> ----
>>> Mingmin
>>>
>>>
>>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
--
----
Mingmin
Re: Beam 2.2.0 release
Posted by Mingmin Xu <mi...@gmail.com>.
I don't think there're some particular permissions, #3782 is merged to
master. So from SQL perspective I'm good to the 2.2.0 release, will do a
quick POC job for verification purpose.
Mingmin
On Mon, Sep 11, 2017 at 9:46 AM, Reuven Lax <re...@google.com.invalid>
wrote:
> Are these permissions that only you have, or does anyone on the PMC have
> these permissions? I'm asking so that in the future if you are unavailable,
> we know who has these permissions. We should also make sure this is all
> documented on the Beam release guide.
>
> Reuven
>
> On Wed, Sep 6, 2017 at 9:25 PM, Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
> > It sounds good to me.
> >
> > By the way, you will need my help to complete the release process (as you
> > need some permissions that you don't have).
> >
> > Regards
> > JB
> >
> >
> > On 09/07/2017 01:00 AM, Reuven Lax wrote:
> >
> >> It sounds like SQL is still not in, and there are a couple of other PRs
> >> that people have requested in 2.2.0. I am mostly out next week, so let's
> >> set September 18 as a target date for cutting the first RC. That should
> >> hopefully give plenty of time to get SQL and the remaining PRs merged
> into
> >> master.
> >>
> >> Reuven
> >>
> >> On Thu, Aug 31, 2017 at 3:04 PM, Mingmin Xu <mi...@gmail.com> wrote:
> >>
> >> Add https://issues.apache.org/jira/browse/BEAM-2833 which is a blocker
> to
> >>> merge DSL_SQL. There may be something wrong in the back-end(maybe
> >>> RunnerApi) to handle parametered CustomCoder in TestPipeline.
> >>>
> >>> On Thu, Aug 31, 2017 at 10:38 AM, Jean-Baptiste Onofré <
> jb@nanthrax.net>
> >>> wrote:
> >>>
> >>> Fair enough.
> >>>>
> >>>> That's fine for me.
> >>>>
> >>>> Regards
> >>>> JB
> >>>>
> >>>> On Aug 31, 2017, 19:03, at 19:03, Steve Niemitz <sn...@apache.org>
> >>>> wrote:
> >>>>
> >>>>> I'll chime in as a user who would love to see 2.2.0 sooner than
> later,
> >>>>> specifically for the file IO Eugene mentioned. We're using the
> AvroIO
> >>>>> enhancements extensively, but I am hesitant to run from HEAD in
> master
> >>>>> in
> >>>>> production.
> >>>>>
> >>>>> On Thu, Aug 31, 2017 at 12:42 PM, Eugene Kirpichov <
> >>>>> kirpichov@google.com.invalid> wrote:
> >>>>>
> >>>>> There are a lot of users including very large production customers
> >>>>>>
> >>>>> who have
> >>>>>
> >>>>>> been asking specifically for the features that are in 2.2.0 (most of
> >>>>>>
> >>>>> them
> >>>>>
> >>>>>> accumulated while 2.1.0 was being iterated on) - mostly I'm
> referring
> >>>>>>
> >>>>> to
> >>>>>
> >>>>>> the vastly improved file IO - and they have been hesitant to use
> Beam
> >>>>>>
> >>>>> at
> >>>>>
> >>>>>> HEAD in production. I think the slight unusualness of having a
> >>>>>>
> >>>>> release
> >>>>>
> >>>>>> published soon after the previous release is a small price to pay
> for
> >>>>>> helping those users :)
> >>>>>>
> >>>>>> On Wed, Aug 30, 2017, 11:30 PM Jean-Baptiste Onofré <
> jb@nanthrax.net>
> >>>>>> wrote:
> >>>>>>
> >>>>>> As we released 2.1.0 couple of weeks ago, it could sound weird to
> >>>>>>>
> >>>>>> the
> >>>>>
> >>>>>> users to
> >>>>>>> do a 2.2.0 so fast. If we have a blocking issue, we can do a 2.1.1
> >>>>>>>
> >>>>>> If
> >>>>>
> >>>>>> it's
> >>>>>>
> >>>>>>> new
> >>>>>>> features, why not having a release pace in October (2.2.0) ?
> >>>>>>>
> >>>>>>> Thoughts ?
> >>>>>>>
> >>>>>>> Regards
> >>>>>>> JB
> >>>>>>>
> >>>>>>> On 08/31/2017 08:27 AM, Eugene Kirpichov wrote:
> >>>>>>>
> >>>>>>>> I'd suggest to do 2.2.0 as quickly as possible, and target 2.3.0
> >>>>>>>>
> >>>>>>> for
> >>>>>
> >>>>>> October. I don't see a reason to delay 2.2.0 until October:
> >>>>>>>>
> >>>>>>> there's a
> >>>>>
> >>>>>> huge
> >>>>>>>
> >>>>>>>> amount of features worth releasing between when 2.1.0 was cut and
> >>>>>>>>
> >>>>>>> the
> >>>>>
> >>>>>> current HEAD.
> >>>>>>>>
> >>>>>>>> On Wed, Aug 30, 2017 at 10:18 PM Jean-Baptiste Onofré
> >>>>>>>>
> >>>>>>> <jb@nanthrax.net
> >>>>>
> >>>>>>
> >>>>>>> wrote:
> >>>>>>>>
> >>>>>>>> With a 2.2.0 in October, I think we can try to move forward on
> >>>>>>>>>
> >>>>>>>> RedisIO.
> >>>>>>
> >>>>>>>
> >>>>>>>>> I'm now back from vacation and I will resume the work on this
> >>>>>>>>>
> >>>>>>>> IO.
> >>>>>
> >>>>>>
> >>>>>>>>> Regards
> >>>>>>>>> JB
> >>>>>>>>>
> >>>>>>>>> On 08/30/2017 11:27 PM, Eugene Kirpichov wrote:
> >>>>>>>>>
> >>>>>>>>>> RedisIO in 2.2.0 is very unlikely. There's still a lot of
> >>>>>>>>>>
> >>>>>>>>> review
> >>>>>
> >>>>>> remaining
> >>>>>>>>>
> >>>>>>>>>> last time I checked on the PR.
> >>>>>>>>>>
> >>>>>>>>>> On Wed, Aug 30, 2017 at 2:24 PM Vilhelm von Ehrenheim <
> >>>>>>>>>> vonehrenheim@gmail.com> wrote:
> >>>>>>>>>>
> >>>>>>>>>> Any chance to get the RedisIO in this release?
> >>>>>>>>>>> [BEAM-1017] Add RedisIO #1687
> >>>>>>>>>>>
> >>>>>>>>>>> Its not my PR but ll be happy to assist if there is anything I
> >>>>>>>>>>>
> >>>>>>>>>> can
> >>>>>
> >>>>>> do
> >>>>>>
> >>>>>>> to
> >>>>>>>
> >>>>>>>> help.
> >>>>>>>>>>>
> >>>>>>>>>>> On 30 Aug 2017 22:46, "Daniel Ribeiro"
> >>>>>>>>>>>
> >>>>>>>>>> <danielrb@squareup.com.invalid
> >>>>>>
> >>>>>>>
> >>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> It would be great to get a bump on pubsub
> >>>>>>>>>>>> <https://github.com/apache/beam/blob/master/pom.xml#L145>
> >>>>>>>>>>>>
> >>>>>>>>>>> dependency.
> >>>>>>>
> >>>>>>>> It
> >>>>>>>>>
> >>>>>>>>>> is
> >>>>>>>>>>>> currently very outdated (v1-rev10-1.22.0, which was released
> >>>>>>>>>>>>
> >>>>>>>>>>> over a
> >>>>>
> >>>>>> year
> >>>>>>>>>
> >>>>>>>>>> ago
> >>>>>>>>>>>> <http://repo1.maven.org/maven2/com/google/apis/google-
> >>>>>>>>>>>> api-services-pubsub/v1-rev10-1.22.0/>
> >>>>>>>>>>>> ).
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Wed, Aug 30, 2017 at 1:27 PM, Eugene Kirpichov <
> >>>>>>>>>>>> kirpichov@google.com.invalid> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks Ismael. I've marked these two issues for fix in
> >>>>>>>>>>>>>
> >>>>>>>>>>>> 2.2.0.
> >>>>>
> >>>>>> Definitely
> >>>>>>>>>>>
> >>>>>>>>>>>> agree that at least the first one must be fixed.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Here's the current burndown list
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> https://issues.apache.org/jira/projects/BEAM/versions/
> 12341044
> >>>>> -
> >>>>>
> >>>>>> we
> >>>>>>
> >>>>>>> should
> >>>>>>>>>>>>
> >>>>>>>>>>>>> clean it up.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Wed, Aug 30, 2017 at 1:20 PM Ismaël Mejía
> >>>>>>>>>>>>>
> >>>>>>>>>>>> <ie...@gmail.com>
> >>>>>
> >>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>> The current master has accumulated a good amount of nice
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>> features
> >>>>>
> >>>>>> since 2.1.0 so a new release is welcomed. I have two
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>> JIRAs/PR
> >>>>>
> >>>>>> that
> >>>>>>
> >>>>>>> I
> >>>>>>>
> >>>>>>>> think are important to check/solve before the cut:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> BEAM-2516 (this is a regression on the performance of
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>> Direct
> >>>>>
> >>>>>> runner
> >>>>>>
> >>>>>>> on
> >>>>>>>>>>>
> >>>>>>>>>>>> Java). We had never really defined if a performance
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>> regression is
> >>>>>
> >>>>>> critical to be a blocker. I executed WordCount with the
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>> kinglear.txt
> >>>>>>>
> >>>>>>>> (170KB) file in version 2.1.0 vs the current 2.2.0-SNAPSHOT
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>> and I
> >>>>>
> >>>>>> found that the execution time passed from 5s to 126s. So
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>> maybe we
> >>>>>
> >>>>>> need
> >>>>>>>>>>>
> >>>>>>>>>>>> to review this one before the release. I can understand if
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>> others
> >>>>>
> >>>>>> consider this a minor issue because the Direct runner is
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>> not
> >>>>>
> >>>>>> supposed
> >>>>>>>
> >>>>>>>> to be used for production, but this performance regression
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>> can
> >>>>>
> >>>>>> cause
> >>>>>>>
> >>>>>>>> a
> >>>>>>>>>>>
> >>>>>>>>>>>> bad impression for a casual user starting with Beam.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> BEAM-2790 (fix reading from Amazon S3 via
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>> HadoopFileSystem). I
> >>>>>
> >>>>>> think
> >>>>>>>
> >>>>>>>> this one is a nice to have. I am not sure that I can tackle
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>> it
> >>>>>
> >>>>>> for
> >>>>>>
> >>>>>>> the
> >>>>>>>>>>>
> >>>>>>>>>>>> wednesday cut. I’m OOO until the beginning of next week,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>> but
> >>>>>
> >>>>>> maybe
> >>>>>>
> >>>>>>> someone else can take a look. In the worst case this is not
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>> a
> >>>>>
> >>>>>> release
> >>>>>>>
> >>>>>>>> blocker but definitely a really nice fix to include.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Wed, Aug 30, 2017 at 8:49 PM, Eugene Kirpichov
> >>>>>>>>>>>>>> <ki...@google.com.invalid> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I'd like to get the following PRs into 2.2.0:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> #3765 <https://github.com/apache/beam/pull/3765>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>> [BEAM-2753
> >>>>>
> >>>>>> <https://issues.apache.org/jira/browse/BEAM-2753>] Fixes
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>> translation
> >>>>>>>>>>>
> >>>>>>>>>>>> of
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> WriteFiles side inputs (important bugfix for
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>> DynamicDestinations
> >>>>>
> >>>>>> in
> >>>>>>>
> >>>>>>>> files)
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> #3725 <https://github.com/apache/beam/pull/3725>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>> [BEAM-2827
> >>>>>
> >>>>>> <https://issues.apache.org/jira/browse/BEAM-2827>]
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Introduces
> >>>>>
> >>>>>> AvroIO.watchForNewFiles (parity for AvroIO with TextIO in
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>> a few
> >>>>>
> >>>>>> important
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> features)
> >>>>>>>>>>>>>>> #3759 <https://github.com/apache/beam/pull/3759>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>> [BEAM-2828
> >>>>>
> >>>>>> <https://issues.apache.org/jira/browse/BEAM-2828>] Moves
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Match
> >>>>>
> >>>>>> into
> >>>>>>>>>>>
> >>>>>>>>>>>> FileIO.match()/matchAll() (to prevent releasing current
> >>>>>>>>>>>>>>> Match.filepatterns() into 2.2.0 and then having to keep it
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>> under
> >>>>>
> >>>>>> that
> >>>>>>>>>>>
> >>>>>>>>>>>> name)
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Wed, Aug 30, 2017 at 11:31 AM Mingmin Xu
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>> <mingmxus@gmail.com
> >>>>>
> >>>>>>
> >>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Glad to see that 2.2.0 is coming. Can we include SQL
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> feature in
> >>>>>
> >>>>>> next
> >>>>>>>>>>>
> >>>>>>>>>>>> release? We're in the final stage and expect to merge
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> back to
> >>>>>
> >>>>>> master
> >>>>>>>>>>>
> >>>>>>>>>>>> this
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> week.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Wed, Aug 30, 2017 at 11:27 AM, Reuven Lax
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> <relax@google.com.invalid
> >>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Now that Beam 2.1.0 has finally completed, I think we
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> should
> >>>>>
> >>>>>> cut
> >>>>>>
> >>>>>>> Beam
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> 2.2.0
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> soon. I volunteer to coordinate this release.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Are there any pending pull requests that people think
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> should
> >>>>>
> >>>>>> be
> >>>>>>
> >>>>>>> merged
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> before we cut 2.2.0? If so, please let me know soon, as
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I
> >>>>>
> >>>>>> would
> >>>>>>
> >>>>>>> like
> >>>>>>>>>>>>
> >>>>>>>>>>>>> to
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> cut
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> by Wednesday of next week.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Reuven
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>> ----
> >>>>>>>>>>>>>>>> Mingmin
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> Jean-Baptiste Onofré
> >>>>>>>>> jbonofre@apache.org
> >>>>>>>>> http://blog.nanthrax.net
> >>>>>>>>> Talend - http://www.talend.com
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>> --
> >>>>>>> Jean-Baptiste Onofré
> >>>>>>> jbonofre@apache.org
> >>>>>>> http://blog.nanthrax.net
> >>>>>>> Talend - http://www.talend.com
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>>
> >>>
> >>> --
> >>> ----
> >>> Mingmin
> >>>
> >>>
> >>
> > --
> > Jean-Baptiste Onofré
> > jbonofre@apache.org
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
> >
>
--
----
Mingmin
Re: Beam 2.2.0 release
Posted by Reuven Lax <re...@google.com.INVALID>.
Are these permissions that only you have, or does anyone on the PMC have
these permissions? I'm asking so that in the future if you are unavailable,
we know who has these permissions. We should also make sure this is all
documented on the Beam release guide.
Reuven
On Wed, Sep 6, 2017 at 9:25 PM, Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> It sounds good to me.
>
> By the way, you will need my help to complete the release process (as you
> need some permissions that you don't have).
>
> Regards
> JB
>
>
> On 09/07/2017 01:00 AM, Reuven Lax wrote:
>
>> It sounds like SQL is still not in, and there are a couple of other PRs
>> that people have requested in 2.2.0. I am mostly out next week, so let's
>> set September 18 as a target date for cutting the first RC. That should
>> hopefully give plenty of time to get SQL and the remaining PRs merged into
>> master.
>>
>> Reuven
>>
>> On Thu, Aug 31, 2017 at 3:04 PM, Mingmin Xu <mi...@gmail.com> wrote:
>>
>> Add https://issues.apache.org/jira/browse/BEAM-2833 which is a blocker to
>>> merge DSL_SQL. There may be something wrong in the back-end(maybe
>>> RunnerApi) to handle parametered CustomCoder in TestPipeline.
>>>
>>> On Thu, Aug 31, 2017 at 10:38 AM, Jean-Baptiste Onofré <jb...@nanthrax.net>
>>> wrote:
>>>
>>> Fair enough.
>>>>
>>>> That's fine for me.
>>>>
>>>> Regards
>>>> JB
>>>>
>>>> On Aug 31, 2017, 19:03, at 19:03, Steve Niemitz <sn...@apache.org>
>>>> wrote:
>>>>
>>>>> I'll chime in as a user who would love to see 2.2.0 sooner than later,
>>>>> specifically for the file IO Eugene mentioned. We're using the AvroIO
>>>>> enhancements extensively, but I am hesitant to run from HEAD in master
>>>>> in
>>>>> production.
>>>>>
>>>>> On Thu, Aug 31, 2017 at 12:42 PM, Eugene Kirpichov <
>>>>> kirpichov@google.com.invalid> wrote:
>>>>>
>>>>> There are a lot of users including very large production customers
>>>>>>
>>>>> who have
>>>>>
>>>>>> been asking specifically for the features that are in 2.2.0 (most of
>>>>>>
>>>>> them
>>>>>
>>>>>> accumulated while 2.1.0 was being iterated on) - mostly I'm referring
>>>>>>
>>>>> to
>>>>>
>>>>>> the vastly improved file IO - and they have been hesitant to use Beam
>>>>>>
>>>>> at
>>>>>
>>>>>> HEAD in production. I think the slight unusualness of having a
>>>>>>
>>>>> release
>>>>>
>>>>>> published soon after the previous release is a small price to pay for
>>>>>> helping those users :)
>>>>>>
>>>>>> On Wed, Aug 30, 2017, 11:30 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
>>>>>> wrote:
>>>>>>
>>>>>> As we released 2.1.0 couple of weeks ago, it could sound weird to
>>>>>>>
>>>>>> the
>>>>>
>>>>>> users to
>>>>>>> do a 2.2.0 so fast. If we have a blocking issue, we can do a 2.1.1
>>>>>>>
>>>>>> If
>>>>>
>>>>>> it's
>>>>>>
>>>>>>> new
>>>>>>> features, why not having a release pace in October (2.2.0) ?
>>>>>>>
>>>>>>> Thoughts ?
>>>>>>>
>>>>>>> Regards
>>>>>>> JB
>>>>>>>
>>>>>>> On 08/31/2017 08:27 AM, Eugene Kirpichov wrote:
>>>>>>>
>>>>>>>> I'd suggest to do 2.2.0 as quickly as possible, and target 2.3.0
>>>>>>>>
>>>>>>> for
>>>>>
>>>>>> October. I don't see a reason to delay 2.2.0 until October:
>>>>>>>>
>>>>>>> there's a
>>>>>
>>>>>> huge
>>>>>>>
>>>>>>>> amount of features worth releasing between when 2.1.0 was cut and
>>>>>>>>
>>>>>>> the
>>>>>
>>>>>> current HEAD.
>>>>>>>>
>>>>>>>> On Wed, Aug 30, 2017 at 10:18 PM Jean-Baptiste Onofré
>>>>>>>>
>>>>>>> <jb@nanthrax.net
>>>>>
>>>>>>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> With a 2.2.0 in October, I think we can try to move forward on
>>>>>>>>>
>>>>>>>> RedisIO.
>>>>>>
>>>>>>>
>>>>>>>>> I'm now back from vacation and I will resume the work on this
>>>>>>>>>
>>>>>>>> IO.
>>>>>
>>>>>>
>>>>>>>>> Regards
>>>>>>>>> JB
>>>>>>>>>
>>>>>>>>> On 08/30/2017 11:27 PM, Eugene Kirpichov wrote:
>>>>>>>>>
>>>>>>>>>> RedisIO in 2.2.0 is very unlikely. There's still a lot of
>>>>>>>>>>
>>>>>>>>> review
>>>>>
>>>>>> remaining
>>>>>>>>>
>>>>>>>>>> last time I checked on the PR.
>>>>>>>>>>
>>>>>>>>>> On Wed, Aug 30, 2017 at 2:24 PM Vilhelm von Ehrenheim <
>>>>>>>>>> vonehrenheim@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>> Any chance to get the RedisIO in this release?
>>>>>>>>>>> [BEAM-1017] Add RedisIO #1687
>>>>>>>>>>>
>>>>>>>>>>> Its not my PR but ll be happy to assist if there is anything I
>>>>>>>>>>>
>>>>>>>>>> can
>>>>>
>>>>>> do
>>>>>>
>>>>>>> to
>>>>>>>
>>>>>>>> help.
>>>>>>>>>>>
>>>>>>>>>>> On 30 Aug 2017 22:46, "Daniel Ribeiro"
>>>>>>>>>>>
>>>>>>>>>> <danielrb@squareup.com.invalid
>>>>>>
>>>>>>>
>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> It would be great to get a bump on pubsub
>>>>>>>>>>>> <https://github.com/apache/beam/blob/master/pom.xml#L145>
>>>>>>>>>>>>
>>>>>>>>>>> dependency.
>>>>>>>
>>>>>>>> It
>>>>>>>>>
>>>>>>>>>> is
>>>>>>>>>>>> currently very outdated (v1-rev10-1.22.0, which was released
>>>>>>>>>>>>
>>>>>>>>>>> over a
>>>>>
>>>>>> year
>>>>>>>>>
>>>>>>>>>> ago
>>>>>>>>>>>> <http://repo1.maven.org/maven2/com/google/apis/google-
>>>>>>>>>>>> api-services-pubsub/v1-rev10-1.22.0/>
>>>>>>>>>>>> ).
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Aug 30, 2017 at 1:27 PM, Eugene Kirpichov <
>>>>>>>>>>>> kirpichov@google.com.invalid> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks Ismael. I've marked these two issues for fix in
>>>>>>>>>>>>>
>>>>>>>>>>>> 2.2.0.
>>>>>
>>>>>> Definitely
>>>>>>>>>>>
>>>>>>>>>>>> agree that at least the first one must be fixed.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Here's the current burndown list
>>>>>>>>>>>>>
>>>>>>>>>>>>> https://issues.apache.org/jira/projects/BEAM/versions/12341044
>>>>> -
>>>>>
>>>>>> we
>>>>>>
>>>>>>> should
>>>>>>>>>>>>
>>>>>>>>>>>>> clean it up.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Aug 30, 2017 at 1:20 PM Ismaël Mejía
>>>>>>>>>>>>>
>>>>>>>>>>>> <ie...@gmail.com>
>>>>>
>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> The current master has accumulated a good amount of nice
>>>>>>>>>>>>>>
>>>>>>>>>>>>> features
>>>>>
>>>>>> since 2.1.0 so a new release is welcomed. I have two
>>>>>>>>>>>>>>
>>>>>>>>>>>>> JIRAs/PR
>>>>>
>>>>>> that
>>>>>>
>>>>>>> I
>>>>>>>
>>>>>>>> think are important to check/solve before the cut:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> BEAM-2516 (this is a regression on the performance of
>>>>>>>>>>>>>>
>>>>>>>>>>>>> Direct
>>>>>
>>>>>> runner
>>>>>>
>>>>>>> on
>>>>>>>>>>>
>>>>>>>>>>>> Java). We had never really defined if a performance
>>>>>>>>>>>>>>
>>>>>>>>>>>>> regression is
>>>>>
>>>>>> critical to be a blocker. I executed WordCount with the
>>>>>>>>>>>>>>
>>>>>>>>>>>>> kinglear.txt
>>>>>>>
>>>>>>>> (170KB) file in version 2.1.0 vs the current 2.2.0-SNAPSHOT
>>>>>>>>>>>>>>
>>>>>>>>>>>>> and I
>>>>>
>>>>>> found that the execution time passed from 5s to 126s. So
>>>>>>>>>>>>>>
>>>>>>>>>>>>> maybe we
>>>>>
>>>>>> need
>>>>>>>>>>>
>>>>>>>>>>>> to review this one before the release. I can understand if
>>>>>>>>>>>>>>
>>>>>>>>>>>>> others
>>>>>
>>>>>> consider this a minor issue because the Direct runner is
>>>>>>>>>>>>>>
>>>>>>>>>>>>> not
>>>>>
>>>>>> supposed
>>>>>>>
>>>>>>>> to be used for production, but this performance regression
>>>>>>>>>>>>>>
>>>>>>>>>>>>> can
>>>>>
>>>>>> cause
>>>>>>>
>>>>>>>> a
>>>>>>>>>>>
>>>>>>>>>>>> bad impression for a casual user starting with Beam.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> BEAM-2790 (fix reading from Amazon S3 via
>>>>>>>>>>>>>>
>>>>>>>>>>>>> HadoopFileSystem). I
>>>>>
>>>>>> think
>>>>>>>
>>>>>>>> this one is a nice to have. I am not sure that I can tackle
>>>>>>>>>>>>>>
>>>>>>>>>>>>> it
>>>>>
>>>>>> for
>>>>>>
>>>>>>> the
>>>>>>>>>>>
>>>>>>>>>>>> wednesday cut. I’m OOO until the beginning of next week,
>>>>>>>>>>>>>>
>>>>>>>>>>>>> but
>>>>>
>>>>>> maybe
>>>>>>
>>>>>>> someone else can take a look. In the worst case this is not
>>>>>>>>>>>>>>
>>>>>>>>>>>>> a
>>>>>
>>>>>> release
>>>>>>>
>>>>>>>> blocker but definitely a really nice fix to include.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Aug 30, 2017 at 8:49 PM, Eugene Kirpichov
>>>>>>>>>>>>>> <ki...@google.com.invalid> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I'd like to get the following PRs into 2.2.0:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> #3765 <https://github.com/apache/beam/pull/3765>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [BEAM-2753
>>>>>
>>>>>> <https://issues.apache.org/jira/browse/BEAM-2753>] Fixes
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> translation
>>>>>>>>>>>
>>>>>>>>>>>> of
>>>>>>>>>>>>>
>>>>>>>>>>>>>> WriteFiles side inputs (important bugfix for
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> DynamicDestinations
>>>>>
>>>>>> in
>>>>>>>
>>>>>>>> files)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> #3725 <https://github.com/apache/beam/pull/3725>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [BEAM-2827
>>>>>
>>>>>> <https://issues.apache.org/jira/browse/BEAM-2827>]
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Introduces
>>>>>
>>>>>> AvroIO.watchForNewFiles (parity for AvroIO with TextIO in
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> a few
>>>>>
>>>>>> important
>>>>>>>>>>>>>
>>>>>>>>>>>>>> features)
>>>>>>>>>>>>>>> #3759 <https://github.com/apache/beam/pull/3759>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [BEAM-2828
>>>>>
>>>>>> <https://issues.apache.org/jira/browse/BEAM-2828>] Moves
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Match
>>>>>
>>>>>> into
>>>>>>>>>>>
>>>>>>>>>>>> FileIO.match()/matchAll() (to prevent releasing current
>>>>>>>>>>>>>>> Match.filepatterns() into 2.2.0 and then having to keep it
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> under
>>>>>
>>>>>> that
>>>>>>>>>>>
>>>>>>>>>>>> name)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Wed, Aug 30, 2017 at 11:31 AM Mingmin Xu
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> <mingmxus@gmail.com
>>>>>
>>>>>>
>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Glad to see that 2.2.0 is coming. Can we include SQL
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> feature in
>>>>>
>>>>>> next
>>>>>>>>>>>
>>>>>>>>>>>> release? We're in the final stage and expect to merge
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> back to
>>>>>
>>>>>> master
>>>>>>>>>>>
>>>>>>>>>>>> this
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> week.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Wed, Aug 30, 2017 at 11:27 AM, Reuven Lax
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> <relax@google.com.invalid
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Now that Beam 2.1.0 has finally completed, I think we
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> should
>>>>>
>>>>>> cut
>>>>>>
>>>>>>> Beam
>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2.2.0
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> soon. I volunteer to coordinate this release.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Are there any pending pull requests that people think
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> should
>>>>>
>>>>>> be
>>>>>>
>>>>>>> merged
>>>>>>>>>>>>>
>>>>>>>>>>>>>> before we cut 2.2.0? If so, please let me know soon, as
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I
>>>>>
>>>>>> would
>>>>>>
>>>>>>> like
>>>>>>>>>>>>
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> cut
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> by Wednesday of next week.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Reuven
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> ----
>>>>>>>>>>>>>>>> Mingmin
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Jean-Baptiste Onofré
>>>>>>>>> jbonofre@apache.org
>>>>>>>>> http://blog.nanthrax.net
>>>>>>>>> Talend - http://www.talend.com
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>> --
>>>>>>> Jean-Baptiste Onofré
>>>>>>> jbonofre@apache.org
>>>>>>> http://blog.nanthrax.net
>>>>>>> Talend - http://www.talend.com
>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>
>>>
>>> --
>>> ----
>>> Mingmin
>>>
>>>
>>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
Re: Beam 2.2.0 release
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
It sounds good to me.
By the way, you will need my help to complete the release process (as you need
some permissions that you don't have).
Regards
JB
On 09/07/2017 01:00 AM, Reuven Lax wrote:
> It sounds like SQL is still not in, and there are a couple of other PRs
> that people have requested in 2.2.0. I am mostly out next week, so let's
> set September 18 as a target date for cutting the first RC. That should
> hopefully give plenty of time to get SQL and the remaining PRs merged into
> master.
>
> Reuven
>
> On Thu, Aug 31, 2017 at 3:04 PM, Mingmin Xu <mi...@gmail.com> wrote:
>
>> Add https://issues.apache.org/jira/browse/BEAM-2833 which is a blocker to
>> merge DSL_SQL. There may be something wrong in the back-end(maybe
>> RunnerApi) to handle parametered CustomCoder in TestPipeline.
>>
>> On Thu, Aug 31, 2017 at 10:38 AM, Jean-Baptiste Onofré <jb...@nanthrax.net>
>> wrote:
>>
>>> Fair enough.
>>>
>>> That's fine for me.
>>>
>>> Regards
>>> JB
>>>
>>> On Aug 31, 2017, 19:03, at 19:03, Steve Niemitz <sn...@apache.org>
>>> wrote:
>>>> I'll chime in as a user who would love to see 2.2.0 sooner than later,
>>>> specifically for the file IO Eugene mentioned. We're using the AvroIO
>>>> enhancements extensively, but I am hesitant to run from HEAD in master
>>>> in
>>>> production.
>>>>
>>>> On Thu, Aug 31, 2017 at 12:42 PM, Eugene Kirpichov <
>>>> kirpichov@google.com.invalid> wrote:
>>>>
>>>>> There are a lot of users including very large production customers
>>>> who have
>>>>> been asking specifically for the features that are in 2.2.0 (most of
>>>> them
>>>>> accumulated while 2.1.0 was being iterated on) - mostly I'm referring
>>>> to
>>>>> the vastly improved file IO - and they have been hesitant to use Beam
>>>> at
>>>>> HEAD in production. I think the slight unusualness of having a
>>>> release
>>>>> published soon after the previous release is a small price to pay for
>>>>> helping those users :)
>>>>>
>>>>> On Wed, Aug 30, 2017, 11:30 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
>>>>> wrote:
>>>>>
>>>>>> As we released 2.1.0 couple of weeks ago, it could sound weird to
>>>> the
>>>>>> users to
>>>>>> do a 2.2.0 so fast. If we have a blocking issue, we can do a 2.1.1
>>>> If
>>>>> it's
>>>>>> new
>>>>>> features, why not having a release pace in October (2.2.0) ?
>>>>>>
>>>>>> Thoughts ?
>>>>>>
>>>>>> Regards
>>>>>> JB
>>>>>>
>>>>>> On 08/31/2017 08:27 AM, Eugene Kirpichov wrote:
>>>>>>> I'd suggest to do 2.2.0 as quickly as possible, and target 2.3.0
>>>> for
>>>>>>> October. I don't see a reason to delay 2.2.0 until October:
>>>> there's a
>>>>>> huge
>>>>>>> amount of features worth releasing between when 2.1.0 was cut and
>>>> the
>>>>>>> current HEAD.
>>>>>>>
>>>>>>> On Wed, Aug 30, 2017 at 10:18 PM Jean-Baptiste Onofré
>>>> <jb@nanthrax.net
>>>>>>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> With a 2.2.0 in October, I think we can try to move forward on
>>>>> RedisIO.
>>>>>>>>
>>>>>>>> I'm now back from vacation and I will resume the work on this
>>>> IO.
>>>>>>>>
>>>>>>>> Regards
>>>>>>>> JB
>>>>>>>>
>>>>>>>> On 08/30/2017 11:27 PM, Eugene Kirpichov wrote:
>>>>>>>>> RedisIO in 2.2.0 is very unlikely. There's still a lot of
>>>> review
>>>>>>>> remaining
>>>>>>>>> last time I checked on the PR.
>>>>>>>>>
>>>>>>>>> On Wed, Aug 30, 2017 at 2:24 PM Vilhelm von Ehrenheim <
>>>>>>>>> vonehrenheim@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Any chance to get the RedisIO in this release?
>>>>>>>>>> [BEAM-1017] Add RedisIO #1687
>>>>>>>>>>
>>>>>>>>>> Its not my PR but ll be happy to assist if there is anything I
>>>> can
>>>>> do
>>>>>> to
>>>>>>>>>> help.
>>>>>>>>>>
>>>>>>>>>> On 30 Aug 2017 22:46, "Daniel Ribeiro"
>>>>> <danielrb@squareup.com.invalid
>>>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> It would be great to get a bump on pubsub
>>>>>>>>>>> <https://github.com/apache/beam/blob/master/pom.xml#L145>
>>>>>> dependency.
>>>>>>>> It
>>>>>>>>>>> is
>>>>>>>>>>> currently very outdated (v1-rev10-1.22.0, which was released
>>>> over a
>>>>>>>> year
>>>>>>>>>>> ago
>>>>>>>>>>> <http://repo1.maven.org/maven2/com/google/apis/google-
>>>>>>>>>>> api-services-pubsub/v1-rev10-1.22.0/>
>>>>>>>>>>> ).
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Aug 30, 2017 at 1:27 PM, Eugene Kirpichov <
>>>>>>>>>>> kirpichov@google.com.invalid> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Thanks Ismael. I've marked these two issues for fix in
>>>> 2.2.0.
>>>>>>>>>> Definitely
>>>>>>>>>>>> agree that at least the first one must be fixed.
>>>>>>>>>>>>
>>>>>>>>>>>> Here's the current burndown list
>>>>>>>>>>>>
>>>> https://issues.apache.org/jira/projects/BEAM/versions/12341044 -
>>>>> we
>>>>>>>>>>> should
>>>>>>>>>>>> clean it up.
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Aug 30, 2017 at 1:20 PM Ismaël Mejía
>>>> <ie...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> The current master has accumulated a good amount of nice
>>>> features
>>>>>>>>>>>>> since 2.1.0 so a new release is welcomed. I have two
>>>> JIRAs/PR
>>>>> that
>>>>>> I
>>>>>>>>>>>>> think are important to check/solve before the cut:
>>>>>>>>>>>>>
>>>>>>>>>>>>> BEAM-2516 (this is a regression on the performance of
>>>> Direct
>>>>> runner
>>>>>>>>>> on
>>>>>>>>>>>>> Java). We had never really defined if a performance
>>>> regression is
>>>>>>>>>>>>> critical to be a blocker. I executed WordCount with the
>>>>>> kinglear.txt
>>>>>>>>>>>>> (170KB) file in version 2.1.0 vs the current 2.2.0-SNAPSHOT
>>>> and I
>>>>>>>>>>>>> found that the execution time passed from 5s to 126s. So
>>>> maybe we
>>>>>>>>>> need
>>>>>>>>>>>>> to review this one before the release. I can understand if
>>>> others
>>>>>>>>>>>>> consider this a minor issue because the Direct runner is
>>>> not
>>>>>> supposed
>>>>>>>>>>>>> to be used for production, but this performance regression
>>>> can
>>>>>> cause
>>>>>>>>>> a
>>>>>>>>>>>>> bad impression for a casual user starting with Beam.
>>>>>>>>>>>>>
>>>>>>>>>>>>> BEAM-2790 (fix reading from Amazon S3 via
>>>> HadoopFileSystem). I
>>>>>> think
>>>>>>>>>>>>> this one is a nice to have. I am not sure that I can tackle
>>>> it
>>>>> for
>>>>>>>>>> the
>>>>>>>>>>>>> wednesday cut. I’m OOO until the beginning of next week,
>>>> but
>>>>> maybe
>>>>>>>>>>>>> someone else can take a look. In the worst case this is not
>>>> a
>>>>>> release
>>>>>>>>>>>>> blocker but definitely a really nice fix to include.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Aug 30, 2017 at 8:49 PM, Eugene Kirpichov
>>>>>>>>>>>>> <ki...@google.com.invalid> wrote:
>>>>>>>>>>>>>> I'd like to get the following PRs into 2.2.0:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> #3765 <https://github.com/apache/beam/pull/3765>
>>>> [BEAM-2753
>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/BEAM-2753>] Fixes
>>>>>>>>>> translation
>>>>>>>>>>>> of
>>>>>>>>>>>>>> WriteFiles side inputs (important bugfix for
>>>> DynamicDestinations
>>>>>> in
>>>>>>>>>>>>> files)
>>>>>>>>>>>>>> #3725 <https://github.com/apache/beam/pull/3725>
>>>> [BEAM-2827
>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/BEAM-2827>]
>>>> Introduces
>>>>>>>>>>>>>> AvroIO.watchForNewFiles (parity for AvroIO with TextIO in
>>>> a few
>>>>>>>>>>>> important
>>>>>>>>>>>>>> features)
>>>>>>>>>>>>>> #3759 <https://github.com/apache/beam/pull/3759>
>>>> [BEAM-2828
>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/BEAM-2828>] Moves
>>>> Match
>>>>>>>>>> into
>>>>>>>>>>>>>> FileIO.match()/matchAll() (to prevent releasing current
>>>>>>>>>>>>>> Match.filepatterns() into 2.2.0 and then having to keep it
>>>> under
>>>>>>>>>> that
>>>>>>>>>>>>> name)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Aug 30, 2017 at 11:31 AM Mingmin Xu
>>>> <mingmxus@gmail.com
>>>>>>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Glad to see that 2.2.0 is coming. Can we include SQL
>>>> feature in
>>>>>>>>>> next
>>>>>>>>>>>>>>> release? We're in the final stage and expect to merge
>>>> back to
>>>>>>>>>> master
>>>>>>>>>>>>> this
>>>>>>>>>>>>>>> week.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Wed, Aug 30, 2017 at 11:27 AM, Reuven Lax
>>>>>>>>>>> <relax@google.com.invalid
>>>>>>>>>>>>>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Now that Beam 2.1.0 has finally completed, I think we
>>>> should
>>>>> cut
>>>>>>>>>>>> Beam
>>>>>>>>>>>>>>> 2.2.0
>>>>>>>>>>>>>>>> soon. I volunteer to coordinate this release.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Are there any pending pull requests that people think
>>>> should
>>>>> be
>>>>>>>>>>>> merged
>>>>>>>>>>>>>>>> before we cut 2.2.0? If so, please let me know soon, as
>>>> I
>>>>> would
>>>>>>>>>>> like
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> cut
>>>>>>>>>>>>>>>> by Wednesday of next week.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Reuven
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> ----
>>>>>>>>>>>>>>> Mingmin
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Jean-Baptiste Onofré
>>>>>>>> jbonofre@apache.org
>>>>>>>> http://blog.nanthrax.net
>>>>>>>> Talend - http://www.talend.com
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Jean-Baptiste Onofré
>>>>>> jbonofre@apache.org
>>>>>> http://blog.nanthrax.net
>>>>>> Talend - http://www.talend.com
>>>>>>
>>>>>
>>>
>>
>>
>>
>> --
>> ----
>> Mingmin
>>
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: Beam 2.2.0 release
Posted by Reuven Lax <re...@google.com.INVALID>.
It sounds like SQL is still not in, and there are a couple of other PRs
that people have requested in 2.2.0. I am mostly out next week, so let's
set September 18 as a target date for cutting the first RC. That should
hopefully give plenty of time to get SQL and the remaining PRs merged into
master.
Reuven
On Thu, Aug 31, 2017 at 3:04 PM, Mingmin Xu <mi...@gmail.com> wrote:
> Add https://issues.apache.org/jira/browse/BEAM-2833 which is a blocker to
> merge DSL_SQL. There may be something wrong in the back-end(maybe
> RunnerApi) to handle parametered CustomCoder in TestPipeline.
>
> On Thu, Aug 31, 2017 at 10:38 AM, Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
> > Fair enough.
> >
> > That's fine for me.
> >
> > Regards
> > JB
> >
> > On Aug 31, 2017, 19:03, at 19:03, Steve Niemitz <sn...@apache.org>
> > wrote:
> > >I'll chime in as a user who would love to see 2.2.0 sooner than later,
> > >specifically for the file IO Eugene mentioned. We're using the AvroIO
> > >enhancements extensively, but I am hesitant to run from HEAD in master
> > >in
> > >production.
> > >
> > >On Thu, Aug 31, 2017 at 12:42 PM, Eugene Kirpichov <
> > >kirpichov@google.com.invalid> wrote:
> > >
> > >> There are a lot of users including very large production customers
> > >who have
> > >> been asking specifically for the features that are in 2.2.0 (most of
> > >them
> > >> accumulated while 2.1.0 was being iterated on) - mostly I'm referring
> > >to
> > >> the vastly improved file IO - and they have been hesitant to use Beam
> > >at
> > >> HEAD in production. I think the slight unusualness of having a
> > >release
> > >> published soon after the previous release is a small price to pay for
> > >> helping those users :)
> > >>
> > >> On Wed, Aug 30, 2017, 11:30 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
> > >> wrote:
> > >>
> > >> > As we released 2.1.0 couple of weeks ago, it could sound weird to
> > >the
> > >> > users to
> > >> > do a 2.2.0 so fast. If we have a blocking issue, we can do a 2.1.1
> > >If
> > >> it's
> > >> > new
> > >> > features, why not having a release pace in October (2.2.0) ?
> > >> >
> > >> > Thoughts ?
> > >> >
> > >> > Regards
> > >> > JB
> > >> >
> > >> > On 08/31/2017 08:27 AM, Eugene Kirpichov wrote:
> > >> > > I'd suggest to do 2.2.0 as quickly as possible, and target 2.3.0
> > >for
> > >> > > October. I don't see a reason to delay 2.2.0 until October:
> > >there's a
> > >> > huge
> > >> > > amount of features worth releasing between when 2.1.0 was cut and
> > >the
> > >> > > current HEAD.
> > >> > >
> > >> > > On Wed, Aug 30, 2017 at 10:18 PM Jean-Baptiste Onofré
> > ><jb@nanthrax.net
> > >> >
> > >> > > wrote:
> > >> > >
> > >> > >> With a 2.2.0 in October, I think we can try to move forward on
> > >> RedisIO.
> > >> > >>
> > >> > >> I'm now back from vacation and I will resume the work on this
> > >IO.
> > >> > >>
> > >> > >> Regards
> > >> > >> JB
> > >> > >>
> > >> > >> On 08/30/2017 11:27 PM, Eugene Kirpichov wrote:
> > >> > >>> RedisIO in 2.2.0 is very unlikely. There's still a lot of
> > >review
> > >> > >> remaining
> > >> > >>> last time I checked on the PR.
> > >> > >>>
> > >> > >>> On Wed, Aug 30, 2017 at 2:24 PM Vilhelm von Ehrenheim <
> > >> > >>> vonehrenheim@gmail.com> wrote:
> > >> > >>>
> > >> > >>>> Any chance to get the RedisIO in this release?
> > >> > >>>> [BEAM-1017] Add RedisIO #1687
> > >> > >>>>
> > >> > >>>> Its not my PR but ll be happy to assist if there is anything I
> > >can
> > >> do
> > >> > to
> > >> > >>>> help.
> > >> > >>>>
> > >> > >>>> On 30 Aug 2017 22:46, "Daniel Ribeiro"
> > >> <danielrb@squareup.com.invalid
> > >> > >
> > >> > >>>> wrote:
> > >> > >>>>
> > >> > >>>>> It would be great to get a bump on pubsub
> > >> > >>>>> <https://github.com/apache/beam/blob/master/pom.xml#L145>
> > >> > dependency.
> > >> > >> It
> > >> > >>>>> is
> > >> > >>>>> currently very outdated (v1-rev10-1.22.0, which was released
> > >over a
> > >> > >> year
> > >> > >>>>> ago
> > >> > >>>>> <http://repo1.maven.org/maven2/com/google/apis/google-
> > >> > >>>>> api-services-pubsub/v1-rev10-1.22.0/>
> > >> > >>>>> ).
> > >> > >>>>>
> > >> > >>>>>
> > >> > >>>>> On Wed, Aug 30, 2017 at 1:27 PM, Eugene Kirpichov <
> > >> > >>>>> kirpichov@google.com.invalid> wrote:
> > >> > >>>>>
> > >> > >>>>>> Thanks Ismael. I've marked these two issues for fix in
> > >2.2.0.
> > >> > >>>> Definitely
> > >> > >>>>>> agree that at least the first one must be fixed.
> > >> > >>>>>>
> > >> > >>>>>> Here's the current burndown list
> > >> > >>>>>>
> > >https://issues.apache.org/jira/projects/BEAM/versions/12341044 -
> > >> we
> > >> > >>>>> should
> > >> > >>>>>> clean it up.
> > >> > >>>>>>
> > >> > >>>>>> On Wed, Aug 30, 2017 at 1:20 PM Ismaël Mejía
> > ><ie...@gmail.com>
> > >> > >>>> wrote:
> > >> > >>>>>>
> > >> > >>>>>>> The current master has accumulated a good amount of nice
> > >features
> > >> > >>>>>>> since 2.1.0 so a new release is welcomed. I have two
> > >JIRAs/PR
> > >> that
> > >> > I
> > >> > >>>>>>> think are important to check/solve before the cut:
> > >> > >>>>>>>
> > >> > >>>>>>> BEAM-2516 (this is a regression on the performance of
> > >Direct
> > >> runner
> > >> > >>>> on
> > >> > >>>>>>> Java). We had never really defined if a performance
> > >regression is
> > >> > >>>>>>> critical to be a blocker. I executed WordCount with the
> > >> > kinglear.txt
> > >> > >>>>>>> (170KB) file in version 2.1.0 vs the current 2.2.0-SNAPSHOT
> > >and I
> > >> > >>>>>>> found that the execution time passed from 5s to 126s. So
> > >maybe we
> > >> > >>>> need
> > >> > >>>>>>> to review this one before the release. I can understand if
> > >others
> > >> > >>>>>>> consider this a minor issue because the Direct runner is
> > >not
> > >> > supposed
> > >> > >>>>>>> to be used for production, but this performance regression
> > >can
> > >> > cause
> > >> > >>>> a
> > >> > >>>>>>> bad impression for a casual user starting with Beam.
> > >> > >>>>>>>
> > >> > >>>>>>> BEAM-2790 (fix reading from Amazon S3 via
> > >HadoopFileSystem). I
> > >> > think
> > >> > >>>>>>> this one is a nice to have. I am not sure that I can tackle
> > >it
> > >> for
> > >> > >>>> the
> > >> > >>>>>>> wednesday cut. I’m OOO until the beginning of next week,
> > >but
> > >> maybe
> > >> > >>>>>>> someone else can take a look. In the worst case this is not
> > >a
> > >> > release
> > >> > >>>>>>> blocker but definitely a really nice fix to include.
> > >> > >>>>>>>
> > >> > >>>>>>>
> > >> > >>>>>>> On Wed, Aug 30, 2017 at 8:49 PM, Eugene Kirpichov
> > >> > >>>>>>> <ki...@google.com.invalid> wrote:
> > >> > >>>>>>>> I'd like to get the following PRs into 2.2.0:
> > >> > >>>>>>>>
> > >> > >>>>>>>> #3765 <https://github.com/apache/beam/pull/3765>
> > >[BEAM-2753
> > >> > >>>>>>>> <https://issues.apache.org/jira/browse/BEAM-2753>] Fixes
> > >> > >>>> translation
> > >> > >>>>>> of
> > >> > >>>>>>>> WriteFiles side inputs (important bugfix for
> > >DynamicDestinations
> > >> > in
> > >> > >>>>>>> files)
> > >> > >>>>>>>> #3725 <https://github.com/apache/beam/pull/3725>
> > >[BEAM-2827
> > >> > >>>>>>>> <https://issues.apache.org/jira/browse/BEAM-2827>]
> > >Introduces
> > >> > >>>>>>>> AvroIO.watchForNewFiles (parity for AvroIO with TextIO in
> > >a few
> > >> > >>>>>> important
> > >> > >>>>>>>> features)
> > >> > >>>>>>>> #3759 <https://github.com/apache/beam/pull/3759>
> > >[BEAM-2828
> > >> > >>>>>>>> <https://issues.apache.org/jira/browse/BEAM-2828>] Moves
> > >Match
> > >> > >>>> into
> > >> > >>>>>>>> FileIO.match()/matchAll() (to prevent releasing current
> > >> > >>>>>>>> Match.filepatterns() into 2.2.0 and then having to keep it
> > >under
> > >> > >>>> that
> > >> > >>>>>>> name)
> > >> > >>>>>>>>
> > >> > >>>>>>>> On Wed, Aug 30, 2017 at 11:31 AM Mingmin Xu
> > ><mingmxus@gmail.com
> > >> >
> > >> > >>>>>> wrote:
> > >> > >>>>>>>>
> > >> > >>>>>>>>> Glad to see that 2.2.0 is coming. Can we include SQL
> > >feature in
> > >> > >>>> next
> > >> > >>>>>>>>> release? We're in the final stage and expect to merge
> > >back to
> > >> > >>>> master
> > >> > >>>>>>> this
> > >> > >>>>>>>>> week.
> > >> > >>>>>>>>>
> > >> > >>>>>>>>> On Wed, Aug 30, 2017 at 11:27 AM, Reuven Lax
> > >> > >>>>> <relax@google.com.invalid
> > >> > >>>>>>>
> > >> > >>>>>>>>> wrote:
> > >> > >>>>>>>>>
> > >> > >>>>>>>>>> Now that Beam 2.1.0 has finally completed, I think we
> > >should
> > >> cut
> > >> > >>>>>> Beam
> > >> > >>>>>>>>> 2.2.0
> > >> > >>>>>>>>>> soon. I volunteer to coordinate this release.
> > >> > >>>>>>>>>>
> > >> > >>>>>>>>>> Are there any pending pull requests that people think
> > >should
> > >> be
> > >> > >>>>>> merged
> > >> > >>>>>>>>>> before we cut 2.2.0? If so, please let me know soon, as
> > >I
> > >> would
> > >> > >>>>> like
> > >> > >>>>>>> to
> > >> > >>>>>>>>> cut
> > >> > >>>>>>>>>> by Wednesday of next week.
> > >> > >>>>>>>>>>
> > >> > >>>>>>>>>> Thanks,
> > >> > >>>>>>>>>>
> > >> > >>>>>>>>>> Reuven
> > >> > >>>>>>>>>>
> > >> > >>>>>>>>>
> > >> > >>>>>>>>>
> > >> > >>>>>>>>>
> > >> > >>>>>>>>> --
> > >> > >>>>>>>>> ----
> > >> > >>>>>>>>> Mingmin
> > >> > >>>>>>>>>
> > >> > >>>>>>>
> > >> > >>>>>>
> > >> > >>>>>
> > >> > >>>>
> > >> > >>>
> > >> > >>
> > >> > >> --
> > >> > >> Jean-Baptiste Onofré
> > >> > >> jbonofre@apache.org
> > >> > >> http://blog.nanthrax.net
> > >> > >> Talend - http://www.talend.com
> > >> > >>
> > >> > >
> > >> >
> > >> > --
> > >> > Jean-Baptiste Onofré
> > >> > jbonofre@apache.org
> > >> > http://blog.nanthrax.net
> > >> > Talend - http://www.talend.com
> > >> >
> > >>
> >
>
>
>
> --
> ----
> Mingmin
>
Re: Beam 2.2.0 release
Posted by Mingmin Xu <mi...@gmail.com>.
Add https://issues.apache.org/jira/browse/BEAM-2833 which is a blocker to
merge DSL_SQL. There may be something wrong in the back-end(maybe
RunnerApi) to handle parametered CustomCoder in TestPipeline.
On Thu, Aug 31, 2017 at 10:38 AM, Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> Fair enough.
>
> That's fine for me.
>
> Regards
> JB
>
> On Aug 31, 2017, 19:03, at 19:03, Steve Niemitz <sn...@apache.org>
> wrote:
> >I'll chime in as a user who would love to see 2.2.0 sooner than later,
> >specifically for the file IO Eugene mentioned. We're using the AvroIO
> >enhancements extensively, but I am hesitant to run from HEAD in master
> >in
> >production.
> >
> >On Thu, Aug 31, 2017 at 12:42 PM, Eugene Kirpichov <
> >kirpichov@google.com.invalid> wrote:
> >
> >> There are a lot of users including very large production customers
> >who have
> >> been asking specifically for the features that are in 2.2.0 (most of
> >them
> >> accumulated while 2.1.0 was being iterated on) - mostly I'm referring
> >to
> >> the vastly improved file IO - and they have been hesitant to use Beam
> >at
> >> HEAD in production. I think the slight unusualness of having a
> >release
> >> published soon after the previous release is a small price to pay for
> >> helping those users :)
> >>
> >> On Wed, Aug 30, 2017, 11:30 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
> >> wrote:
> >>
> >> > As we released 2.1.0 couple of weeks ago, it could sound weird to
> >the
> >> > users to
> >> > do a 2.2.0 so fast. If we have a blocking issue, we can do a 2.1.1
> >If
> >> it's
> >> > new
> >> > features, why not having a release pace in October (2.2.0) ?
> >> >
> >> > Thoughts ?
> >> >
> >> > Regards
> >> > JB
> >> >
> >> > On 08/31/2017 08:27 AM, Eugene Kirpichov wrote:
> >> > > I'd suggest to do 2.2.0 as quickly as possible, and target 2.3.0
> >for
> >> > > October. I don't see a reason to delay 2.2.0 until October:
> >there's a
> >> > huge
> >> > > amount of features worth releasing between when 2.1.0 was cut and
> >the
> >> > > current HEAD.
> >> > >
> >> > > On Wed, Aug 30, 2017 at 10:18 PM Jean-Baptiste Onofré
> ><jb@nanthrax.net
> >> >
> >> > > wrote:
> >> > >
> >> > >> With a 2.2.0 in October, I think we can try to move forward on
> >> RedisIO.
> >> > >>
> >> > >> I'm now back from vacation and I will resume the work on this
> >IO.
> >> > >>
> >> > >> Regards
> >> > >> JB
> >> > >>
> >> > >> On 08/30/2017 11:27 PM, Eugene Kirpichov wrote:
> >> > >>> RedisIO in 2.2.0 is very unlikely. There's still a lot of
> >review
> >> > >> remaining
> >> > >>> last time I checked on the PR.
> >> > >>>
> >> > >>> On Wed, Aug 30, 2017 at 2:24 PM Vilhelm von Ehrenheim <
> >> > >>> vonehrenheim@gmail.com> wrote:
> >> > >>>
> >> > >>>> Any chance to get the RedisIO in this release?
> >> > >>>> [BEAM-1017] Add RedisIO #1687
> >> > >>>>
> >> > >>>> Its not my PR but ll be happy to assist if there is anything I
> >can
> >> do
> >> > to
> >> > >>>> help.
> >> > >>>>
> >> > >>>> On 30 Aug 2017 22:46, "Daniel Ribeiro"
> >> <danielrb@squareup.com.invalid
> >> > >
> >> > >>>> wrote:
> >> > >>>>
> >> > >>>>> It would be great to get a bump on pubsub
> >> > >>>>> <https://github.com/apache/beam/blob/master/pom.xml#L145>
> >> > dependency.
> >> > >> It
> >> > >>>>> is
> >> > >>>>> currently very outdated (v1-rev10-1.22.0, which was released
> >over a
> >> > >> year
> >> > >>>>> ago
> >> > >>>>> <http://repo1.maven.org/maven2/com/google/apis/google-
> >> > >>>>> api-services-pubsub/v1-rev10-1.22.0/>
> >> > >>>>> ).
> >> > >>>>>
> >> > >>>>>
> >> > >>>>> On Wed, Aug 30, 2017 at 1:27 PM, Eugene Kirpichov <
> >> > >>>>> kirpichov@google.com.invalid> wrote:
> >> > >>>>>
> >> > >>>>>> Thanks Ismael. I've marked these two issues for fix in
> >2.2.0.
> >> > >>>> Definitely
> >> > >>>>>> agree that at least the first one must be fixed.
> >> > >>>>>>
> >> > >>>>>> Here's the current burndown list
> >> > >>>>>>
> >https://issues.apache.org/jira/projects/BEAM/versions/12341044 -
> >> we
> >> > >>>>> should
> >> > >>>>>> clean it up.
> >> > >>>>>>
> >> > >>>>>> On Wed, Aug 30, 2017 at 1:20 PM Ismaël Mejía
> ><ie...@gmail.com>
> >> > >>>> wrote:
> >> > >>>>>>
> >> > >>>>>>> The current master has accumulated a good amount of nice
> >features
> >> > >>>>>>> since 2.1.0 so a new release is welcomed. I have two
> >JIRAs/PR
> >> that
> >> > I
> >> > >>>>>>> think are important to check/solve before the cut:
> >> > >>>>>>>
> >> > >>>>>>> BEAM-2516 (this is a regression on the performance of
> >Direct
> >> runner
> >> > >>>> on
> >> > >>>>>>> Java). We had never really defined if a performance
> >regression is
> >> > >>>>>>> critical to be a blocker. I executed WordCount with the
> >> > kinglear.txt
> >> > >>>>>>> (170KB) file in version 2.1.0 vs the current 2.2.0-SNAPSHOT
> >and I
> >> > >>>>>>> found that the execution time passed from 5s to 126s. So
> >maybe we
> >> > >>>> need
> >> > >>>>>>> to review this one before the release. I can understand if
> >others
> >> > >>>>>>> consider this a minor issue because the Direct runner is
> >not
> >> > supposed
> >> > >>>>>>> to be used for production, but this performance regression
> >can
> >> > cause
> >> > >>>> a
> >> > >>>>>>> bad impression for a casual user starting with Beam.
> >> > >>>>>>>
> >> > >>>>>>> BEAM-2790 (fix reading from Amazon S3 via
> >HadoopFileSystem). I
> >> > think
> >> > >>>>>>> this one is a nice to have. I am not sure that I can tackle
> >it
> >> for
> >> > >>>> the
> >> > >>>>>>> wednesday cut. I’m OOO until the beginning of next week,
> >but
> >> maybe
> >> > >>>>>>> someone else can take a look. In the worst case this is not
> >a
> >> > release
> >> > >>>>>>> blocker but definitely a really nice fix to include.
> >> > >>>>>>>
> >> > >>>>>>>
> >> > >>>>>>> On Wed, Aug 30, 2017 at 8:49 PM, Eugene Kirpichov
> >> > >>>>>>> <ki...@google.com.invalid> wrote:
> >> > >>>>>>>> I'd like to get the following PRs into 2.2.0:
> >> > >>>>>>>>
> >> > >>>>>>>> #3765 <https://github.com/apache/beam/pull/3765>
> >[BEAM-2753
> >> > >>>>>>>> <https://issues.apache.org/jira/browse/BEAM-2753>] Fixes
> >> > >>>> translation
> >> > >>>>>> of
> >> > >>>>>>>> WriteFiles side inputs (important bugfix for
> >DynamicDestinations
> >> > in
> >> > >>>>>>> files)
> >> > >>>>>>>> #3725 <https://github.com/apache/beam/pull/3725>
> >[BEAM-2827
> >> > >>>>>>>> <https://issues.apache.org/jira/browse/BEAM-2827>]
> >Introduces
> >> > >>>>>>>> AvroIO.watchForNewFiles (parity for AvroIO with TextIO in
> >a few
> >> > >>>>>> important
> >> > >>>>>>>> features)
> >> > >>>>>>>> #3759 <https://github.com/apache/beam/pull/3759>
> >[BEAM-2828
> >> > >>>>>>>> <https://issues.apache.org/jira/browse/BEAM-2828>] Moves
> >Match
> >> > >>>> into
> >> > >>>>>>>> FileIO.match()/matchAll() (to prevent releasing current
> >> > >>>>>>>> Match.filepatterns() into 2.2.0 and then having to keep it
> >under
> >> > >>>> that
> >> > >>>>>>> name)
> >> > >>>>>>>>
> >> > >>>>>>>> On Wed, Aug 30, 2017 at 11:31 AM Mingmin Xu
> ><mingmxus@gmail.com
> >> >
> >> > >>>>>> wrote:
> >> > >>>>>>>>
> >> > >>>>>>>>> Glad to see that 2.2.0 is coming. Can we include SQL
> >feature in
> >> > >>>> next
> >> > >>>>>>>>> release? We're in the final stage and expect to merge
> >back to
> >> > >>>> master
> >> > >>>>>>> this
> >> > >>>>>>>>> week.
> >> > >>>>>>>>>
> >> > >>>>>>>>> On Wed, Aug 30, 2017 at 11:27 AM, Reuven Lax
> >> > >>>>> <relax@google.com.invalid
> >> > >>>>>>>
> >> > >>>>>>>>> wrote:
> >> > >>>>>>>>>
> >> > >>>>>>>>>> Now that Beam 2.1.0 has finally completed, I think we
> >should
> >> cut
> >> > >>>>>> Beam
> >> > >>>>>>>>> 2.2.0
> >> > >>>>>>>>>> soon. I volunteer to coordinate this release.
> >> > >>>>>>>>>>
> >> > >>>>>>>>>> Are there any pending pull requests that people think
> >should
> >> be
> >> > >>>>>> merged
> >> > >>>>>>>>>> before we cut 2.2.0? If so, please let me know soon, as
> >I
> >> would
> >> > >>>>> like
> >> > >>>>>>> to
> >> > >>>>>>>>> cut
> >> > >>>>>>>>>> by Wednesday of next week.
> >> > >>>>>>>>>>
> >> > >>>>>>>>>> Thanks,
> >> > >>>>>>>>>>
> >> > >>>>>>>>>> Reuven
> >> > >>>>>>>>>>
> >> > >>>>>>>>>
> >> > >>>>>>>>>
> >> > >>>>>>>>>
> >> > >>>>>>>>> --
> >> > >>>>>>>>> ----
> >> > >>>>>>>>> Mingmin
> >> > >>>>>>>>>
> >> > >>>>>>>
> >> > >>>>>>
> >> > >>>>>
> >> > >>>>
> >> > >>>
> >> > >>
> >> > >> --
> >> > >> Jean-Baptiste Onofré
> >> > >> jbonofre@apache.org
> >> > >> http://blog.nanthrax.net
> >> > >> Talend - http://www.talend.com
> >> > >>
> >> > >
> >> >
> >> > --
> >> > Jean-Baptiste Onofré
> >> > jbonofre@apache.org
> >> > http://blog.nanthrax.net
> >> > Talend - http://www.talend.com
> >> >
> >>
>
--
----
Mingmin
Re: Beam 2.2.0 release
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Fair enough.
That's fine for me.
Regards
JB
On Aug 31, 2017, 19:03, at 19:03, Steve Niemitz <sn...@apache.org> wrote:
>I'll chime in as a user who would love to see 2.2.0 sooner than later,
>specifically for the file IO Eugene mentioned. We're using the AvroIO
>enhancements extensively, but I am hesitant to run from HEAD in master
>in
>production.
>
>On Thu, Aug 31, 2017 at 12:42 PM, Eugene Kirpichov <
>kirpichov@google.com.invalid> wrote:
>
>> There are a lot of users including very large production customers
>who have
>> been asking specifically for the features that are in 2.2.0 (most of
>them
>> accumulated while 2.1.0 was being iterated on) - mostly I'm referring
>to
>> the vastly improved file IO - and they have been hesitant to use Beam
>at
>> HEAD in production. I think the slight unusualness of having a
>release
>> published soon after the previous release is a small price to pay for
>> helping those users :)
>>
>> On Wed, Aug 30, 2017, 11:30 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
>> wrote:
>>
>> > As we released 2.1.0 couple of weeks ago, it could sound weird to
>the
>> > users to
>> > do a 2.2.0 so fast. If we have a blocking issue, we can do a 2.1.1
>If
>> it's
>> > new
>> > features, why not having a release pace in October (2.2.0) ?
>> >
>> > Thoughts ?
>> >
>> > Regards
>> > JB
>> >
>> > On 08/31/2017 08:27 AM, Eugene Kirpichov wrote:
>> > > I'd suggest to do 2.2.0 as quickly as possible, and target 2.3.0
>for
>> > > October. I don't see a reason to delay 2.2.0 until October:
>there's a
>> > huge
>> > > amount of features worth releasing between when 2.1.0 was cut and
>the
>> > > current HEAD.
>> > >
>> > > On Wed, Aug 30, 2017 at 10:18 PM Jean-Baptiste Onofré
><jb@nanthrax.net
>> >
>> > > wrote:
>> > >
>> > >> With a 2.2.0 in October, I think we can try to move forward on
>> RedisIO.
>> > >>
>> > >> I'm now back from vacation and I will resume the work on this
>IO.
>> > >>
>> > >> Regards
>> > >> JB
>> > >>
>> > >> On 08/30/2017 11:27 PM, Eugene Kirpichov wrote:
>> > >>> RedisIO in 2.2.0 is very unlikely. There's still a lot of
>review
>> > >> remaining
>> > >>> last time I checked on the PR.
>> > >>>
>> > >>> On Wed, Aug 30, 2017 at 2:24 PM Vilhelm von Ehrenheim <
>> > >>> vonehrenheim@gmail.com> wrote:
>> > >>>
>> > >>>> Any chance to get the RedisIO in this release?
>> > >>>> [BEAM-1017] Add RedisIO #1687
>> > >>>>
>> > >>>> Its not my PR but ll be happy to assist if there is anything I
>can
>> do
>> > to
>> > >>>> help.
>> > >>>>
>> > >>>> On 30 Aug 2017 22:46, "Daniel Ribeiro"
>> <danielrb@squareup.com.invalid
>> > >
>> > >>>> wrote:
>> > >>>>
>> > >>>>> It would be great to get a bump on pubsub
>> > >>>>> <https://github.com/apache/beam/blob/master/pom.xml#L145>
>> > dependency.
>> > >> It
>> > >>>>> is
>> > >>>>> currently very outdated (v1-rev10-1.22.0, which was released
>over a
>> > >> year
>> > >>>>> ago
>> > >>>>> <http://repo1.maven.org/maven2/com/google/apis/google-
>> > >>>>> api-services-pubsub/v1-rev10-1.22.0/>
>> > >>>>> ).
>> > >>>>>
>> > >>>>>
>> > >>>>> On Wed, Aug 30, 2017 at 1:27 PM, Eugene Kirpichov <
>> > >>>>> kirpichov@google.com.invalid> wrote:
>> > >>>>>
>> > >>>>>> Thanks Ismael. I've marked these two issues for fix in
>2.2.0.
>> > >>>> Definitely
>> > >>>>>> agree that at least the first one must be fixed.
>> > >>>>>>
>> > >>>>>> Here's the current burndown list
>> > >>>>>>
>https://issues.apache.org/jira/projects/BEAM/versions/12341044 -
>> we
>> > >>>>> should
>> > >>>>>> clean it up.
>> > >>>>>>
>> > >>>>>> On Wed, Aug 30, 2017 at 1:20 PM Ismaël Mejía
><ie...@gmail.com>
>> > >>>> wrote:
>> > >>>>>>
>> > >>>>>>> The current master has accumulated a good amount of nice
>features
>> > >>>>>>> since 2.1.0 so a new release is welcomed. I have two
>JIRAs/PR
>> that
>> > I
>> > >>>>>>> think are important to check/solve before the cut:
>> > >>>>>>>
>> > >>>>>>> BEAM-2516 (this is a regression on the performance of
>Direct
>> runner
>> > >>>> on
>> > >>>>>>> Java). We had never really defined if a performance
>regression is
>> > >>>>>>> critical to be a blocker. I executed WordCount with the
>> > kinglear.txt
>> > >>>>>>> (170KB) file in version 2.1.0 vs the current 2.2.0-SNAPSHOT
>and I
>> > >>>>>>> found that the execution time passed from 5s to 126s. So
>maybe we
>> > >>>> need
>> > >>>>>>> to review this one before the release. I can understand if
>others
>> > >>>>>>> consider this a minor issue because the Direct runner is
>not
>> > supposed
>> > >>>>>>> to be used for production, but this performance regression
>can
>> > cause
>> > >>>> a
>> > >>>>>>> bad impression for a casual user starting with Beam.
>> > >>>>>>>
>> > >>>>>>> BEAM-2790 (fix reading from Amazon S3 via
>HadoopFileSystem). I
>> > think
>> > >>>>>>> this one is a nice to have. I am not sure that I can tackle
>it
>> for
>> > >>>> the
>> > >>>>>>> wednesday cut. I’m OOO until the beginning of next week,
>but
>> maybe
>> > >>>>>>> someone else can take a look. In the worst case this is not
>a
>> > release
>> > >>>>>>> blocker but definitely a really nice fix to include.
>> > >>>>>>>
>> > >>>>>>>
>> > >>>>>>> On Wed, Aug 30, 2017 at 8:49 PM, Eugene Kirpichov
>> > >>>>>>> <ki...@google.com.invalid> wrote:
>> > >>>>>>>> I'd like to get the following PRs into 2.2.0:
>> > >>>>>>>>
>> > >>>>>>>> #3765 <https://github.com/apache/beam/pull/3765>
>[BEAM-2753
>> > >>>>>>>> <https://issues.apache.org/jira/browse/BEAM-2753>] Fixes
>> > >>>> translation
>> > >>>>>> of
>> > >>>>>>>> WriteFiles side inputs (important bugfix for
>DynamicDestinations
>> > in
>> > >>>>>>> files)
>> > >>>>>>>> #3725 <https://github.com/apache/beam/pull/3725>
>[BEAM-2827
>> > >>>>>>>> <https://issues.apache.org/jira/browse/BEAM-2827>]
>Introduces
>> > >>>>>>>> AvroIO.watchForNewFiles (parity for AvroIO with TextIO in
>a few
>> > >>>>>> important
>> > >>>>>>>> features)
>> > >>>>>>>> #3759 <https://github.com/apache/beam/pull/3759>
>[BEAM-2828
>> > >>>>>>>> <https://issues.apache.org/jira/browse/BEAM-2828>] Moves
>Match
>> > >>>> into
>> > >>>>>>>> FileIO.match()/matchAll() (to prevent releasing current
>> > >>>>>>>> Match.filepatterns() into 2.2.0 and then having to keep it
>under
>> > >>>> that
>> > >>>>>>> name)
>> > >>>>>>>>
>> > >>>>>>>> On Wed, Aug 30, 2017 at 11:31 AM Mingmin Xu
><mingmxus@gmail.com
>> >
>> > >>>>>> wrote:
>> > >>>>>>>>
>> > >>>>>>>>> Glad to see that 2.2.0 is coming. Can we include SQL
>feature in
>> > >>>> next
>> > >>>>>>>>> release? We're in the final stage and expect to merge
>back to
>> > >>>> master
>> > >>>>>>> this
>> > >>>>>>>>> week.
>> > >>>>>>>>>
>> > >>>>>>>>> On Wed, Aug 30, 2017 at 11:27 AM, Reuven Lax
>> > >>>>> <relax@google.com.invalid
>> > >>>>>>>
>> > >>>>>>>>> wrote:
>> > >>>>>>>>>
>> > >>>>>>>>>> Now that Beam 2.1.0 has finally completed, I think we
>should
>> cut
>> > >>>>>> Beam
>> > >>>>>>>>> 2.2.0
>> > >>>>>>>>>> soon. I volunteer to coordinate this release.
>> > >>>>>>>>>>
>> > >>>>>>>>>> Are there any pending pull requests that people think
>should
>> be
>> > >>>>>> merged
>> > >>>>>>>>>> before we cut 2.2.0? If so, please let me know soon, as
>I
>> would
>> > >>>>> like
>> > >>>>>>> to
>> > >>>>>>>>> cut
>> > >>>>>>>>>> by Wednesday of next week.
>> > >>>>>>>>>>
>> > >>>>>>>>>> Thanks,
>> > >>>>>>>>>>
>> > >>>>>>>>>> Reuven
>> > >>>>>>>>>>
>> > >>>>>>>>>
>> > >>>>>>>>>
>> > >>>>>>>>>
>> > >>>>>>>>> --
>> > >>>>>>>>> ----
>> > >>>>>>>>> Mingmin
>> > >>>>>>>>>
>> > >>>>>>>
>> > >>>>>>
>> > >>>>>
>> > >>>>
>> > >>>
>> > >>
>> > >> --
>> > >> Jean-Baptiste Onofré
>> > >> jbonofre@apache.org
>> > >> http://blog.nanthrax.net
>> > >> Talend - http://www.talend.com
>> > >>
>> > >
>> >
>> > --
>> > Jean-Baptiste Onofré
>> > jbonofre@apache.org
>> > http://blog.nanthrax.net
>> > Talend - http://www.talend.com
>> >
>>
Re: Beam 2.2.0 release
Posted by Steve Niemitz <sn...@apache.org>.
I'll chime in as a user who would love to see 2.2.0 sooner than later,
specifically for the file IO Eugene mentioned. We're using the AvroIO
enhancements extensively, but I am hesitant to run from HEAD in master in
production.
On Thu, Aug 31, 2017 at 12:42 PM, Eugene Kirpichov <
kirpichov@google.com.invalid> wrote:
> There are a lot of users including very large production customers who have
> been asking specifically for the features that are in 2.2.0 (most of them
> accumulated while 2.1.0 was being iterated on) - mostly I'm referring to
> the vastly improved file IO - and they have been hesitant to use Beam at
> HEAD in production. I think the slight unusualness of having a release
> published soon after the previous release is a small price to pay for
> helping those users :)
>
> On Wed, Aug 30, 2017, 11:30 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
> > As we released 2.1.0 couple of weeks ago, it could sound weird to the
> > users to
> > do a 2.2.0 so fast. If we have a blocking issue, we can do a 2.1.1 If
> it's
> > new
> > features, why not having a release pace in October (2.2.0) ?
> >
> > Thoughts ?
> >
> > Regards
> > JB
> >
> > On 08/31/2017 08:27 AM, Eugene Kirpichov wrote:
> > > I'd suggest to do 2.2.0 as quickly as possible, and target 2.3.0 for
> > > October. I don't see a reason to delay 2.2.0 until October: there's a
> > huge
> > > amount of features worth releasing between when 2.1.0 was cut and the
> > > current HEAD.
> > >
> > > On Wed, Aug 30, 2017 at 10:18 PM Jean-Baptiste Onofré <jb@nanthrax.net
> >
> > > wrote:
> > >
> > >> With a 2.2.0 in October, I think we can try to move forward on
> RedisIO.
> > >>
> > >> I'm now back from vacation and I will resume the work on this IO.
> > >>
> > >> Regards
> > >> JB
> > >>
> > >> On 08/30/2017 11:27 PM, Eugene Kirpichov wrote:
> > >>> RedisIO in 2.2.0 is very unlikely. There's still a lot of review
> > >> remaining
> > >>> last time I checked on the PR.
> > >>>
> > >>> On Wed, Aug 30, 2017 at 2:24 PM Vilhelm von Ehrenheim <
> > >>> vonehrenheim@gmail.com> wrote:
> > >>>
> > >>>> Any chance to get the RedisIO in this release?
> > >>>> [BEAM-1017] Add RedisIO #1687
> > >>>>
> > >>>> Its not my PR but ll be happy to assist if there is anything I can
> do
> > to
> > >>>> help.
> > >>>>
> > >>>> On 30 Aug 2017 22:46, "Daniel Ribeiro"
> <danielrb@squareup.com.invalid
> > >
> > >>>> wrote:
> > >>>>
> > >>>>> It would be great to get a bump on pubsub
> > >>>>> <https://github.com/apache/beam/blob/master/pom.xml#L145>
> > dependency.
> > >> It
> > >>>>> is
> > >>>>> currently very outdated (v1-rev10-1.22.0, which was released over a
> > >> year
> > >>>>> ago
> > >>>>> <http://repo1.maven.org/maven2/com/google/apis/google-
> > >>>>> api-services-pubsub/v1-rev10-1.22.0/>
> > >>>>> ).
> > >>>>>
> > >>>>>
> > >>>>> On Wed, Aug 30, 2017 at 1:27 PM, Eugene Kirpichov <
> > >>>>> kirpichov@google.com.invalid> wrote:
> > >>>>>
> > >>>>>> Thanks Ismael. I've marked these two issues for fix in 2.2.0.
> > >>>> Definitely
> > >>>>>> agree that at least the first one must be fixed.
> > >>>>>>
> > >>>>>> Here's the current burndown list
> > >>>>>> https://issues.apache.org/jira/projects/BEAM/versions/12341044 -
> we
> > >>>>> should
> > >>>>>> clean it up.
> > >>>>>>
> > >>>>>> On Wed, Aug 30, 2017 at 1:20 PM Ismaël Mejía <ie...@gmail.com>
> > >>>> wrote:
> > >>>>>>
> > >>>>>>> The current master has accumulated a good amount of nice features
> > >>>>>>> since 2.1.0 so a new release is welcomed. I have two JIRAs/PR
> that
> > I
> > >>>>>>> think are important to check/solve before the cut:
> > >>>>>>>
> > >>>>>>> BEAM-2516 (this is a regression on the performance of Direct
> runner
> > >>>> on
> > >>>>>>> Java). We had never really defined if a performance regression is
> > >>>>>>> critical to be a blocker. I executed WordCount with the
> > kinglear.txt
> > >>>>>>> (170KB) file in version 2.1.0 vs the current 2.2.0-SNAPSHOT and I
> > >>>>>>> found that the execution time passed from 5s to 126s. So maybe we
> > >>>> need
> > >>>>>>> to review this one before the release. I can understand if others
> > >>>>>>> consider this a minor issue because the Direct runner is not
> > supposed
> > >>>>>>> to be used for production, but this performance regression can
> > cause
> > >>>> a
> > >>>>>>> bad impression for a casual user starting with Beam.
> > >>>>>>>
> > >>>>>>> BEAM-2790 (fix reading from Amazon S3 via HadoopFileSystem). I
> > think
> > >>>>>>> this one is a nice to have. I am not sure that I can tackle it
> for
> > >>>> the
> > >>>>>>> wednesday cut. I’m OOO until the beginning of next week, but
> maybe
> > >>>>>>> someone else can take a look. In the worst case this is not a
> > release
> > >>>>>>> blocker but definitely a really nice fix to include.
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> On Wed, Aug 30, 2017 at 8:49 PM, Eugene Kirpichov
> > >>>>>>> <ki...@google.com.invalid> wrote:
> > >>>>>>>> I'd like to get the following PRs into 2.2.0:
> > >>>>>>>>
> > >>>>>>>> #3765 <https://github.com/apache/beam/pull/3765> [BEAM-2753
> > >>>>>>>> <https://issues.apache.org/jira/browse/BEAM-2753>] Fixes
> > >>>> translation
> > >>>>>> of
> > >>>>>>>> WriteFiles side inputs (important bugfix for DynamicDestinations
> > in
> > >>>>>>> files)
> > >>>>>>>> #3725 <https://github.com/apache/beam/pull/3725> [BEAM-2827
> > >>>>>>>> <https://issues.apache.org/jira/browse/BEAM-2827>] Introduces
> > >>>>>>>> AvroIO.watchForNewFiles (parity for AvroIO with TextIO in a few
> > >>>>>> important
> > >>>>>>>> features)
> > >>>>>>>> #3759 <https://github.com/apache/beam/pull/3759> [BEAM-2828
> > >>>>>>>> <https://issues.apache.org/jira/browse/BEAM-2828>] Moves Match
> > >>>> into
> > >>>>>>>> FileIO.match()/matchAll() (to prevent releasing current
> > >>>>>>>> Match.filepatterns() into 2.2.0 and then having to keep it under
> > >>>> that
> > >>>>>>> name)
> > >>>>>>>>
> > >>>>>>>> On Wed, Aug 30, 2017 at 11:31 AM Mingmin Xu <mingmxus@gmail.com
> >
> > >>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> Glad to see that 2.2.0 is coming. Can we include SQL feature in
> > >>>> next
> > >>>>>>>>> release? We're in the final stage and expect to merge back to
> > >>>> master
> > >>>>>>> this
> > >>>>>>>>> week.
> > >>>>>>>>>
> > >>>>>>>>> On Wed, Aug 30, 2017 at 11:27 AM, Reuven Lax
> > >>>>> <relax@google.com.invalid
> > >>>>>>>
> > >>>>>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>>> Now that Beam 2.1.0 has finally completed, I think we should
> cut
> > >>>>>> Beam
> > >>>>>>>>> 2.2.0
> > >>>>>>>>>> soon. I volunteer to coordinate this release.
> > >>>>>>>>>>
> > >>>>>>>>>> Are there any pending pull requests that people think should
> be
> > >>>>>> merged
> > >>>>>>>>>> before we cut 2.2.0? If so, please let me know soon, as I
> would
> > >>>>> like
> > >>>>>>> to
> > >>>>>>>>> cut
> > >>>>>>>>>> by Wednesday of next week.
> > >>>>>>>>>>
> > >>>>>>>>>> Thanks,
> > >>>>>>>>>>
> > >>>>>>>>>> Reuven
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> --
> > >>>>>>>>> ----
> > >>>>>>>>> Mingmin
> > >>>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
> > >> --
> > >> Jean-Baptiste Onofré
> > >> jbonofre@apache.org
> > >> http://blog.nanthrax.net
> > >> Talend - http://www.talend.com
> > >>
> > >
> >
> > --
> > Jean-Baptiste Onofré
> > jbonofre@apache.org
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
> >
>
Re: Beam 2.2.0 release
Posted by Eugene Kirpichov <ki...@google.com.INVALID>.
There are a lot of users including very large production customers who have
been asking specifically for the features that are in 2.2.0 (most of them
accumulated while 2.1.0 was being iterated on) - mostly I'm referring to
the vastly improved file IO - and they have been hesitant to use Beam at
HEAD in production. I think the slight unusualness of having a release
published soon after the previous release is a small price to pay for
helping those users :)
On Wed, Aug 30, 2017, 11:30 PM Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
> As we released 2.1.0 couple of weeks ago, it could sound weird to the
> users to
> do a 2.2.0 so fast. If we have a blocking issue, we can do a 2.1.1 If it's
> new
> features, why not having a release pace in October (2.2.0) ?
>
> Thoughts ?
>
> Regards
> JB
>
> On 08/31/2017 08:27 AM, Eugene Kirpichov wrote:
> > I'd suggest to do 2.2.0 as quickly as possible, and target 2.3.0 for
> > October. I don't see a reason to delay 2.2.0 until October: there's a
> huge
> > amount of features worth releasing between when 2.1.0 was cut and the
> > current HEAD.
> >
> > On Wed, Aug 30, 2017 at 10:18 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
> > wrote:
> >
> >> With a 2.2.0 in October, I think we can try to move forward on RedisIO.
> >>
> >> I'm now back from vacation and I will resume the work on this IO.
> >>
> >> Regards
> >> JB
> >>
> >> On 08/30/2017 11:27 PM, Eugene Kirpichov wrote:
> >>> RedisIO in 2.2.0 is very unlikely. There's still a lot of review
> >> remaining
> >>> last time I checked on the PR.
> >>>
> >>> On Wed, Aug 30, 2017 at 2:24 PM Vilhelm von Ehrenheim <
> >>> vonehrenheim@gmail.com> wrote:
> >>>
> >>>> Any chance to get the RedisIO in this release?
> >>>> [BEAM-1017] Add RedisIO #1687
> >>>>
> >>>> Its not my PR but ll be happy to assist if there is anything I can do
> to
> >>>> help.
> >>>>
> >>>> On 30 Aug 2017 22:46, "Daniel Ribeiro" <danielrb@squareup.com.invalid
> >
> >>>> wrote:
> >>>>
> >>>>> It would be great to get a bump on pubsub
> >>>>> <https://github.com/apache/beam/blob/master/pom.xml#L145>
> dependency.
> >> It
> >>>>> is
> >>>>> currently very outdated (v1-rev10-1.22.0, which was released over a
> >> year
> >>>>> ago
> >>>>> <http://repo1.maven.org/maven2/com/google/apis/google-
> >>>>> api-services-pubsub/v1-rev10-1.22.0/>
> >>>>> ).
> >>>>>
> >>>>>
> >>>>> On Wed, Aug 30, 2017 at 1:27 PM, Eugene Kirpichov <
> >>>>> kirpichov@google.com.invalid> wrote:
> >>>>>
> >>>>>> Thanks Ismael. I've marked these two issues for fix in 2.2.0.
> >>>> Definitely
> >>>>>> agree that at least the first one must be fixed.
> >>>>>>
> >>>>>> Here's the current burndown list
> >>>>>> https://issues.apache.org/jira/projects/BEAM/versions/12341044 - we
> >>>>> should
> >>>>>> clean it up.
> >>>>>>
> >>>>>> On Wed, Aug 30, 2017 at 1:20 PM Ismaël Mejía <ie...@gmail.com>
> >>>> wrote:
> >>>>>>
> >>>>>>> The current master has accumulated a good amount of nice features
> >>>>>>> since 2.1.0 so a new release is welcomed. I have two JIRAs/PR that
> I
> >>>>>>> think are important to check/solve before the cut:
> >>>>>>>
> >>>>>>> BEAM-2516 (this is a regression on the performance of Direct runner
> >>>> on
> >>>>>>> Java). We had never really defined if a performance regression is
> >>>>>>> critical to be a blocker. I executed WordCount with the
> kinglear.txt
> >>>>>>> (170KB) file in version 2.1.0 vs the current 2.2.0-SNAPSHOT and I
> >>>>>>> found that the execution time passed from 5s to 126s. So maybe we
> >>>> need
> >>>>>>> to review this one before the release. I can understand if others
> >>>>>>> consider this a minor issue because the Direct runner is not
> supposed
> >>>>>>> to be used for production, but this performance regression can
> cause
> >>>> a
> >>>>>>> bad impression for a casual user starting with Beam.
> >>>>>>>
> >>>>>>> BEAM-2790 (fix reading from Amazon S3 via HadoopFileSystem). I
> think
> >>>>>>> this one is a nice to have. I am not sure that I can tackle it for
> >>>> the
> >>>>>>> wednesday cut. I’m OOO until the beginning of next week, but maybe
> >>>>>>> someone else can take a look. In the worst case this is not a
> release
> >>>>>>> blocker but definitely a really nice fix to include.
> >>>>>>>
> >>>>>>>
> >>>>>>> On Wed, Aug 30, 2017 at 8:49 PM, Eugene Kirpichov
> >>>>>>> <ki...@google.com.invalid> wrote:
> >>>>>>>> I'd like to get the following PRs into 2.2.0:
> >>>>>>>>
> >>>>>>>> #3765 <https://github.com/apache/beam/pull/3765> [BEAM-2753
> >>>>>>>> <https://issues.apache.org/jira/browse/BEAM-2753>] Fixes
> >>>> translation
> >>>>>> of
> >>>>>>>> WriteFiles side inputs (important bugfix for DynamicDestinations
> in
> >>>>>>> files)
> >>>>>>>> #3725 <https://github.com/apache/beam/pull/3725> [BEAM-2827
> >>>>>>>> <https://issues.apache.org/jira/browse/BEAM-2827>] Introduces
> >>>>>>>> AvroIO.watchForNewFiles (parity for AvroIO with TextIO in a few
> >>>>>> important
> >>>>>>>> features)
> >>>>>>>> #3759 <https://github.com/apache/beam/pull/3759> [BEAM-2828
> >>>>>>>> <https://issues.apache.org/jira/browse/BEAM-2828>] Moves Match
> >>>> into
> >>>>>>>> FileIO.match()/matchAll() (to prevent releasing current
> >>>>>>>> Match.filepatterns() into 2.2.0 and then having to keep it under
> >>>> that
> >>>>>>> name)
> >>>>>>>>
> >>>>>>>> On Wed, Aug 30, 2017 at 11:31 AM Mingmin Xu <mi...@gmail.com>
> >>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Glad to see that 2.2.0 is coming. Can we include SQL feature in
> >>>> next
> >>>>>>>>> release? We're in the final stage and expect to merge back to
> >>>> master
> >>>>>>> this
> >>>>>>>>> week.
> >>>>>>>>>
> >>>>>>>>> On Wed, Aug 30, 2017 at 11:27 AM, Reuven Lax
> >>>>> <relax@google.com.invalid
> >>>>>>>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Now that Beam 2.1.0 has finally completed, I think we should cut
> >>>>>> Beam
> >>>>>>>>> 2.2.0
> >>>>>>>>>> soon. I volunteer to coordinate this release.
> >>>>>>>>>>
> >>>>>>>>>> Are there any pending pull requests that people think should be
> >>>>>> merged
> >>>>>>>>>> before we cut 2.2.0? If so, please let me know soon, as I would
> >>>>> like
> >>>>>>> to
> >>>>>>>>> cut
> >>>>>>>>>> by Wednesday of next week.
> >>>>>>>>>>
> >>>>>>>>>> Thanks,
> >>>>>>>>>>
> >>>>>>>>>> Reuven
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> ----
> >>>>>>>>> Mingmin
> >>>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> >> --
> >> Jean-Baptiste Onofré
> >> jbonofre@apache.org
> >> http://blog.nanthrax.net
> >> Talend - http://www.talend.com
> >>
> >
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
Re: Beam 2.2.0 release
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
As we released 2.1.0 couple of weeks ago, it could sound weird to the users to
do a 2.2.0 so fast. If we have a blocking issue, we can do a 2.1.1 If it's new
features, why not having a release pace in October (2.2.0) ?
Thoughts ?
Regards
JB
On 08/31/2017 08:27 AM, Eugene Kirpichov wrote:
> I'd suggest to do 2.2.0 as quickly as possible, and target 2.3.0 for
> October. I don't see a reason to delay 2.2.0 until October: there's a huge
> amount of features worth releasing between when 2.1.0 was cut and the
> current HEAD.
>
> On Wed, Aug 30, 2017 at 10:18 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
>> With a 2.2.0 in October, I think we can try to move forward on RedisIO.
>>
>> I'm now back from vacation and I will resume the work on this IO.
>>
>> Regards
>> JB
>>
>> On 08/30/2017 11:27 PM, Eugene Kirpichov wrote:
>>> RedisIO in 2.2.0 is very unlikely. There's still a lot of review
>> remaining
>>> last time I checked on the PR.
>>>
>>> On Wed, Aug 30, 2017 at 2:24 PM Vilhelm von Ehrenheim <
>>> vonehrenheim@gmail.com> wrote:
>>>
>>>> Any chance to get the RedisIO in this release?
>>>> [BEAM-1017] Add RedisIO #1687
>>>>
>>>> Its not my PR but ll be happy to assist if there is anything I can do to
>>>> help.
>>>>
>>>> On 30 Aug 2017 22:46, "Daniel Ribeiro" <da...@squareup.com.invalid>
>>>> wrote:
>>>>
>>>>> It would be great to get a bump on pubsub
>>>>> <https://github.com/apache/beam/blob/master/pom.xml#L145> dependency.
>> It
>>>>> is
>>>>> currently very outdated (v1-rev10-1.22.0, which was released over a
>> year
>>>>> ago
>>>>> <http://repo1.maven.org/maven2/com/google/apis/google-
>>>>> api-services-pubsub/v1-rev10-1.22.0/>
>>>>> ).
>>>>>
>>>>>
>>>>> On Wed, Aug 30, 2017 at 1:27 PM, Eugene Kirpichov <
>>>>> kirpichov@google.com.invalid> wrote:
>>>>>
>>>>>> Thanks Ismael. I've marked these two issues for fix in 2.2.0.
>>>> Definitely
>>>>>> agree that at least the first one must be fixed.
>>>>>>
>>>>>> Here's the current burndown list
>>>>>> https://issues.apache.org/jira/projects/BEAM/versions/12341044 - we
>>>>> should
>>>>>> clean it up.
>>>>>>
>>>>>> On Wed, Aug 30, 2017 at 1:20 PM Ismaël Mejía <ie...@gmail.com>
>>>> wrote:
>>>>>>
>>>>>>> The current master has accumulated a good amount of nice features
>>>>>>> since 2.1.0 so a new release is welcomed. I have two JIRAs/PR that I
>>>>>>> think are important to check/solve before the cut:
>>>>>>>
>>>>>>> BEAM-2516 (this is a regression on the performance of Direct runner
>>>> on
>>>>>>> Java). We had never really defined if a performance regression is
>>>>>>> critical to be a blocker. I executed WordCount with the kinglear.txt
>>>>>>> (170KB) file in version 2.1.0 vs the current 2.2.0-SNAPSHOT and I
>>>>>>> found that the execution time passed from 5s to 126s. So maybe we
>>>> need
>>>>>>> to review this one before the release. I can understand if others
>>>>>>> consider this a minor issue because the Direct runner is not supposed
>>>>>>> to be used for production, but this performance regression can cause
>>>> a
>>>>>>> bad impression for a casual user starting with Beam.
>>>>>>>
>>>>>>> BEAM-2790 (fix reading from Amazon S3 via HadoopFileSystem). I think
>>>>>>> this one is a nice to have. I am not sure that I can tackle it for
>>>> the
>>>>>>> wednesday cut. I’m OOO until the beginning of next week, but maybe
>>>>>>> someone else can take a look. In the worst case this is not a release
>>>>>>> blocker but definitely a really nice fix to include.
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Aug 30, 2017 at 8:49 PM, Eugene Kirpichov
>>>>>>> <ki...@google.com.invalid> wrote:
>>>>>>>> I'd like to get the following PRs into 2.2.0:
>>>>>>>>
>>>>>>>> #3765 <https://github.com/apache/beam/pull/3765> [BEAM-2753
>>>>>>>> <https://issues.apache.org/jira/browse/BEAM-2753>] Fixes
>>>> translation
>>>>>> of
>>>>>>>> WriteFiles side inputs (important bugfix for DynamicDestinations in
>>>>>>> files)
>>>>>>>> #3725 <https://github.com/apache/beam/pull/3725> [BEAM-2827
>>>>>>>> <https://issues.apache.org/jira/browse/BEAM-2827>] Introduces
>>>>>>>> AvroIO.watchForNewFiles (parity for AvroIO with TextIO in a few
>>>>>> important
>>>>>>>> features)
>>>>>>>> #3759 <https://github.com/apache/beam/pull/3759> [BEAM-2828
>>>>>>>> <https://issues.apache.org/jira/browse/BEAM-2828>] Moves Match
>>>> into
>>>>>>>> FileIO.match()/matchAll() (to prevent releasing current
>>>>>>>> Match.filepatterns() into 2.2.0 and then having to keep it under
>>>> that
>>>>>>> name)
>>>>>>>>
>>>>>>>> On Wed, Aug 30, 2017 at 11:31 AM Mingmin Xu <mi...@gmail.com>
>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Glad to see that 2.2.0 is coming. Can we include SQL feature in
>>>> next
>>>>>>>>> release? We're in the final stage and expect to merge back to
>>>> master
>>>>>>> this
>>>>>>>>> week.
>>>>>>>>>
>>>>>>>>> On Wed, Aug 30, 2017 at 11:27 AM, Reuven Lax
>>>>> <relax@google.com.invalid
>>>>>>>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Now that Beam 2.1.0 has finally completed, I think we should cut
>>>>>> Beam
>>>>>>>>> 2.2.0
>>>>>>>>>> soon. I volunteer to coordinate this release.
>>>>>>>>>>
>>>>>>>>>> Are there any pending pull requests that people think should be
>>>>>> merged
>>>>>>>>>> before we cut 2.2.0? If so, please let me know soon, as I would
>>>>> like
>>>>>>> to
>>>>>>>>> cut
>>>>>>>>>> by Wednesday of next week.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>>
>>>>>>>>>> Reuven
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> ----
>>>>>>>>> Mingmin
>>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>> --
>> Jean-Baptiste Onofré
>> jbonofre@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>>
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: Beam 2.2.0 release
Posted by Eugene Kirpichov <ki...@google.com.INVALID>.
I'd suggest to do 2.2.0 as quickly as possible, and target 2.3.0 for
October. I don't see a reason to delay 2.2.0 until October: there's a huge
amount of features worth releasing between when 2.1.0 was cut and the
current HEAD.
On Wed, Aug 30, 2017 at 10:18 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> With a 2.2.0 in October, I think we can try to move forward on RedisIO.
>
> I'm now back from vacation and I will resume the work on this IO.
>
> Regards
> JB
>
> On 08/30/2017 11:27 PM, Eugene Kirpichov wrote:
> > RedisIO in 2.2.0 is very unlikely. There's still a lot of review
> remaining
> > last time I checked on the PR.
> >
> > On Wed, Aug 30, 2017 at 2:24 PM Vilhelm von Ehrenheim <
> > vonehrenheim@gmail.com> wrote:
> >
> >> Any chance to get the RedisIO in this release?
> >> [BEAM-1017] Add RedisIO #1687
> >>
> >> Its not my PR but ll be happy to assist if there is anything I can do to
> >> help.
> >>
> >> On 30 Aug 2017 22:46, "Daniel Ribeiro" <da...@squareup.com.invalid>
> >> wrote:
> >>
> >>> It would be great to get a bump on pubsub
> >>> <https://github.com/apache/beam/blob/master/pom.xml#L145> dependency.
> It
> >>> is
> >>> currently very outdated (v1-rev10-1.22.0, which was released over a
> year
> >>> ago
> >>> <http://repo1.maven.org/maven2/com/google/apis/google-
> >>> api-services-pubsub/v1-rev10-1.22.0/>
> >>> ).
> >>>
> >>>
> >>> On Wed, Aug 30, 2017 at 1:27 PM, Eugene Kirpichov <
> >>> kirpichov@google.com.invalid> wrote:
> >>>
> >>>> Thanks Ismael. I've marked these two issues for fix in 2.2.0.
> >> Definitely
> >>>> agree that at least the first one must be fixed.
> >>>>
> >>>> Here's the current burndown list
> >>>> https://issues.apache.org/jira/projects/BEAM/versions/12341044 - we
> >>> should
> >>>> clean it up.
> >>>>
> >>>> On Wed, Aug 30, 2017 at 1:20 PM Ismaël Mejía <ie...@gmail.com>
> >> wrote:
> >>>>
> >>>>> The current master has accumulated a good amount of nice features
> >>>>> since 2.1.0 so a new release is welcomed. I have two JIRAs/PR that I
> >>>>> think are important to check/solve before the cut:
> >>>>>
> >>>>> BEAM-2516 (this is a regression on the performance of Direct runner
> >> on
> >>>>> Java). We had never really defined if a performance regression is
> >>>>> critical to be a blocker. I executed WordCount with the kinglear.txt
> >>>>> (170KB) file in version 2.1.0 vs the current 2.2.0-SNAPSHOT and I
> >>>>> found that the execution time passed from 5s to 126s. So maybe we
> >> need
> >>>>> to review this one before the release. I can understand if others
> >>>>> consider this a minor issue because the Direct runner is not supposed
> >>>>> to be used for production, but this performance regression can cause
> >> a
> >>>>> bad impression for a casual user starting with Beam.
> >>>>>
> >>>>> BEAM-2790 (fix reading from Amazon S3 via HadoopFileSystem). I think
> >>>>> this one is a nice to have. I am not sure that I can tackle it for
> >> the
> >>>>> wednesday cut. I’m OOO until the beginning of next week, but maybe
> >>>>> someone else can take a look. In the worst case this is not a release
> >>>>> blocker but definitely a really nice fix to include.
> >>>>>
> >>>>>
> >>>>> On Wed, Aug 30, 2017 at 8:49 PM, Eugene Kirpichov
> >>>>> <ki...@google.com.invalid> wrote:
> >>>>>> I'd like to get the following PRs into 2.2.0:
> >>>>>>
> >>>>>> #3765 <https://github.com/apache/beam/pull/3765> [BEAM-2753
> >>>>>> <https://issues.apache.org/jira/browse/BEAM-2753>] Fixes
> >> translation
> >>>> of
> >>>>>> WriteFiles side inputs (important bugfix for DynamicDestinations in
> >>>>> files)
> >>>>>> #3725 <https://github.com/apache/beam/pull/3725> [BEAM-2827
> >>>>>> <https://issues.apache.org/jira/browse/BEAM-2827>] Introduces
> >>>>>> AvroIO.watchForNewFiles (parity for AvroIO with TextIO in a few
> >>>> important
> >>>>>> features)
> >>>>>> #3759 <https://github.com/apache/beam/pull/3759> [BEAM-2828
> >>>>>> <https://issues.apache.org/jira/browse/BEAM-2828>] Moves Match
> >> into
> >>>>>> FileIO.match()/matchAll() (to prevent releasing current
> >>>>>> Match.filepatterns() into 2.2.0 and then having to keep it under
> >> that
> >>>>> name)
> >>>>>>
> >>>>>> On Wed, Aug 30, 2017 at 11:31 AM Mingmin Xu <mi...@gmail.com>
> >>>> wrote:
> >>>>>>
> >>>>>>> Glad to see that 2.2.0 is coming. Can we include SQL feature in
> >> next
> >>>>>>> release? We're in the final stage and expect to merge back to
> >> master
> >>>>> this
> >>>>>>> week.
> >>>>>>>
> >>>>>>> On Wed, Aug 30, 2017 at 11:27 AM, Reuven Lax
> >>> <relax@google.com.invalid
> >>>>>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Now that Beam 2.1.0 has finally completed, I think we should cut
> >>>> Beam
> >>>>>>> 2.2.0
> >>>>>>>> soon. I volunteer to coordinate this release.
> >>>>>>>>
> >>>>>>>> Are there any pending pull requests that people think should be
> >>>> merged
> >>>>>>>> before we cut 2.2.0? If so, please let me know soon, as I would
> >>> like
> >>>>> to
> >>>>>>> cut
> >>>>>>>> by Wednesday of next week.
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>>
> >>>>>>>> Reuven
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> --
> >>>>>>> ----
> >>>>>>> Mingmin
> >>>>>>>
> >>>>>
> >>>>
> >>>
> >>
> >
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
Re: Beam 2.2.0 release
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
With a 2.2.0 in October, I think we can try to move forward on RedisIO.
I'm now back from vacation and I will resume the work on this IO.
Regards
JB
On 08/30/2017 11:27 PM, Eugene Kirpichov wrote:
> RedisIO in 2.2.0 is very unlikely. There's still a lot of review remaining
> last time I checked on the PR.
>
> On Wed, Aug 30, 2017 at 2:24 PM Vilhelm von Ehrenheim <
> vonehrenheim@gmail.com> wrote:
>
>> Any chance to get the RedisIO in this release?
>> [BEAM-1017] Add RedisIO #1687
>>
>> Its not my PR but ll be happy to assist if there is anything I can do to
>> help.
>>
>> On 30 Aug 2017 22:46, "Daniel Ribeiro" <da...@squareup.com.invalid>
>> wrote:
>>
>>> It would be great to get a bump on pubsub
>>> <https://github.com/apache/beam/blob/master/pom.xml#L145> dependency. It
>>> is
>>> currently very outdated (v1-rev10-1.22.0, which was released over a year
>>> ago
>>> <http://repo1.maven.org/maven2/com/google/apis/google-
>>> api-services-pubsub/v1-rev10-1.22.0/>
>>> ).
>>>
>>>
>>> On Wed, Aug 30, 2017 at 1:27 PM, Eugene Kirpichov <
>>> kirpichov@google.com.invalid> wrote:
>>>
>>>> Thanks Ismael. I've marked these two issues for fix in 2.2.0.
>> Definitely
>>>> agree that at least the first one must be fixed.
>>>>
>>>> Here's the current burndown list
>>>> https://issues.apache.org/jira/projects/BEAM/versions/12341044 - we
>>> should
>>>> clean it up.
>>>>
>>>> On Wed, Aug 30, 2017 at 1:20 PM Ismaël Mejía <ie...@gmail.com>
>> wrote:
>>>>
>>>>> The current master has accumulated a good amount of nice features
>>>>> since 2.1.0 so a new release is welcomed. I have two JIRAs/PR that I
>>>>> think are important to check/solve before the cut:
>>>>>
>>>>> BEAM-2516 (this is a regression on the performance of Direct runner
>> on
>>>>> Java). We had never really defined if a performance regression is
>>>>> critical to be a blocker. I executed WordCount with the kinglear.txt
>>>>> (170KB) file in version 2.1.0 vs the current 2.2.0-SNAPSHOT and I
>>>>> found that the execution time passed from 5s to 126s. So maybe we
>> need
>>>>> to review this one before the release. I can understand if others
>>>>> consider this a minor issue because the Direct runner is not supposed
>>>>> to be used for production, but this performance regression can cause
>> a
>>>>> bad impression for a casual user starting with Beam.
>>>>>
>>>>> BEAM-2790 (fix reading from Amazon S3 via HadoopFileSystem). I think
>>>>> this one is a nice to have. I am not sure that I can tackle it for
>> the
>>>>> wednesday cut. I’m OOO until the beginning of next week, but maybe
>>>>> someone else can take a look. In the worst case this is not a release
>>>>> blocker but definitely a really nice fix to include.
>>>>>
>>>>>
>>>>> On Wed, Aug 30, 2017 at 8:49 PM, Eugene Kirpichov
>>>>> <ki...@google.com.invalid> wrote:
>>>>>> I'd like to get the following PRs into 2.2.0:
>>>>>>
>>>>>> #3765 <https://github.com/apache/beam/pull/3765> [BEAM-2753
>>>>>> <https://issues.apache.org/jira/browse/BEAM-2753>] Fixes
>> translation
>>>> of
>>>>>> WriteFiles side inputs (important bugfix for DynamicDestinations in
>>>>> files)
>>>>>> #3725 <https://github.com/apache/beam/pull/3725> [BEAM-2827
>>>>>> <https://issues.apache.org/jira/browse/BEAM-2827>] Introduces
>>>>>> AvroIO.watchForNewFiles (parity for AvroIO with TextIO in a few
>>>> important
>>>>>> features)
>>>>>> #3759 <https://github.com/apache/beam/pull/3759> [BEAM-2828
>>>>>> <https://issues.apache.org/jira/browse/BEAM-2828>] Moves Match
>> into
>>>>>> FileIO.match()/matchAll() (to prevent releasing current
>>>>>> Match.filepatterns() into 2.2.0 and then having to keep it under
>> that
>>>>> name)
>>>>>>
>>>>>> On Wed, Aug 30, 2017 at 11:31 AM Mingmin Xu <mi...@gmail.com>
>>>> wrote:
>>>>>>
>>>>>>> Glad to see that 2.2.0 is coming. Can we include SQL feature in
>> next
>>>>>>> release? We're in the final stage and expect to merge back to
>> master
>>>>> this
>>>>>>> week.
>>>>>>>
>>>>>>> On Wed, Aug 30, 2017 at 11:27 AM, Reuven Lax
>>> <relax@google.com.invalid
>>>>>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Now that Beam 2.1.0 has finally completed, I think we should cut
>>>> Beam
>>>>>>> 2.2.0
>>>>>>>> soon. I volunteer to coordinate this release.
>>>>>>>>
>>>>>>>> Are there any pending pull requests that people think should be
>>>> merged
>>>>>>>> before we cut 2.2.0? If so, please let me know soon, as I would
>>> like
>>>>> to
>>>>>>> cut
>>>>>>>> by Wednesday of next week.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> Reuven
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> ----
>>>>>>> Mingmin
>>>>>>>
>>>>>
>>>>
>>>
>>
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: Beam 2.2.0 release
Posted by Eugene Kirpichov <ki...@google.com.INVALID>.
RedisIO in 2.2.0 is very unlikely. There's still a lot of review remaining
last time I checked on the PR.
On Wed, Aug 30, 2017 at 2:24 PM Vilhelm von Ehrenheim <
vonehrenheim@gmail.com> wrote:
> Any chance to get the RedisIO in this release?
> [BEAM-1017] Add RedisIO #1687
>
> Its not my PR but ll be happy to assist if there is anything I can do to
> help.
>
> On 30 Aug 2017 22:46, "Daniel Ribeiro" <da...@squareup.com.invalid>
> wrote:
>
> > It would be great to get a bump on pubsub
> > <https://github.com/apache/beam/blob/master/pom.xml#L145> dependency. It
> > is
> > currently very outdated (v1-rev10-1.22.0, which was released over a year
> > ago
> > <http://repo1.maven.org/maven2/com/google/apis/google-
> > api-services-pubsub/v1-rev10-1.22.0/>
> > ).
> >
> >
> > On Wed, Aug 30, 2017 at 1:27 PM, Eugene Kirpichov <
> > kirpichov@google.com.invalid> wrote:
> >
> > > Thanks Ismael. I've marked these two issues for fix in 2.2.0.
> Definitely
> > > agree that at least the first one must be fixed.
> > >
> > > Here's the current burndown list
> > > https://issues.apache.org/jira/projects/BEAM/versions/12341044 - we
> > should
> > > clean it up.
> > >
> > > On Wed, Aug 30, 2017 at 1:20 PM Ismaël Mejía <ie...@gmail.com>
> wrote:
> > >
> > > > The current master has accumulated a good amount of nice features
> > > > since 2.1.0 so a new release is welcomed. I have two JIRAs/PR that I
> > > > think are important to check/solve before the cut:
> > > >
> > > > BEAM-2516 (this is a regression on the performance of Direct runner
> on
> > > > Java). We had never really defined if a performance regression is
> > > > critical to be a blocker. I executed WordCount with the kinglear.txt
> > > > (170KB) file in version 2.1.0 vs the current 2.2.0-SNAPSHOT and I
> > > > found that the execution time passed from 5s to 126s. So maybe we
> need
> > > > to review this one before the release. I can understand if others
> > > > consider this a minor issue because the Direct runner is not supposed
> > > > to be used for production, but this performance regression can cause
> a
> > > > bad impression for a casual user starting with Beam.
> > > >
> > > > BEAM-2790 (fix reading from Amazon S3 via HadoopFileSystem). I think
> > > > this one is a nice to have. I am not sure that I can tackle it for
> the
> > > > wednesday cut. I’m OOO until the beginning of next week, but maybe
> > > > someone else can take a look. In the worst case this is not a release
> > > > blocker but definitely a really nice fix to include.
> > > >
> > > >
> > > > On Wed, Aug 30, 2017 at 8:49 PM, Eugene Kirpichov
> > > > <ki...@google.com.invalid> wrote:
> > > > > I'd like to get the following PRs into 2.2.0:
> > > > >
> > > > > #3765 <https://github.com/apache/beam/pull/3765> [BEAM-2753
> > > > > <https://issues.apache.org/jira/browse/BEAM-2753>] Fixes
> translation
> > > of
> > > > > WriteFiles side inputs (important bugfix for DynamicDestinations in
> > > > files)
> > > > > #3725 <https://github.com/apache/beam/pull/3725> [BEAM-2827
> > > > > <https://issues.apache.org/jira/browse/BEAM-2827>] Introduces
> > > > > AvroIO.watchForNewFiles (parity for AvroIO with TextIO in a few
> > > important
> > > > > features)
> > > > > #3759 <https://github.com/apache/beam/pull/3759> [BEAM-2828
> > > > > <https://issues.apache.org/jira/browse/BEAM-2828>] Moves Match
> into
> > > > > FileIO.match()/matchAll() (to prevent releasing current
> > > > > Match.filepatterns() into 2.2.0 and then having to keep it under
> that
> > > > name)
> > > > >
> > > > > On Wed, Aug 30, 2017 at 11:31 AM Mingmin Xu <mi...@gmail.com>
> > > wrote:
> > > > >
> > > > >> Glad to see that 2.2.0 is coming. Can we include SQL feature in
> next
> > > > >> release? We're in the final stage and expect to merge back to
> master
> > > > this
> > > > >> week.
> > > > >>
> > > > >> On Wed, Aug 30, 2017 at 11:27 AM, Reuven Lax
> > <relax@google.com.invalid
> > > >
> > > > >> wrote:
> > > > >>
> > > > >> > Now that Beam 2.1.0 has finally completed, I think we should cut
> > > Beam
> > > > >> 2.2.0
> > > > >> > soon. I volunteer to coordinate this release.
> > > > >> >
> > > > >> > Are there any pending pull requests that people think should be
> > > merged
> > > > >> > before we cut 2.2.0? If so, please let me know soon, as I would
> > like
> > > > to
> > > > >> cut
> > > > >> > by Wednesday of next week.
> > > > >> >
> > > > >> > Thanks,
> > > > >> >
> > > > >> > Reuven
> > > > >> >
> > > > >>
> > > > >>
> > > > >>
> > > > >> --
> > > > >> ----
> > > > >> Mingmin
> > > > >>
> > > >
> > >
> >
>
Re: Beam 2.2.0 release
Posted by Vilhelm von Ehrenheim <vo...@gmail.com>.
Any chance to get the RedisIO in this release?
[BEAM-1017] Add RedisIO #1687
Its not my PR but ll be happy to assist if there is anything I can do to
help.
On 30 Aug 2017 22:46, "Daniel Ribeiro" <da...@squareup.com.invalid>
wrote:
> It would be great to get a bump on pubsub
> <https://github.com/apache/beam/blob/master/pom.xml#L145> dependency. It
> is
> currently very outdated (v1-rev10-1.22.0, which was released over a year
> ago
> <http://repo1.maven.org/maven2/com/google/apis/google-
> api-services-pubsub/v1-rev10-1.22.0/>
> ).
>
>
> On Wed, Aug 30, 2017 at 1:27 PM, Eugene Kirpichov <
> kirpichov@google.com.invalid> wrote:
>
> > Thanks Ismael. I've marked these two issues for fix in 2.2.0. Definitely
> > agree that at least the first one must be fixed.
> >
> > Here's the current burndown list
> > https://issues.apache.org/jira/projects/BEAM/versions/12341044 - we
> should
> > clean it up.
> >
> > On Wed, Aug 30, 2017 at 1:20 PM Ismaël Mejía <ie...@gmail.com> wrote:
> >
> > > The current master has accumulated a good amount of nice features
> > > since 2.1.0 so a new release is welcomed. I have two JIRAs/PR that I
> > > think are important to check/solve before the cut:
> > >
> > > BEAM-2516 (this is a regression on the performance of Direct runner on
> > > Java). We had never really defined if a performance regression is
> > > critical to be a blocker. I executed WordCount with the kinglear.txt
> > > (170KB) file in version 2.1.0 vs the current 2.2.0-SNAPSHOT and I
> > > found that the execution time passed from 5s to 126s. So maybe we need
> > > to review this one before the release. I can understand if others
> > > consider this a minor issue because the Direct runner is not supposed
> > > to be used for production, but this performance regression can cause a
> > > bad impression for a casual user starting with Beam.
> > >
> > > BEAM-2790 (fix reading from Amazon S3 via HadoopFileSystem). I think
> > > this one is a nice to have. I am not sure that I can tackle it for the
> > > wednesday cut. I’m OOO until the beginning of next week, but maybe
> > > someone else can take a look. In the worst case this is not a release
> > > blocker but definitely a really nice fix to include.
> > >
> > >
> > > On Wed, Aug 30, 2017 at 8:49 PM, Eugene Kirpichov
> > > <ki...@google.com.invalid> wrote:
> > > > I'd like to get the following PRs into 2.2.0:
> > > >
> > > > #3765 <https://github.com/apache/beam/pull/3765> [BEAM-2753
> > > > <https://issues.apache.org/jira/browse/BEAM-2753>] Fixes translation
> > of
> > > > WriteFiles side inputs (important bugfix for DynamicDestinations in
> > > files)
> > > > #3725 <https://github.com/apache/beam/pull/3725> [BEAM-2827
> > > > <https://issues.apache.org/jira/browse/BEAM-2827>] Introduces
> > > > AvroIO.watchForNewFiles (parity for AvroIO with TextIO in a few
> > important
> > > > features)
> > > > #3759 <https://github.com/apache/beam/pull/3759> [BEAM-2828
> > > > <https://issues.apache.org/jira/browse/BEAM-2828>] Moves Match into
> > > > FileIO.match()/matchAll() (to prevent releasing current
> > > > Match.filepatterns() into 2.2.0 and then having to keep it under that
> > > name)
> > > >
> > > > On Wed, Aug 30, 2017 at 11:31 AM Mingmin Xu <mi...@gmail.com>
> > wrote:
> > > >
> > > >> Glad to see that 2.2.0 is coming. Can we include SQL feature in next
> > > >> release? We're in the final stage and expect to merge back to master
> > > this
> > > >> week.
> > > >>
> > > >> On Wed, Aug 30, 2017 at 11:27 AM, Reuven Lax
> <relax@google.com.invalid
> > >
> > > >> wrote:
> > > >>
> > > >> > Now that Beam 2.1.0 has finally completed, I think we should cut
> > Beam
> > > >> 2.2.0
> > > >> > soon. I volunteer to coordinate this release.
> > > >> >
> > > >> > Are there any pending pull requests that people think should be
> > merged
> > > >> > before we cut 2.2.0? If so, please let me know soon, as I would
> like
> > > to
> > > >> cut
> > > >> > by Wednesday of next week.
> > > >> >
> > > >> > Thanks,
> > > >> >
> > > >> > Reuven
> > > >> >
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> ----
> > > >> Mingmin
> > > >>
> > >
> >
>
Re: Beam 2.2.0 release
Posted by Daniel Ribeiro <da...@squareup.com.INVALID>.
It would be great to get a bump on pubsub
<https://github.com/apache/beam/blob/master/pom.xml#L145> dependency. It is
currently very outdated (v1-rev10-1.22.0, which was released over a year ago
<http://repo1.maven.org/maven2/com/google/apis/google-api-services-pubsub/v1-rev10-1.22.0/>
).
On Wed, Aug 30, 2017 at 1:27 PM, Eugene Kirpichov <
kirpichov@google.com.invalid> wrote:
> Thanks Ismael. I've marked these two issues for fix in 2.2.0. Definitely
> agree that at least the first one must be fixed.
>
> Here's the current burndown list
> https://issues.apache.org/jira/projects/BEAM/versions/12341044 - we should
> clean it up.
>
> On Wed, Aug 30, 2017 at 1:20 PM Ismaël Mejía <ie...@gmail.com> wrote:
>
> > The current master has accumulated a good amount of nice features
> > since 2.1.0 so a new release is welcomed. I have two JIRAs/PR that I
> > think are important to check/solve before the cut:
> >
> > BEAM-2516 (this is a regression on the performance of Direct runner on
> > Java). We had never really defined if a performance regression is
> > critical to be a blocker. I executed WordCount with the kinglear.txt
> > (170KB) file in version 2.1.0 vs the current 2.2.0-SNAPSHOT and I
> > found that the execution time passed from 5s to 126s. So maybe we need
> > to review this one before the release. I can understand if others
> > consider this a minor issue because the Direct runner is not supposed
> > to be used for production, but this performance regression can cause a
> > bad impression for a casual user starting with Beam.
> >
> > BEAM-2790 (fix reading from Amazon S3 via HadoopFileSystem). I think
> > this one is a nice to have. I am not sure that I can tackle it for the
> > wednesday cut. I’m OOO until the beginning of next week, but maybe
> > someone else can take a look. In the worst case this is not a release
> > blocker but definitely a really nice fix to include.
> >
> >
> > On Wed, Aug 30, 2017 at 8:49 PM, Eugene Kirpichov
> > <ki...@google.com.invalid> wrote:
> > > I'd like to get the following PRs into 2.2.0:
> > >
> > > #3765 <https://github.com/apache/beam/pull/3765> [BEAM-2753
> > > <https://issues.apache.org/jira/browse/BEAM-2753>] Fixes translation
> of
> > > WriteFiles side inputs (important bugfix for DynamicDestinations in
> > files)
> > > #3725 <https://github.com/apache/beam/pull/3725> [BEAM-2827
> > > <https://issues.apache.org/jira/browse/BEAM-2827>] Introduces
> > > AvroIO.watchForNewFiles (parity for AvroIO with TextIO in a few
> important
> > > features)
> > > #3759 <https://github.com/apache/beam/pull/3759> [BEAM-2828
> > > <https://issues.apache.org/jira/browse/BEAM-2828>] Moves Match into
> > > FileIO.match()/matchAll() (to prevent releasing current
> > > Match.filepatterns() into 2.2.0 and then having to keep it under that
> > name)
> > >
> > > On Wed, Aug 30, 2017 at 11:31 AM Mingmin Xu <mi...@gmail.com>
> wrote:
> > >
> > >> Glad to see that 2.2.0 is coming. Can we include SQL feature in next
> > >> release? We're in the final stage and expect to merge back to master
> > this
> > >> week.
> > >>
> > >> On Wed, Aug 30, 2017 at 11:27 AM, Reuven Lax <relax@google.com.invalid
> >
> > >> wrote:
> > >>
> > >> > Now that Beam 2.1.0 has finally completed, I think we should cut
> Beam
> > >> 2.2.0
> > >> > soon. I volunteer to coordinate this release.
> > >> >
> > >> > Are there any pending pull requests that people think should be
> merged
> > >> > before we cut 2.2.0? If so, please let me know soon, as I would like
> > to
> > >> cut
> > >> > by Wednesday of next week.
> > >> >
> > >> > Thanks,
> > >> >
> > >> > Reuven
> > >> >
> > >>
> > >>
> > >>
> > >> --
> > >> ----
> > >> Mingmin
> > >>
> >
>
Re: Beam 2.2.0 release
Posted by Eugene Kirpichov <ki...@google.com.INVALID>.
Thanks Ismael. I've marked these two issues for fix in 2.2.0. Definitely
agree that at least the first one must be fixed.
Here's the current burndown list
https://issues.apache.org/jira/projects/BEAM/versions/12341044 - we should
clean it up.
On Wed, Aug 30, 2017 at 1:20 PM Ismaël Mejía <ie...@gmail.com> wrote:
> The current master has accumulated a good amount of nice features
> since 2.1.0 so a new release is welcomed. I have two JIRAs/PR that I
> think are important to check/solve before the cut:
>
> BEAM-2516 (this is a regression on the performance of Direct runner on
> Java). We had never really defined if a performance regression is
> critical to be a blocker. I executed WordCount with the kinglear.txt
> (170KB) file in version 2.1.0 vs the current 2.2.0-SNAPSHOT and I
> found that the execution time passed from 5s to 126s. So maybe we need
> to review this one before the release. I can understand if others
> consider this a minor issue because the Direct runner is not supposed
> to be used for production, but this performance regression can cause a
> bad impression for a casual user starting with Beam.
>
> BEAM-2790 (fix reading from Amazon S3 via HadoopFileSystem). I think
> this one is a nice to have. I am not sure that I can tackle it for the
> wednesday cut. I’m OOO until the beginning of next week, but maybe
> someone else can take a look. In the worst case this is not a release
> blocker but definitely a really nice fix to include.
>
>
> On Wed, Aug 30, 2017 at 8:49 PM, Eugene Kirpichov
> <ki...@google.com.invalid> wrote:
> > I'd like to get the following PRs into 2.2.0:
> >
> > #3765 <https://github.com/apache/beam/pull/3765> [BEAM-2753
> > <https://issues.apache.org/jira/browse/BEAM-2753>] Fixes translation of
> > WriteFiles side inputs (important bugfix for DynamicDestinations in
> files)
> > #3725 <https://github.com/apache/beam/pull/3725> [BEAM-2827
> > <https://issues.apache.org/jira/browse/BEAM-2827>] Introduces
> > AvroIO.watchForNewFiles (parity for AvroIO with TextIO in a few important
> > features)
> > #3759 <https://github.com/apache/beam/pull/3759> [BEAM-2828
> > <https://issues.apache.org/jira/browse/BEAM-2828>] Moves Match into
> > FileIO.match()/matchAll() (to prevent releasing current
> > Match.filepatterns() into 2.2.0 and then having to keep it under that
> name)
> >
> > On Wed, Aug 30, 2017 at 11:31 AM Mingmin Xu <mi...@gmail.com> wrote:
> >
> >> Glad to see that 2.2.0 is coming. Can we include SQL feature in next
> >> release? We're in the final stage and expect to merge back to master
> this
> >> week.
> >>
> >> On Wed, Aug 30, 2017 at 11:27 AM, Reuven Lax <re...@google.com.invalid>
> >> wrote:
> >>
> >> > Now that Beam 2.1.0 has finally completed, I think we should cut Beam
> >> 2.2.0
> >> > soon. I volunteer to coordinate this release.
> >> >
> >> > Are there any pending pull requests that people think should be merged
> >> > before we cut 2.2.0? If so, please let me know soon, as I would like
> to
> >> cut
> >> > by Wednesday of next week.
> >> >
> >> > Thanks,
> >> >
> >> > Reuven
> >> >
> >>
> >>
> >>
> >> --
> >> ----
> >> Mingmin
> >>
>
Re: Beam 2.2.0 release
Posted by Ismaël Mejía <ie...@gmail.com>.
The current master has accumulated a good amount of nice features
since 2.1.0 so a new release is welcomed. I have two JIRAs/PR that I
think are important to check/solve before the cut:
BEAM-2516 (this is a regression on the performance of Direct runner on
Java). We had never really defined if a performance regression is
critical to be a blocker. I executed WordCount with the kinglear.txt
(170KB) file in version 2.1.0 vs the current 2.2.0-SNAPSHOT and I
found that the execution time passed from 5s to 126s. So maybe we need
to review this one before the release. I can understand if others
consider this a minor issue because the Direct runner is not supposed
to be used for production, but this performance regression can cause a
bad impression for a casual user starting with Beam.
BEAM-2790 (fix reading from Amazon S3 via HadoopFileSystem). I think
this one is a nice to have. I am not sure that I can tackle it for the
wednesday cut. I’m OOO until the beginning of next week, but maybe
someone else can take a look. In the worst case this is not a release
blocker but definitely a really nice fix to include.
On Wed, Aug 30, 2017 at 8:49 PM, Eugene Kirpichov
<ki...@google.com.invalid> wrote:
> I'd like to get the following PRs into 2.2.0:
>
> #3765 <https://github.com/apache/beam/pull/3765> [BEAM-2753
> <https://issues.apache.org/jira/browse/BEAM-2753>] Fixes translation of
> WriteFiles side inputs (important bugfix for DynamicDestinations in files)
> #3725 <https://github.com/apache/beam/pull/3725> [BEAM-2827
> <https://issues.apache.org/jira/browse/BEAM-2827>] Introduces
> AvroIO.watchForNewFiles (parity for AvroIO with TextIO in a few important
> features)
> #3759 <https://github.com/apache/beam/pull/3759> [BEAM-2828
> <https://issues.apache.org/jira/browse/BEAM-2828>] Moves Match into
> FileIO.match()/matchAll() (to prevent releasing current
> Match.filepatterns() into 2.2.0 and then having to keep it under that name)
>
> On Wed, Aug 30, 2017 at 11:31 AM Mingmin Xu <mi...@gmail.com> wrote:
>
>> Glad to see that 2.2.0 is coming. Can we include SQL feature in next
>> release? We're in the final stage and expect to merge back to master this
>> week.
>>
>> On Wed, Aug 30, 2017 at 11:27 AM, Reuven Lax <re...@google.com.invalid>
>> wrote:
>>
>> > Now that Beam 2.1.0 has finally completed, I think we should cut Beam
>> 2.2.0
>> > soon. I volunteer to coordinate this release.
>> >
>> > Are there any pending pull requests that people think should be merged
>> > before we cut 2.2.0? If so, please let me know soon, as I would like to
>> cut
>> > by Wednesday of next week.
>> >
>> > Thanks,
>> >
>> > Reuven
>> >
>>
>>
>>
>> --
>> ----
>> Mingmin
>>
Re: Beam 2.2.0 release
Posted by Eugene Kirpichov <ki...@google.com.INVALID>.
I'd like to get the following PRs into 2.2.0:
#3765 <https://github.com/apache/beam/pull/3765> [BEAM-2753
<https://issues.apache.org/jira/browse/BEAM-2753>] Fixes translation of
WriteFiles side inputs (important bugfix for DynamicDestinations in files)
#3725 <https://github.com/apache/beam/pull/3725> [BEAM-2827
<https://issues.apache.org/jira/browse/BEAM-2827>] Introduces
AvroIO.watchForNewFiles (parity for AvroIO with TextIO in a few important
features)
#3759 <https://github.com/apache/beam/pull/3759> [BEAM-2828
<https://issues.apache.org/jira/browse/BEAM-2828>] Moves Match into
FileIO.match()/matchAll() (to prevent releasing current
Match.filepatterns() into 2.2.0 and then having to keep it under that name)
On Wed, Aug 30, 2017 at 11:31 AM Mingmin Xu <mi...@gmail.com> wrote:
> Glad to see that 2.2.0 is coming. Can we include SQL feature in next
> release? We're in the final stage and expect to merge back to master this
> week.
>
> On Wed, Aug 30, 2017 at 11:27 AM, Reuven Lax <re...@google.com.invalid>
> wrote:
>
> > Now that Beam 2.1.0 has finally completed, I think we should cut Beam
> 2.2.0
> > soon. I volunteer to coordinate this release.
> >
> > Are there any pending pull requests that people think should be merged
> > before we cut 2.2.0? If so, please let me know soon, as I would like to
> cut
> > by Wednesday of next week.
> >
> > Thanks,
> >
> > Reuven
> >
>
>
>
> --
> ----
> Mingmin
>
Re: Beam 2.2.0 release
Posted by Mingmin Xu <mi...@gmail.com>.
Glad to see that 2.2.0 is coming. Can we include SQL feature in next
release? We're in the final stage and expect to merge back to master this
week.
On Wed, Aug 30, 2017 at 11:27 AM, Reuven Lax <re...@google.com.invalid>
wrote:
> Now that Beam 2.1.0 has finally completed, I think we should cut Beam 2.2.0
> soon. I volunteer to coordinate this release.
>
> Are there any pending pull requests that people think should be merged
> before we cut 2.2.0? If so, please let me know soon, as I would like to cut
> by Wednesday of next week.
>
> Thanks,
>
> Reuven
>
--
----
Mingmin
Re: Beam 2.2.0 release
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi Reuven,
sorry I missed the initial message in the thread.
Thanks to be volunteer for this release !
For the target deadline, I think October make sense. We have some new feature in
progress (M/R runner, Spark 2.x runner, etc). We can finalize it during
September and so, October is a good timing for 2.2.0 release IMHO.
Regards
JB
On 08/30/2017 08:27 PM, Reuven Lax wrote:
> Now that Beam 2.1.0 has finally completed, I think we should cut Beam 2.2.0
> soon. I volunteer to coordinate this release.
>
> Are there any pending pull requests that people think should be merged
> before we cut 2.2.0? If so, please let me know soon, as I would like to cut
> by Wednesday of next week.
>
> Thanks,
>
> Reuven
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com