You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Jean-Baptiste Onofré <jb...@nanthrax.net> on 2018/06/06 08:44:31 UTC
[VOTE] Apache Beam, version 2.5.0, release candidate #1
Hi everyone,
Please review and vote on the release candidate #1 for the version
2.5.0, as follows:
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)
NB: this is the first release using Gradle, so don't be too harsh ;) A
PR about the release guide will follow thanks to this release.
The complete staging area is available for your review, which includes:
* JIRA release notes [1],
* the official Apache source release to be deployed to dist.apache.org
[2], which is signed with the key with fingerprint C8282E76 [3],
* all artifacts to be deployed to the Maven Central Repository [4],
* source code tag "v2.5.0-RC1" [5],
* website pull request listing the release and publishing the API
reference manual [6].
* Java artifacts were built with Gradle 4.7 (wrapper) and OpenJDK/Oracle
JDK 1.8.0_172 (Oracle Corporation 25.172-b11).
* Python artifacts are deployed along with the source release to the
dist.apache.org [2].
The vote will be open for at least 72 hours. It is adopted by majority
approval, with at least 3 PMC affirmative votes.
Thanks,
JB
[1]
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12342847
[2] https://dist.apache.org/repos/dist/dev/beam/2.5.0/
[3] https://dist.apache.org/repos/dist/release/beam/KEYS
[4] https://repository.apache.org/content/repositories/orgapachebeam-1041/
[5] https://github.com/apache/beam/tree/v2.5.0-RC1
[6] https://github.com/apache/beam-site/pull/463
Re: [VOTE] Apache Beam, version 2.5.0, release candidate #1
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
I updated dist.apache.org dev with Python distribution.
Regards
JB
On 06/06/2018 18:19, Jean-Baptiste Onofré wrote:
> Hi Robert,
>
> sorry, I missed this step, let me add on dist.apache.org.
>
> Thanks for the catch and sorry about that !
>
> Regards
> JB
>
> On 06/06/2018 18:06, Robert Bradshaw wrote:
>> Thank you JB! Glad to see this finally rolling out. I don't see the
>> Python artifacts, did you mean to stage them
>> in https://dist.apache.org/repos/dist/dev/beam/2.5.0/? If you want help
>> building wheels, let me know.
>>
>>
>>
>> On Wed, Jun 6, 2018 at 1:50 AM Etienne Chauchot <echauchot@apache.org
>> <ma...@apache.org>> wrote:
>>
>> Thanks JB for all your work ! I believe doing the first gradle
>> release must have been hard.
>> I'll run Nexmark on the release and keep you posted.
>>
>> Best
>> Etienne
>>
>>
>> Le mercredi 06 juin 2018 à 10:44 +0200, Jean-Baptiste Onofré a écrit :
>>> Hi everyone,
>>>
>>> Please review and vote on the release candidate #1 for the version
>>> 2.5.0, as follows:
>>>
>>> [ ] +1, Approve the release
>>> [ ] -1, Do not approve the release (please provide specific comments)
>>>
>>> NB: this is the first release using Gradle, so don't be too harsh ;) A
>>> PR about the release guide will follow thanks to this release.
>>>
>>> The complete staging area is available for your review, which includes:
>>> * JIRA release notes [1],
>>> * the official Apache source release to be deployed to dist.apache.org <http://dist.apache.org>
>>> [2], which is signed with the key with fingerprint C8282E76 [3],
>>> * all artifacts to be deployed to the Maven Central Repository [4],
>>> * source code tag "v2.5.0-RC1" [5],
>>> * website pull request listing the release and publishing the API
>>> reference manual [6].
>>> * Java artifacts were built with Gradle 4.7 (wrapper) and OpenJDK/Oracle
>>> JDK 1.8.0_172 (Oracle Corporation 25.172-b11).
>>> * Python artifacts are deployed along with the source release to the
>>> dist.apache.org <http://dist.apache.org> [2].
>>>
>>> The vote will be open for at least 72 hours. It is adopted by majority
>>> approval, with at least 3 PMC affirmative votes.
>>>
>>> Thanks,
>>> JB
>>>
>>> [1]
>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12342847
>>> [2] https://dist.apache.org/repos/dist/dev/beam/2.5.0/
>>> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
>>> [4] https://repository.apache.org/content/repositories/orgapachebeam-1041/
>>> [5] https://github.com/apache/beam/tree/v2.5.0-RC1
>>> [6] https://github.com/apache/beam-site/pull/463
>>>
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: [VOTE] Apache Beam, version 2.5.0, release candidate #1
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi Robert,
sorry, I missed this step, let me add on dist.apache.org.
Thanks for the catch and sorry about that !
Regards
JB
On 06/06/2018 18:06, Robert Bradshaw wrote:
> Thank you JB! Glad to see this finally rolling out. I don't see the
> Python artifacts, did you mean to stage them
> in https://dist.apache.org/repos/dist/dev/beam/2.5.0/? If you want help
> building wheels, let me know.
>
>
>
> On Wed, Jun 6, 2018 at 1:50 AM Etienne Chauchot <echauchot@apache.org
> <ma...@apache.org>> wrote:
>
> Thanks JB for all your work ! I believe doing the first gradle
> release must have been hard.
> I'll run Nexmark on the release and keep you posted.
>
> Best
> Etienne
>
>
> Le mercredi 06 juin 2018 à 10:44 +0200, Jean-Baptiste Onofré a écrit :
>> Hi everyone,
>>
>> Please review and vote on the release candidate #1 for the version
>> 2.5.0, as follows:
>>
>> [ ] +1, Approve the release
>> [ ] -1, Do not approve the release (please provide specific comments)
>>
>> NB: this is the first release using Gradle, so don't be too harsh ;) A
>> PR about the release guide will follow thanks to this release.
>>
>> The complete staging area is available for your review, which includes:
>> * JIRA release notes [1],
>> * the official Apache source release to be deployed to dist.apache.org <http://dist.apache.org>
>> [2], which is signed with the key with fingerprint C8282E76 [3],
>> * all artifacts to be deployed to the Maven Central Repository [4],
>> * source code tag "v2.5.0-RC1" [5],
>> * website pull request listing the release and publishing the API
>> reference manual [6].
>> * Java artifacts were built with Gradle 4.7 (wrapper) and OpenJDK/Oracle
>> JDK 1.8.0_172 (Oracle Corporation 25.172-b11).
>> * Python artifacts are deployed along with the source release to the
>> dist.apache.org <http://dist.apache.org> [2].
>>
>> The vote will be open for at least 72 hours. It is adopted by majority
>> approval, with at least 3 PMC affirmative votes.
>>
>> Thanks,
>> JB
>>
>> [1]
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12342847
>> [2] https://dist.apache.org/repos/dist/dev/beam/2.5.0/
>> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
>> [4] https://repository.apache.org/content/repositories/orgapachebeam-1041/
>> [5] https://github.com/apache/beam/tree/v2.5.0-RC1
>> [6] https://github.com/apache/beam-site/pull/463
>>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: [VOTE] Apache Beam, version 2.5.0, release candidate #1
Posted by Robert Bradshaw <ro...@google.com>.
Thank you JB! Glad to see this finally rolling out. I don't see the Python
artifacts, did you mean to stage them in
https://dist.apache.org/repos/dist/dev/beam/2.5.0/? If you want help
building wheels, let me know.
On Wed, Jun 6, 2018 at 1:50 AM Etienne Chauchot <ec...@apache.org>
wrote:
> Thanks JB for all your work ! I believe doing the first gradle release
> must have been hard.
> I'll run Nexmark on the release and keep you posted.
>
> Best
> Etienne
>
>
> Le mercredi 06 juin 2018 à 10:44 +0200, Jean-Baptiste Onofré a écrit :
>
> Hi everyone,
>
>
> Please review and vote on the release candidate #1 for the version
>
> 2.5.0, as follows:
>
>
> [ ] +1, Approve the release
>
> [ ] -1, Do not approve the release (please provide specific comments)
>
>
> NB: this is the first release using Gradle, so don't be too harsh ;) A
>
> PR about the release guide will follow thanks to this release.
>
>
> The complete staging area is available for your review, which includes:
>
> * JIRA release notes [1],
>
> * the official Apache source release to be deployed to dist.apache.org
>
> [2], which is signed with the key with fingerprint C8282E76 [3],
>
> * all artifacts to be deployed to the Maven Central Repository [4],
>
> * source code tag "v2.5.0-RC1" [5],
>
> * website pull request listing the release and publishing the API
>
> reference manual [6].
>
> * Java artifacts were built with Gradle 4.7 (wrapper) and OpenJDK/Oracle
>
> JDK 1.8.0_172 (Oracle Corporation 25.172-b11).
>
> * Python artifacts are deployed along with the source release to the
>
> dist.apache.org [2].
>
>
> The vote will be open for at least 72 hours. It is adopted by majority
>
> approval, with at least 3 PMC affirmative votes.
>
>
> Thanks,
>
> JB
>
>
> [1]
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12342847
>
> [2] https://dist.apache.org/repos/dist/dev/beam/2.5.0/
>
> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
>
> [4] https://repository.apache.org/content/repositories/orgapachebeam-1041/
>
> [5] https://github.com/apache/beam/tree/v2.5.0-RC1
>
> [6] https://github.com/apache/beam-site/pull/463
>
>
>
Re: [VOTE] Apache Beam, version 2.5.0, release candidate #1
Posted by Lukasz Cwik <lc...@google.com>.
I have added the 2.5.0 tab to the validation spreadsheet[1], please mark
down which things you intend to validate for the release and update the
community on progress.
1:
https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=152451807
On Wed, Jun 6, 2018 at 1:51 PM Reuven Lax <re...@google.com> wrote:
> Agreed 💯! It's not ready being the first to try something. Thank you so
> much for helping blaze the way!
>
> Reuven
>
> On Wed, Jun 6, 2018, 11:50 AM Etienne Chauchot <ec...@apache.org>
> wrote:
>
>> Thanks JB for all your work ! I believe doing the first gradle release
>> must have been hard.
>> I'll run Nexmark on the release and keep you posted.
>>
>> Best
>> Etienne
>>
>>
>> Le mercredi 06 juin 2018 à 10:44 +0200, Jean-Baptiste Onofré a écrit :
>>
>> Hi everyone,
>>
>>
>> Please review and vote on the release candidate #1 for the version
>>
>> 2.5.0, as follows:
>>
>>
>> [ ] +1, Approve the release
>>
>> [ ] -1, Do not approve the release (please provide specific comments)
>>
>>
>> NB: this is the first release using Gradle, so don't be too harsh ;) A
>>
>> PR about the release guide will follow thanks to this release.
>>
>>
>> The complete staging area is available for your review, which includes:
>>
>> * JIRA release notes [1],
>>
>> * the official Apache source release to be deployed to dist.apache.org
>>
>> [2], which is signed with the key with fingerprint C8282E76 [3],
>>
>> * all artifacts to be deployed to the Maven Central Repository [4],
>>
>> * source code tag "v2.5.0-RC1" [5],
>>
>> * website pull request listing the release and publishing the API
>>
>> reference manual [6].
>>
>> * Java artifacts were built with Gradle 4.7 (wrapper) and OpenJDK/Oracle
>>
>> JDK 1.8.0_172 (Oracle Corporation 25.172-b11).
>>
>> * Python artifacts are deployed along with the source release to the
>>
>> dist.apache.org [2].
>>
>>
>> The vote will be open for at least 72 hours. It is adopted by majority
>>
>> approval, with at least 3 PMC affirmative votes.
>>
>>
>> Thanks,
>>
>> JB
>>
>>
>> [1]
>>
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12342847
>>
>> [2] https://dist.apache.org/repos/dist/dev/beam/2.5.0/
>>
>> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
>>
>> [4] https://repository.apache.org/content/repositories/orgapachebeam-1041/
>>
>> [5] https://github.com/apache/beam/tree/v2.5.0-RC1
>>
>> [6] https://github.com/apache/beam-site/pull/463
>>
>>
>>
Re: [VOTE] Apache Beam, version 2.5.0, release candidate #1
Posted by Reuven Lax <re...@google.com>.
Agreed 💯! It's not ready being the first to try something. Thank you so
much for helping blaze the way!
Reuven
On Wed, Jun 6, 2018, 11:50 AM Etienne Chauchot <ec...@apache.org> wrote:
> Thanks JB for all your work ! I believe doing the first gradle release
> must have been hard.
> I'll run Nexmark on the release and keep you posted.
>
> Best
> Etienne
>
>
> Le mercredi 06 juin 2018 à 10:44 +0200, Jean-Baptiste Onofré a écrit :
>
> Hi everyone,
>
>
> Please review and vote on the release candidate #1 for the version
>
> 2.5.0, as follows:
>
>
> [ ] +1, Approve the release
>
> [ ] -1, Do not approve the release (please provide specific comments)
>
>
> NB: this is the first release using Gradle, so don't be too harsh ;) A
>
> PR about the release guide will follow thanks to this release.
>
>
> The complete staging area is available for your review, which includes:
>
> * JIRA release notes [1],
>
> * the official Apache source release to be deployed to dist.apache.org
>
> [2], which is signed with the key with fingerprint C8282E76 [3],
>
> * all artifacts to be deployed to the Maven Central Repository [4],
>
> * source code tag "v2.5.0-RC1" [5],
>
> * website pull request listing the release and publishing the API
>
> reference manual [6].
>
> * Java artifacts were built with Gradle 4.7 (wrapper) and OpenJDK/Oracle
>
> JDK 1.8.0_172 (Oracle Corporation 25.172-b11).
>
> * Python artifacts are deployed along with the source release to the
>
> dist.apache.org [2].
>
>
> The vote will be open for at least 72 hours. It is adopted by majority
>
> approval, with at least 3 PMC affirmative votes.
>
>
> Thanks,
>
> JB
>
>
> [1]
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12342847
>
> [2] https://dist.apache.org/repos/dist/dev/beam/2.5.0/
>
> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
>
> [4] https://repository.apache.org/content/repositories/orgapachebeam-1041/
>
> [5] https://github.com/apache/beam/tree/v2.5.0-RC1
>
> [6] https://github.com/apache/beam-site/pull/463
>
>
>
Re: [VOTE] Apache Beam, version 2.5.0, release candidate #1
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi,
no problem, I can cut RC2 as soon as the cherry pick is done.
Thanks for catching up !
Please let me know when the cherry pick is done, or you can do the PR
and I will do it, up to you.
Regards
JB
On 12/06/2018 04:02, Udi Meiri wrote:
> Another bug: reading from PubSub with_attributes=True is broken on
> Python with Dataflow.
> https://issues.apache.org/jira/browse/BEAM-4536
>
> JB, I'm making a PR that removes this keyword and I'd like to propose it
> as a cherrypick to 2.5.0.
> (feature should be fixed in the next release)
>
> On Mon, Jun 11, 2018 at 6:19 PM Chamikara Jayalath <chamikara@google.com
> <ma...@google.com>> wrote:
>
> FYI: looks like Python tests are failing for Windows. JIRA
> is https://issues.apache.org/jira/browse/BEAM-4535.
>
> I don't think this is a release blocker but this should probably go
> in release notes (for any user that tries to run tests on Python
> source build). And we should try to incorporate a fix if we happen
> to cut another release candidate for some reason.
>
> Thanks,
> Cham
>
> On Mon, Jun 11, 2018 at 5:46 PM Pablo Estrada <pabloem@google.com
> <ma...@google.com>> wrote:
>
> Thanks everyone who has pitched in to validate the release!
>
> Boyuan Zhang and I have also run a few pipelines, and verified
> that they work properly (see release validation spreadsheet[1]).
>
> We have also found that the Game Stats pipeline is failing in
> Python Streaming Dataflow. I have filed BEAM-4534[2]. This is
> not a blocker, since Python streaming is not yet fully supported.
>
> It seems that the uploaded artifacts look good.
>
> We have noticed that the Python artifacts are still missing
> Python wheel files (compare [3] and [4]). JB, could you please
> add the wheel files? Boyuan and I can try to help you prepare
> them / upload them if necessary. Please let us know.
>
> Thanks again!
> -P.
>
> [1] https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=152451807
> [2] https://issues.apache.org/jira/browse/BEAM-4534
> [3] https://dist.apache.org/repos/dist/dev/beam/2.4.0/
> [4] https://dist.apache.org/repos/dist/dev/beam/2.5.0/
>
> On Mon, Jun 11, 2018 at 12:37 PM Alan Myrvold
> <amyrvold@google.com <ma...@google.com>> wrote:
>
> +1 (non-binding)
>
> tested some of the quickstarts
>
> On Sun, Jun 10, 2018 at 1:39 AM Tim
> <timrobertson100@gmail.com
> <ma...@gmail.com>> wrote:
>
> Tested by our team:
> - mvn inclusion
> - Avro, ES, Hadoop IF IO
> - Pipelines run on Spark (Cloudera 5.12.0 YARN cluster)
> - Reviewed release notes
>
> +1
>
> Thanks also to everyone who helped get over the gradle
> hurdle and in particular to JB.
>
> Tim
>
> > On 9 Jun 2018, at 05:56, Jean-Baptiste Onofré
> <jb@nanthrax.net <ma...@nanthrax.net>> wrote:
> >
> > No problem Pablo.
> >
> > The vote period is a minimum, it can be extended as
> requested or if we
> > don't have the minimum of 3 binding votes.
> >
> > Regards
> > JB
> >
> >> On 09/06/2018 01:54, Pablo Estrada wrote:
> >> Hello all,
> >> I'd like to request an extension of the voting period
> until Monday
> >> evening (US time, so later in other geographical
> regions). This is
> >> because we were only now able to publish Dataflow
> Workers, and have not
> >> had the chance to run release validation tests on
> them. The extension
> >> will allow us to validate and vote by Monday.
> >> Is this acceptable to the community?
> >>
> >> Best
> >> -P.
> >>
> >> On Fri, Jun 8, 2018 at 6:20 AM Alexey Romanenko
> >> <aromanenko.dev@gmail.com
> <ma...@gmail.com>
> <mailto:aromanenko.dev@gmail.com
> <ma...@gmail.com>>> wrote:
> >>
> >> Thank you JB for your work!
> >>
> >> I tested running simple streaming (/KafkaIO/) and
> batch (/TextIO /
> >> HDFS/) pipelines with SparkRunner on YARN cluster
> - it works fine.
> >>
> >> WBR,
> >> Alexey
> >>
> >>
> >>> On 8 Jun 2018, at 10:00, Etienne Chauchot
> <echauchot@apache.org <ma...@apache.org>
> >>> <mailto:echauchot@apache.org
> <ma...@apache.org>>> wrote:
> >>>
> >>> I forgot to vote:
> >>> +1 (non binding).
> >>> What I tested:
> >>> - no functional or performance regression
> comparing to v2.4
> >>> - dependencies in the poms are ok
> >>>
> >>> Etienne
> >>>> Le vendredi 08 juin 2018 à 08:27 +0200, Romain
> Manni-Bucau a écrit :
> >>>> +1 (non-binding), mainstream usage is not broken
> by the pom
> >>>> changes and runtime has no known regression
> compared to the 2.4.0
> >>>>
> >>>> (side note: kudo to JB for this build tool
> change release, I know
> >>>> how it can hurt ;))
> >>>>
> >>>> Romain Manni-Bucau
> >>>> @rmannibucau <https://twitter.com/rmannibucau>
> | Blog
> >>>> <https://rmannibucau.metawerx.net/> | Old Blog
> >>>> <http://rmannibucau.wordpress.com/> | Github
> >>>> <https://github.com/rmannibucau> | LinkedIn
> >>>> <https://www.linkedin.com/in/rmannibucau> | Book
> >>>>
> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
> >>>>
> >>>>
> >>>> Le jeu. 7 juin 2018 à 16:17, Jean-Baptiste Onofré
> >>>> <jb@nanthrax.net <ma...@nanthrax.net>
> <mailto:jb@nanthrax.net <ma...@nanthrax.net>>> a écrit :
> >>>>> Thanks for the details Etienne !
> >>>>>
> >>>>> The good news is that the artifacts seem OK and
> the overall Nexmark
> >>>>> results are consistent with the 2.4.0 release ones.
> >>>>>
> >>>>> I'm starting a complete review using the
> beam-samples as well.
> >>>>>
> >>>>> Regards
> >>>>> JB
> >>>>>
> >>>>>> On 07/06/2018 16:14, Etienne Chauchot wrote:
> >>>>>> Hi,
> >>>>>> I've just run the nexmark queries on v2.5.0-RC1 tag
> >>>>>> What we can notice:
> >>>>>> - query 3 (exercises CoGroupByKey, state and
> timer) shows
> >>>>> different
> >>>>>> output with DR between batch and streaming and
> with the other
> >>>>> runners =>
> >>>>>> I compared with v2.4 there were still these
> differences but with
> >>>>>> different output size numbers
> >>>>>>
> >>>>>> - query 6 (exercises specialized combiner) shows
> different output
> >>>>>> between the runners => the correct output is 401.
> strange that
> >>>>> in batch
> >>>>>> mode some runners output les Sellers. I compared
> with v2.4
> >>>>> same output
> >>>>>>
> >>>>>> - response time of query 7 (exercices Max
> transform, fanout
> >>>>> and side
> >>>>>> input) is very slow on DR => I compared with v2.4
> , comparable
> >>>>> execution
> >>>>>> times
> >>>>>>
> >>>>>> I'm not comparing q10 because it is a write to
> GCS so it is
> >>>>> very specific.
> >>>>>>
> >>>>>> => Basically no regression comparing to v2.4
> >>>>>>
> >>>>>> For the record here is the output (waiting for
> ongoing perfkit
> >>>>> integration):
> >>>>>>
> >>>>>>
> >>>>>> 1. DR batch
> >>>>>>
> >>>>>> Performance:
> >>>>>>
> >>>>>>
> >>>>> Conf Runtime(sec) (Baseline) Events(/sec)
> (Baseline) Results (Baseline)
> >>>>>>
> >>>>>>
> >>>>> 0000 5,8 17283,1
> 100000
> >>>>>>
> >>>>>>
> >>>>> 0001 3,2 31104,2
> 92000
> >>>>>>
> >>>>>>
> >>>>> 0002 1,2 82918,7
> 351
> >>>>>>
> >>>>>>
> >>>>> 0003 2,2 46210,7
> 458
> >>>>>>
> >>>>>>
> >>>>> 0004 1,2 8503,4
> 40
> >>>>>>
> >>>>>>
> >>>>> 0005 4,0 25220,7
> 12
> >>>>>>
> >>>>>>
> >>>>> 0006 0,9 11148,3
> 401
> >>>>>>
> >>>>>>
> >>>>> 0007 13,2 7580,9
> 1
> >>>>>>
> >>>>>>
> >>>>> 0008 1,5 67340,1
> 6000
> >>>>>>
> >>>>>>
> >>>>> 0009 0,7 14025,2
> 298
> >>>>>>
> >>>>>>
> >>>>> 0010 12,8 7793,0
> 1
> >>>>>>
> >>>>>>
> >>>>> 0011 2,4 42319,1
> 1919
> >>>>>>
> >>>>>>
> >>>>> 0012 1,6 61462,8
> 1919
> >>>>>>
> >>>>>
> ==========================================================================================
> >>>>>>
> >>>>>> 2. DR streaming
> >>>>>>
> >>>>>> Performance:
> >>>>>>
> >>>>>>
> >>>>> Conf Runtime(sec) (Baseline) Events(/sec)
> (Baseline) Results (Baseline)
> >>>>>>
> >>>>>>
> >>>>> 0000 6,5 15285,8
> 100000
> >>>>>>
> >>>>>>
> >>>>> 0001 3,7 27397,3
> 92000
> >>>>>>
> >>>>>>
> >>>>> 0002 1,4 69108,5
> 351
> >>>>>>
> >>>>>>
> >>>>> 0003 3,2 31181,8
> 447
> >>>>>>
> >>>>>>
> >>>>> 0004 1,2 8361,2
> 40
> >>>>>>
> >>>>>>
> >>>>> 0005 5,3 18903,6
> 12
> >>>>>>
> >>>>>>
> >>>>> 0006 0,9 11111,1
> 401
> >>>>>>
> >>>>>>
> >>>>> 0007 82,5 1212,2
> 1
> >>>>>>
> >>>>>>
> >>>>> 0008 2,0 51072,5
> 6000
> >>>>>>
> >>>>>>
> >>>>> 0009 0,8 12903,2
> 298
> >>>>>>
> >>>>>>
> >>>>> 0010 49,5 2021,8
> 1
> >>>>>>
> >>>>>>
> >>>>> 0011 3,9 25667,4
> 1919
> >>>>>>
> >>>>>>
> >>>>> 0012 2,4 41067,8
> 1919
> >>>>>>
> >>>>>
> ==========================================================================================
> >>>>>>
> >>>>>> 3. Flink batch
> >>>>>> Performance:
> >>>>>>
> >>>>>>
> >>>>> Conf Runtime(sec) (Baseline) Events(/sec)
> (Baseline) Results (Baseline)
> >>>>>>
> >>>>>>
> >>>>> 0000 1,0 97656,3
> 100000
> >>>>>>
> >>>>>>
> >>>>> 0001 0,7 141643,1
> 92000
> >>>>>>
> >>>>>>
> >>>>> 0002 0,4 228310,5
> 351
> >>>>>>
> >>>>>>
> >>>>> 0003 1,6 64020,5
> 580
> >>>>>>
> >>>>>>
> >>>>> 0004 0,7 13831,3
> 40
> >>>>>>
> >>>>>>
> >>>>> 0005 1,4 72939,5
> 12
> >>>>>>
> >>>>>>
> >>>>> 0006 0,5 20491,8
> 103
> >>>>>>
> >>>>>>
> >>>>> 0007 1,3 74239,0
> 1
> >>>>>>
> >>>>>>
> >>>>> 0008 0,8 121506,7
> 6000
> >>>>>>
> >>>>>>
> >>>>> 0009 0,6 17953,3
> 298
> >>>>>>
> >>>>>>
> >>>>> 0010 1,3 74682,6
> 1
> >>>>>>
> >>>>>>
> >>>>> 0011 1,1 92936,8
> 1919
> >>>>>>
> >>>>>>
> >>>>> 0012 0,8 123001,2
> 1919
> >>>>>>
> >>>>>
> ==========================================================================================
> >>>>>>
> >>>>>> 4. Flink streaming
> >>>>>> Performance:
> >>>>>>
> >>>>>>
> >>>>> Conf Runtime(sec) (Baseline) Events(/sec)
> (Baseline) Results (Baseline)
> >>>>>>
> >>>>>>
> >>>>> 0000 5,4 18677,6
> 100000
> >>>>>>
> >>>>>>
> >>>>> 0001 2,8 35511,4
> 92000
> >>>>>>
> >>>>>>
> >>>>> 0002 1,8 54318,3
> 351
> >>>>>>
> >>>>>>
> >>>>> 0003 2,4 41614,6
> 580
> >>>>>>
> >>>>>>
> >>>>> 0004 1,0 10341,3
> 40
> >>>>>>
> >>>>>>
> >>>>> 0005 3,4 29568,3
> 12
> >>>>>>
> >>>>>>
> >>>>> 0006 0,7 13369,0
> 401
> >>>>>>
> >>>>>>
> >>>>> 0007 2,8 36192,5
> 1
> >>>>>>
> >>>>>>
> >>>>> 0008 1,8 54854,6
> 6000
> >>>>>>
> >>>>>>
> >>>>> 0009 0,7 13369,0
> 298
> >>>>>>
> >>>>>>
> >>>>> 0010 3,4 29841,8
> 2
> >>>>>>
> >>>>>>
> >>>>> 0011 5,0 19932,2
> 1919
> >>>>>>
> >>>>>>
> >>>>> 0012 2,6 38835,0
> 1919
> >>>>>>
> >>>>>
> ==========================================================================================
> >>>>>>
> >>>>>> 5. Spark batch
> >>>>>> Performance:
> >>>>>>
> >>>>>>
> >>>>> Conf Runtime(sec) (Baseline) Events(/sec)
> (Baseline) Results (Baseline)
> >>>>>>
> >>>>>>
> >>>>> 0000 1,5 65445,0
> 100000
> >>>>>>
> >>>>>>
> >>>>> 0001 1,3 79491,3
> 92000
> >>>>>>
> >>>>>>
> >>>>> 0002 0,9 112107,6
> 351
> >>>>>>
> >>>>>>
> >>>>> 0003 2,0 48804,3
> 580
> >>>>>>
> >>>>>>
> >>>>> 0004 1,2 8382,2
> 40
> >>>>>>
> >>>>>>
> >>>>> 0005 2,0 50838,8
> 12
> >>>>>>
> >>>>>>
> >>>>> 0006 1,0 9699,3
> 103
> >>>>>>
> >>>>>>
> >>>>> 0007 2,3 43308,8
> 1
> >>>>>>
> >>>>>>
> >>>>> 0008 2,1 46794,6
> 6000
> >>>>>>
> >>>>>>
> >>>>> 0009 1,1 8976,7
> 298
> >>>>>>
> >>>>>>
> >>>>> 0010 1,6 62111,8
> 1
> >>>>>>
> >>>>>>
> >>>>> 0011 2,1 46598,3
> 1919
> >>>>>>
> >>>>>>
> >>>>> 0012 2,3 43687,2
> 1919
> >>>>>>
> >>>>>
> ==========================================================================================
> >>>>>>
> >>>>>>> Le mercredi 06 juin 2018 à 10:50 +0200, Etienne
> Chauchot a écrit :
> >>>>>>> Thanks JB for all your work ! I believe doing
> the first
> >>>>> gradle release
> >>>>>>> must have been hard.
> >>>>>>> I'll run Nexmark on the release and keep you posted.
> >>>>>>>
> >>>>>>> Best
> >>>>>>> Etienne
> >>>>>>>
> >>>>>>>
> >>>>>>> Le mercredi 06 juin 2018 à 10:44 +0200,
> Jean-Baptiste Onofré
> >>>>> a écrit :
> >>>>>>>> Hi everyone,
> >>>>>>>>
> >>>>>>>> Please review and vote on the release candidate
> #1 for the
> >>>>> version
> >>>>>>>> 2.5.0, as follows:
> >>>>>>>>
> >>>>>>>> [ ] +1, Approve the release
> >>>>>>>> [ ] -1, Do not approve the release (please
> provide specific
> >>>>> comments)
> >>>>>>>>
> >>>>>>>> NB: this is the first release using Gradle, so
> don't be too
> >>>>> harsh ;) A
> >>>>>>>> PR about the release guide will follow thanks
> to this release.
> >>>>>>>>
> >>>>>>>> The complete staging area is available for your
> review,
> >>>>> which includes:
> >>>>>>>> * JIRA release notes [1],
> >>>>>>>> * the official Apache source release to be
> deployed to
> >>>>> dist.apache.org <http://dist.apache.org>
> <http://dist.apache.org/>
> >>>>>>>> [2], which is signed with the key with
> fingerprint C8282E76 [3],
> >>>>>>>> * all artifacts to be deployed to the Maven Central
> >>>>> Repository [4],
> >>>>>>>> * source code tag "v2.5.0-RC1" [5],
> >>>>>>>> * website pull request listing the release and
> publishing
> >>>>> the API
> >>>>>>>> reference manual [6].
> >>>>>>>> * Java artifacts were built with Gradle 4.7
> (wrapper) and
> >>>>> OpenJDK/Oracle
> >>>>>>>> JDK 1.8.0_172 (Oracle Corporation 25.172-b11).
> >>>>>>>> * Python artifacts are deployed along with the
> source
> >>>>> release to the
> >>>>>>>> dist.apache.org <http://dist.apache.org>
> <http://dist.apache.org/> [2].
> >>>>>>>>
> >>>>>>>> The vote will be open for at least 72 hours. It
> is adopted
> >>>>> by majority
> >>>>>>>> approval, with at least 3 PMC affirmative votes.
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> JB
> >>>>>>>>
> >>>>>>>> [1]
> >>>>>>>>
> >>>>>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12342847
> >>>>>>>> [2]
> https://dist.apache.org/repos/dist/dev/beam/2.5.0/
> >>>>>>>> [3]
> https://dist.apache.org/repos/dist/release/beam/KEYS
> >>>>>>>> [4]
> >>>>>
> https://repository.apache.org/content/repositories/orgapachebeam-1041/
> >>>>>>>> [5] https://github.com/apache/beam/tree/v2.5.0-RC1
> >>>>>>>> [6] https://github.com/apache/beam-site/pull/463
> >>>>>>>>
> >>>>>
> >>
> >> --
> >> Got feedback? go/pabloem-feedback
> <https://goto.google.com/pabloem-feedback>
> >
> > --
> > Jean-Baptiste Onofré
> > jbonofre@apache.org <ma...@apache.org>
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
>
> --
> Got feedback? go/pabloem-feedback
> <https://goto.google.com/pabloem-feedback>
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: [VOTE] Apache Beam, version 2.5.0, release candidate #1
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi Udi,
are you voting -1 ?
It would be great if you can cast your vote (-1): I will cancel RC1
based on your vote.
Thanks,
Regards
JB
On 12/06/2018 04:02, Udi Meiri wrote:
> Another bug: reading from PubSub with_attributes=True is broken on
> Python with Dataflow.
> https://issues.apache.org/jira/browse/BEAM-4536
>
> JB, I'm making a PR that removes this keyword and I'd like to propose it
> as a cherrypick to 2.5.0.
> (feature should be fixed in the next release)
>
> On Mon, Jun 11, 2018 at 6:19 PM Chamikara Jayalath <chamikara@google.com
> <ma...@google.com>> wrote:
>
> FYI: looks like Python tests are failing for Windows. JIRA
> is https://issues.apache.org/jira/browse/BEAM-4535.
>
> I don't think this is a release blocker but this should probably go
> in release notes (for any user that tries to run tests on Python
> source build). And we should try to incorporate a fix if we happen
> to cut another release candidate for some reason.
>
> Thanks,
> Cham
>
> On Mon, Jun 11, 2018 at 5:46 PM Pablo Estrada <pabloem@google.com
> <ma...@google.com>> wrote:
>
> Thanks everyone who has pitched in to validate the release!
>
> Boyuan Zhang and I have also run a few pipelines, and verified
> that they work properly (see release validation spreadsheet[1]).
>
> We have also found that the Game Stats pipeline is failing in
> Python Streaming Dataflow. I have filed BEAM-4534[2]. This is
> not a blocker, since Python streaming is not yet fully supported.
>
> It seems that the uploaded artifacts look good.
>
> We have noticed that the Python artifacts are still missing
> Python wheel files (compare [3] and [4]). JB, could you please
> add the wheel files? Boyuan and I can try to help you prepare
> them / upload them if necessary. Please let us know.
>
> Thanks again!
> -P.
>
> [1] https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=152451807
> [2] https://issues.apache.org/jira/browse/BEAM-4534
> [3] https://dist.apache.org/repos/dist/dev/beam/2.4.0/
> [4] https://dist.apache.org/repos/dist/dev/beam/2.5.0/
>
> On Mon, Jun 11, 2018 at 12:37 PM Alan Myrvold
> <amyrvold@google.com <ma...@google.com>> wrote:
>
> +1 (non-binding)
>
> tested some of the quickstarts
>
> On Sun, Jun 10, 2018 at 1:39 AM Tim
> <timrobertson100@gmail.com
> <ma...@gmail.com>> wrote:
>
> Tested by our team:
> - mvn inclusion
> - Avro, ES, Hadoop IF IO
> - Pipelines run on Spark (Cloudera 5.12.0 YARN cluster)
> - Reviewed release notes
>
> +1
>
> Thanks also to everyone who helped get over the gradle
> hurdle and in particular to JB.
>
> Tim
>
> > On 9 Jun 2018, at 05:56, Jean-Baptiste Onofré
> <jb@nanthrax.net <ma...@nanthrax.net>> wrote:
> >
> > No problem Pablo.
> >
> > The vote period is a minimum, it can be extended as
> requested or if we
> > don't have the minimum of 3 binding votes.
> >
> > Regards
> > JB
> >
> >> On 09/06/2018 01:54, Pablo Estrada wrote:
> >> Hello all,
> >> I'd like to request an extension of the voting period
> until Monday
> >> evening (US time, so later in other geographical
> regions). This is
> >> because we were only now able to publish Dataflow
> Workers, and have not
> >> had the chance to run release validation tests on
> them. The extension
> >> will allow us to validate and vote by Monday.
> >> Is this acceptable to the community?
> >>
> >> Best
> >> -P.
> >>
> >> On Fri, Jun 8, 2018 at 6:20 AM Alexey Romanenko
> >> <aromanenko.dev@gmail.com
> <ma...@gmail.com>
> <mailto:aromanenko.dev@gmail.com
> <ma...@gmail.com>>> wrote:
> >>
> >> Thank you JB for your work!
> >>
> >> I tested running simple streaming (/KafkaIO/) and
> batch (/TextIO /
> >> HDFS/) pipelines with SparkRunner on YARN cluster
> - it works fine.
> >>
> >> WBR,
> >> Alexey
> >>
> >>
> >>> On 8 Jun 2018, at 10:00, Etienne Chauchot
> <echauchot@apache.org <ma...@apache.org>
> >>> <mailto:echauchot@apache.org
> <ma...@apache.org>>> wrote:
> >>>
> >>> I forgot to vote:
> >>> +1 (non binding).
> >>> What I tested:
> >>> - no functional or performance regression
> comparing to v2.4
> >>> - dependencies in the poms are ok
> >>>
> >>> Etienne
> >>>> Le vendredi 08 juin 2018 à 08:27 +0200, Romain
> Manni-Bucau a écrit :
> >>>> +1 (non-binding), mainstream usage is not broken
> by the pom
> >>>> changes and runtime has no known regression
> compared to the 2.4.0
> >>>>
> >>>> (side note: kudo to JB for this build tool
> change release, I know
> >>>> how it can hurt ;))
> >>>>
> >>>> Romain Manni-Bucau
> >>>> @rmannibucau <https://twitter.com/rmannibucau>
> | Blog
> >>>> <https://rmannibucau.metawerx.net/> | Old Blog
> >>>> <http://rmannibucau.wordpress.com/> | Github
> >>>> <https://github.com/rmannibucau> | LinkedIn
> >>>> <https://www.linkedin.com/in/rmannibucau> | Book
> >>>>
> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
> >>>>
> >>>>
> >>>> Le jeu. 7 juin 2018 à 16:17, Jean-Baptiste Onofré
> >>>> <jb@nanthrax.net <ma...@nanthrax.net>
> <mailto:jb@nanthrax.net <ma...@nanthrax.net>>> a écrit :
> >>>>> Thanks for the details Etienne !
> >>>>>
> >>>>> The good news is that the artifacts seem OK and
> the overall Nexmark
> >>>>> results are consistent with the 2.4.0 release ones.
> >>>>>
> >>>>> I'm starting a complete review using the
> beam-samples as well.
> >>>>>
> >>>>> Regards
> >>>>> JB
> >>>>>
> >>>>>> On 07/06/2018 16:14, Etienne Chauchot wrote:
> >>>>>> Hi,
> >>>>>> I've just run the nexmark queries on v2.5.0-RC1 tag
> >>>>>> What we can notice:
> >>>>>> - query 3 (exercises CoGroupByKey, state and
> timer) shows
> >>>>> different
> >>>>>> output with DR between batch and streaming and
> with the other
> >>>>> runners =>
> >>>>>> I compared with v2.4 there were still these
> differences but with
> >>>>>> different output size numbers
> >>>>>>
> >>>>>> - query 6 (exercises specialized combiner) shows
> different output
> >>>>>> between the runners => the correct output is 401.
> strange that
> >>>>> in batch
> >>>>>> mode some runners output les Sellers. I compared
> with v2.4
> >>>>> same output
> >>>>>>
> >>>>>> - response time of query 7 (exercices Max
> transform, fanout
> >>>>> and side
> >>>>>> input) is very slow on DR => I compared with v2.4
> , comparable
> >>>>> execution
> >>>>>> times
> >>>>>>
> >>>>>> I'm not comparing q10 because it is a write to
> GCS so it is
> >>>>> very specific.
> >>>>>>
> >>>>>> => Basically no regression comparing to v2.4
> >>>>>>
> >>>>>> For the record here is the output (waiting for
> ongoing perfkit
> >>>>> integration):
> >>>>>>
> >>>>>>
> >>>>>> 1. DR batch
> >>>>>>
> >>>>>> Performance:
> >>>>>>
> >>>>>>
> >>>>> Conf Runtime(sec) (Baseline) Events(/sec)
> (Baseline) Results (Baseline)
> >>>>>>
> >>>>>>
> >>>>> 0000 5,8 17283,1
> 100000
> >>>>>>
> >>>>>>
> >>>>> 0001 3,2 31104,2
> 92000
> >>>>>>
> >>>>>>
> >>>>> 0002 1,2 82918,7
> 351
> >>>>>>
> >>>>>>
> >>>>> 0003 2,2 46210,7
> 458
> >>>>>>
> >>>>>>
> >>>>> 0004 1,2 8503,4
> 40
> >>>>>>
> >>>>>>
> >>>>> 0005 4,0 25220,7
> 12
> >>>>>>
> >>>>>>
> >>>>> 0006 0,9 11148,3
> 401
> >>>>>>
> >>>>>>
> >>>>> 0007 13,2 7580,9
> 1
> >>>>>>
> >>>>>>
> >>>>> 0008 1,5 67340,1
> 6000
> >>>>>>
> >>>>>>
> >>>>> 0009 0,7 14025,2
> 298
> >>>>>>
> >>>>>>
> >>>>> 0010 12,8 7793,0
> 1
> >>>>>>
> >>>>>>
> >>>>> 0011 2,4 42319,1
> 1919
> >>>>>>
> >>>>>>
> >>>>> 0012 1,6 61462,8
> 1919
> >>>>>>
> >>>>>
> ==========================================================================================
> >>>>>>
> >>>>>> 2. DR streaming
> >>>>>>
> >>>>>> Performance:
> >>>>>>
> >>>>>>
> >>>>> Conf Runtime(sec) (Baseline) Events(/sec)
> (Baseline) Results (Baseline)
> >>>>>>
> >>>>>>
> >>>>> 0000 6,5 15285,8
> 100000
> >>>>>>
> >>>>>>
> >>>>> 0001 3,7 27397,3
> 92000
> >>>>>>
> >>>>>>
> >>>>> 0002 1,4 69108,5
> 351
> >>>>>>
> >>>>>>
> >>>>> 0003 3,2 31181,8
> 447
> >>>>>>
> >>>>>>
> >>>>> 0004 1,2 8361,2
> 40
> >>>>>>
> >>>>>>
> >>>>> 0005 5,3 18903,6
> 12
> >>>>>>
> >>>>>>
> >>>>> 0006 0,9 11111,1
> 401
> >>>>>>
> >>>>>>
> >>>>> 0007 82,5 1212,2
> 1
> >>>>>>
> >>>>>>
> >>>>> 0008 2,0 51072,5
> 6000
> >>>>>>
> >>>>>>
> >>>>> 0009 0,8 12903,2
> 298
> >>>>>>
> >>>>>>
> >>>>> 0010 49,5 2021,8
> 1
> >>>>>>
> >>>>>>
> >>>>> 0011 3,9 25667,4
> 1919
> >>>>>>
> >>>>>>
> >>>>> 0012 2,4 41067,8
> 1919
> >>>>>>
> >>>>>
> ==========================================================================================
> >>>>>>
> >>>>>> 3. Flink batch
> >>>>>> Performance:
> >>>>>>
> >>>>>>
> >>>>> Conf Runtime(sec) (Baseline) Events(/sec)
> (Baseline) Results (Baseline)
> >>>>>>
> >>>>>>
> >>>>> 0000 1,0 97656,3
> 100000
> >>>>>>
> >>>>>>
> >>>>> 0001 0,7 141643,1
> 92000
> >>>>>>
> >>>>>>
> >>>>> 0002 0,4 228310,5
> 351
> >>>>>>
> >>>>>>
> >>>>> 0003 1,6 64020,5
> 580
> >>>>>>
> >>>>>>
> >>>>> 0004 0,7 13831,3
> 40
> >>>>>>
> >>>>>>
> >>>>> 0005 1,4 72939,5
> 12
> >>>>>>
> >>>>>>
> >>>>> 0006 0,5 20491,8
> 103
> >>>>>>
> >>>>>>
> >>>>> 0007 1,3 74239,0
> 1
> >>>>>>
> >>>>>>
> >>>>> 0008 0,8 121506,7
> 6000
> >>>>>>
> >>>>>>
> >>>>> 0009 0,6 17953,3
> 298
> >>>>>>
> >>>>>>
> >>>>> 0010 1,3 74682,6
> 1
> >>>>>>
> >>>>>>
> >>>>> 0011 1,1 92936,8
> 1919
> >>>>>>
> >>>>>>
> >>>>> 0012 0,8 123001,2
> 1919
> >>>>>>
> >>>>>
> ==========================================================================================
> >>>>>>
> >>>>>> 4. Flink streaming
> >>>>>> Performance:
> >>>>>>
> >>>>>>
> >>>>> Conf Runtime(sec) (Baseline) Events(/sec)
> (Baseline) Results (Baseline)
> >>>>>>
> >>>>>>
> >>>>> 0000 5,4 18677,6
> 100000
> >>>>>>
> >>>>>>
> >>>>> 0001 2,8 35511,4
> 92000
> >>>>>>
> >>>>>>
> >>>>> 0002 1,8 54318,3
> 351
> >>>>>>
> >>>>>>
> >>>>> 0003 2,4 41614,6
> 580
> >>>>>>
> >>>>>>
> >>>>> 0004 1,0 10341,3
> 40
> >>>>>>
> >>>>>>
> >>>>> 0005 3,4 29568,3
> 12
> >>>>>>
> >>>>>>
> >>>>> 0006 0,7 13369,0
> 401
> >>>>>>
> >>>>>>
> >>>>> 0007 2,8 36192,5
> 1
> >>>>>>
> >>>>>>
> >>>>> 0008 1,8 54854,6
> 6000
> >>>>>>
> >>>>>>
> >>>>> 0009 0,7 13369,0
> 298
> >>>>>>
> >>>>>>
> >>>>> 0010 3,4 29841,8
> 2
> >>>>>>
> >>>>>>
> >>>>> 0011 5,0 19932,2
> 1919
> >>>>>>
> >>>>>>
> >>>>> 0012 2,6 38835,0
> 1919
> >>>>>>
> >>>>>
> ==========================================================================================
> >>>>>>
> >>>>>> 5. Spark batch
> >>>>>> Performance:
> >>>>>>
> >>>>>>
> >>>>> Conf Runtime(sec) (Baseline) Events(/sec)
> (Baseline) Results (Baseline)
> >>>>>>
> >>>>>>
> >>>>> 0000 1,5 65445,0
> 100000
> >>>>>>
> >>>>>>
> >>>>> 0001 1,3 79491,3
> 92000
> >>>>>>
> >>>>>>
> >>>>> 0002 0,9 112107,6
> 351
> >>>>>>
> >>>>>>
> >>>>> 0003 2,0 48804,3
> 580
> >>>>>>
> >>>>>>
> >>>>> 0004 1,2 8382,2
> 40
> >>>>>>
> >>>>>>
> >>>>> 0005 2,0 50838,8
> 12
> >>>>>>
> >>>>>>
> >>>>> 0006 1,0 9699,3
> 103
> >>>>>>
> >>>>>>
> >>>>> 0007 2,3 43308,8
> 1
> >>>>>>
> >>>>>>
> >>>>> 0008 2,1 46794,6
> 6000
> >>>>>>
> >>>>>>
> >>>>> 0009 1,1 8976,7
> 298
> >>>>>>
> >>>>>>
> >>>>> 0010 1,6 62111,8
> 1
> >>>>>>
> >>>>>>
> >>>>> 0011 2,1 46598,3
> 1919
> >>>>>>
> >>>>>>
> >>>>> 0012 2,3 43687,2
> 1919
> >>>>>>
> >>>>>
> ==========================================================================================
> >>>>>>
> >>>>>>> Le mercredi 06 juin 2018 à 10:50 +0200, Etienne
> Chauchot a écrit :
> >>>>>>> Thanks JB for all your work ! I believe doing
> the first
> >>>>> gradle release
> >>>>>>> must have been hard.
> >>>>>>> I'll run Nexmark on the release and keep you posted.
> >>>>>>>
> >>>>>>> Best
> >>>>>>> Etienne
> >>>>>>>
> >>>>>>>
> >>>>>>> Le mercredi 06 juin 2018 à 10:44 +0200,
> Jean-Baptiste Onofré
> >>>>> a écrit :
> >>>>>>>> Hi everyone,
> >>>>>>>>
> >>>>>>>> Please review and vote on the release candidate
> #1 for the
> >>>>> version
> >>>>>>>> 2.5.0, as follows:
> >>>>>>>>
> >>>>>>>> [ ] +1, Approve the release
> >>>>>>>> [ ] -1, Do not approve the release (please
> provide specific
> >>>>> comments)
> >>>>>>>>
> >>>>>>>> NB: this is the first release using Gradle, so
> don't be too
> >>>>> harsh ;) A
> >>>>>>>> PR about the release guide will follow thanks
> to this release.
> >>>>>>>>
> >>>>>>>> The complete staging area is available for your
> review,
> >>>>> which includes:
> >>>>>>>> * JIRA release notes [1],
> >>>>>>>> * the official Apache source release to be
> deployed to
> >>>>> dist.apache.org <http://dist.apache.org>
> <http://dist.apache.org/>
> >>>>>>>> [2], which is signed with the key with
> fingerprint C8282E76 [3],
> >>>>>>>> * all artifacts to be deployed to the Maven Central
> >>>>> Repository [4],
> >>>>>>>> * source code tag "v2.5.0-RC1" [5],
> >>>>>>>> * website pull request listing the release and
> publishing
> >>>>> the API
> >>>>>>>> reference manual [6].
> >>>>>>>> * Java artifacts were built with Gradle 4.7
> (wrapper) and
> >>>>> OpenJDK/Oracle
> >>>>>>>> JDK 1.8.0_172 (Oracle Corporation 25.172-b11).
> >>>>>>>> * Python artifacts are deployed along with the
> source
> >>>>> release to the
> >>>>>>>> dist.apache.org <http://dist.apache.org>
> <http://dist.apache.org/> [2].
> >>>>>>>>
> >>>>>>>> The vote will be open for at least 72 hours. It
> is adopted
> >>>>> by majority
> >>>>>>>> approval, with at least 3 PMC affirmative votes.
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> JB
> >>>>>>>>
> >>>>>>>> [1]
> >>>>>>>>
> >>>>>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12342847
> >>>>>>>> [2]
> https://dist.apache.org/repos/dist/dev/beam/2.5.0/
> >>>>>>>> [3]
> https://dist.apache.org/repos/dist/release/beam/KEYS
> >>>>>>>> [4]
> >>>>>
> https://repository.apache.org/content/repositories/orgapachebeam-1041/
> >>>>>>>> [5] https://github.com/apache/beam/tree/v2.5.0-RC1
> >>>>>>>> [6] https://github.com/apache/beam-site/pull/463
> >>>>>>>>
> >>>>>
> >>
> >> --
> >> Got feedback? go/pabloem-feedback
> <https://goto.google.com/pabloem-feedback>
> >
> > --
> > Jean-Baptiste Onofré
> > jbonofre@apache.org <ma...@apache.org>
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
>
> --
> Got feedback? go/pabloem-feedback
> <https://goto.google.com/pabloem-feedback>
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: [CANCEL][VOTE] Apache Beam, version 2.5.0, release candidate #1
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
It looks good to me, I'm merging and moving forward.
Regards
JB
On 14/06/2018 00:45, Pablo Estrada wrote:
> Sent out https://github.com/apache/beam/pull/5640 to ignore the flaky
> test. As JB is the release manager, I'l let him make the call on what to
> do about it.
> Best
> -P.
>
> On Wed, Jun 13, 2018 at 3:34 PM Ahmet Altay <altay@google.com
> <ma...@google.com>> wrote:
>
> I would vote for second option, not a release blocker and disable
> the test in the release branch. My reasoning is:
> - ReferenceRunner is not yet the official alternative to existing
> direct runners.
> - It is bad to have flaky tests on the release branch, and we would
> not get good signal during validation.
>
> On Wed, Jun 13, 2018 at 3:14 PM, Pablo Estrada <pabloem@google.com
> <ma...@google.com>> wrote:
>
> Hello all,
> cherrypicks for the release branch seem to be going well, but
> thanks to them we were able to surface a flaky test in the
> release branch. JIRA is
> filed: https://issues.apache.org/jira/projects/BEAM/issues/BEAM-4558
>
> Given that test issue, I see the following options:
> - Consider that this test is not a release blocker. Go ahead
> with RC2 after cherrypicks are brought in, or
> - Consider that this test is not a release blocker, so we
> disable it before cutting RC2.
> - Consider this test a release blocker, and triage the bug for
> fixing.
>
> What do you think?
>
> Best
> -P.
>
> On Wed, Jun 13, 2018 at 9:54 AM Pablo Estrada
> <pabloem@google.com <ma...@google.com>> wrote:
>
> Precommits for
> PR https://github.com/apache/beam/pull/5609 are now passing.
> For now I've simply set failOnWarning to false to cherrypick
> into the release, and fix in master later on.
> Best
> -P.
>
> On Wed, Jun 13, 2018 at 9:08 AM Scott Wegner
> <swegner@google.com <ma...@google.com>> wrote:
>
> From my understanding, the @SuppressFBWarnings usage is
> in a dependency (ByteBuddy) rather than directly in our
> code; so we're not able to modify the usage.
>
> Pablo, feel free to disable failOnWarning for the
> sdks-java-core project temporarily. This isn't a major
> regression since we've only recently made the change to
> enable it [1]. We can work separately on figuring out
> how to resolve the warnings.
>
> [1] https://github.com/apache/beam/pull/5319
>
> On Tue, Jun 12, 2018 at 11:57 PM Tim Robertson
> <timrobertson100@gmail.com
> <ma...@gmail.com>> wrote:
>
> Hi Pablo,
>
> I'm afraid I couldn't find one either... there is an
> issue about it [1] which is old so it doesn't look
> likely to be resolved either.
>
> If you have time (sorry I am a bit busy) could you
> please verify the version does work if you install
> that version locally? I know the maven version of
> that [2] but not sure on the gradle equivalent. If
> we know it works, we can then find a repository that
> fits ok with Apache/Beam policy.
>
> Alternatively, we could consider using a fully
> qualified reference (i.e.
> @edu.umd.cs.findbugs.annotations.SuppressWarnings)
> to the deprecated version and leave the dependency
> at the 1.3.9-1. I believe our general direction is
> to remove findbugs when errorprone covers all
> aspects so I *expect* this should be considered
> reasonable.
>
> I hope this helps,
> Tim
>
> [1] https://github.com/stephenc/findbugs-annotations/issues/4
> [2] https://maven.apache.org/guides/mini/guide-3rd-party-jars-local.html
>
> On Wed, Jun 13, 2018 at 8:39 AM, Pablo Estrada
> <pabloem@google.com <ma...@google.com>> wrote:
>
> Hi Tim,
> you're right. Thanks for pointing that out.
> There's just one problem that I'm running into
> now: The 3.0.1-1 version does not seem to be
> available in Maven Central[1]. Looking at the
> website, I am not quite sure if there's another
> repository where they do stage the newer
> versions?[2]
>
> -P
>
> [1] https://repo.maven.apache.org/maven2/com/github/stephenc/findbugs/findbugs-annotations/
> [2] http://stephenc.github.io/findbugs-annotations/
>
> On Tue, Jun 12, 2018 at 11:10 PM Tim Robertson
> <timrobertson100@gmail.com
> <ma...@gmail.com>> wrote:
>
> Hi Pablo,
>
> I took only a quick look.
>
> "- The JAR from the non-LGPL findbugs does
> not contain the SuppressFBWarnings annotation"
>
> Unless I misunderstand you it looks like
> SuppressFBWarnings was added in Stephen's
> version in this commit [1] which was
> introduced in version 2.0.3-1 - I've
> checked is in the 3.0.1-1 build [2]
> I notice in your commits [1] you've been
> exploring version 3.0.0 already though...
> what happens when you use 3.0.1-1? It sounds
> like the wrong version is coming in rather
> than the annotation being missing.
>
> Thanks,
> Tim
>
> [1] https://github.com/stephenc/findbugs-annotations/commits/master/src/main/java/edu/umd/cs/findbugs/annotations/SuppressWarnings.java
> [2] https://github.com/stephenc/findbugs-annotations/releases
> [3] https://github.com/apache/beam/pull/5609/commits/32c7df706e970557f154ff6bc521b2e00f9d09ab
>
>
>
>
>
>
>
> On Wed, Jun 13, 2018 at 2:37 AM, Pablo
> Estrada <pabloem@google.com
> <ma...@google.com>> wrote:
>
> Hi all,
> I'll humbly declare that after wrestling
> with he build to stop depending on the
> wrong findbugs_annotations, I feel
> somewhat lost. The issue is actually
> quite small:
>
> - The JAR from the non-LGPL findbugs
> does not contain the SuppressFBWarnings
> annotation. This means that when
> building, ByteBuddy produces a few
> warnings (nothing critical).
> - The easiest way to avoid this failure
> is to call
> applyJavaNature(failOnWarning: false),
> but this would be bad, since we want to
> keep a high standard for tasks like
> ErrorProne and FindBugs itself.
> - So I find myself lost: How do we
> suppress trivial warnings coming from
> missing annotations, and honor warnings
> coming from other plugins?
>
> Any help / a PR from someone more
> capable would be appreciated.
> Best
> -P.
>
> On Tue, Jun 12, 2018 at 3:02 PM Ismaël
> Mejía <iemejia@gmail.com
> <ma...@gmail.com>> wrote:
>
> Yes, ok I was not aware it was
> already being addressed, nice.
> On Tue, Jun 12, 2018 at 11:56 PM
> Ahmet Altay <altay@google.com
> <ma...@google.com>> wrote:
> >
> > Ismaël,
> >
> > I believe Pablo's
> https://github.com/apache/beam/pull/5609
> is fixing the issue by changing the
> findbugs back to
> "com.github.stephenc.findbugs". Is
> this what you are referring to?
> >
> > Ahmet
> >
> > On Tue, Jun 12, 2018 at 2:51 PM,
> Boyuan Zhang <boyuanz@google.com
> <ma...@google.com>> wrote:
> >>
> >> Hey JB,
> >>
> >> I added some instructions about
> how to create python wheels in this
> PR:
> https://github.com/apache/beam-site/pull/467
> . Hope it would be helpful.
> >>
> >> Boyuan
> >>
> >
>
> --
> Got feedback? go/pabloem-feedback
> <https://goto.google.com/pabloem-feedback>
>
>
> --
> Got feedback? go/pabloem-feedback
> <https://goto.google.com/pabloem-feedback>
>
>
> --
> Got feedback? go/pabloem-feedback
> <https://goto.google.com/pabloem-feedback>
>
> --
> Got feedback? go/pabloem-feedback
> <https://goto.google.com/pabloem-feedback>
>
>
> --
> Got feedback? go/pabloem-feedback
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: [CANCEL][VOTE] Apache Beam, version 2.5.0, release candidate #1
Posted by Ismaël Mejía <ie...@gmail.com>.
There is another issue highlighted by Scott Wegner in a non-related to
the vote PR discussion.
https://github.com/apache/beam/pull/5540
It seems that in the migration to gradle we changed the
findbugs-annotations library from com.github.stephenc.findbugs to
com.google.code.findbugs:findbugs-annotations and this one is LGPL
licensed so we should address that too for the next RC.
On Tue, Jun 12, 2018 at 9:33 PM Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>
> Hi all,
>
> due to issues found by Udi and Pablo, I cancel RC1 to prepare a new release.
>
> Thanks Udi and Pablo for catching up.
>
> I will prepare a RC2 after cherry pick/fixed the found issues (tomorrow
> morning my time).
>
> Regards
> JB
>
>
> On 12/06/2018 21:04, Pablo Estrada wrote:
> > Hello all,
> > I've found another release blocker: The current findbugs dependency in
> > build_rules.gradle has a GPL license. Here's the PR to depend on the
> > correct findbugs: https://github.com/apache/beam/pull/5609
> >
> > Best
> > -P.
> >
> > On Tue, Jun 12, 2018 at 11:03 AM Udi Meiri <ehudm@google.com
> > <ma...@google.com>> wrote:
> >
> > Cherrypick created: https://github.com/apache/beam/pull/5607
> > Tests still running.
> >
> > On Tue, Jun 12, 2018 at 9:59 AM Udi Meiri <ehudm@google.com
> > <ma...@google.com>> wrote:
> >
> > -1: Would like to cherry pick a fix
> > for https://issues.apache.org/jira/browse/BEAM-4536
> >
> > On Tue, Jun 12, 2018 at 9:48 AM Jean-Baptiste Onofré
> > <jb@nanthrax.net <ma...@nanthrax.net>> wrote:
> >
> > Any update about your vote and fix ?
> >
> > Thanks !
> > Regards
> > JB
> >
> > On 12/06/2018 04:02, Udi Meiri wrote:
> > > Another bug: reading from PubSub with_attributes=True is
> > broken on
> > > Python with Dataflow.
> > > https://issues.apache.org/jira/browse/BEAM-4536
> > >
> > > JB, I'm making a PR that removes this keyword and I'd like
> > to propose it
> > > as a cherrypick to 2.5.0.
> > > (feature should be fixed in the next release)
> > >
> > > On Mon, Jun 11, 2018 at 6:19 PM Chamikara Jayalath
> > <chamikara@google.com <ma...@google.com>
> > > <mailto:chamikara@google.com
> > <ma...@google.com>>> wrote:
> > >
> > > FYI: looks like Python tests are failing for Windows. JIRA
> > > is https://issues.apache.org/jira/browse/BEAM-4535.
> > >
> > > I don't think this is a release blocker but this
> > should probably go
> > > in release notes (for any user that tries to run tests
> > on Python
> > > source build). And we should try to incorporate a fix
> > if we happen
> > > to cut another release candidate for some reason.
> > >
> > > Thanks,
> > > Cham
> > >
> > > On Mon, Jun 11, 2018 at 5:46 PM Pablo Estrada
> > <pabloem@google.com <ma...@google.com>
> > > <mailto:pabloem@google.com
> > <ma...@google.com>>> wrote:
> > >
> > > Thanks everyone who has pitched in to validate the
> > release!
> > >
> > > Boyuan Zhang and I have also run a few pipelines,
> > and verified
> > > that they work properly (see release validation
> > spreadsheet[1]).
> > >
> > > We have also found that the Game Stats pipeline is
> > failing in
> > > Python Streaming Dataflow. I have filed
> > BEAM-4534[2]. This is
> > > not a blocker, since Python streaming is not yet
> > fully supported.
> > >
> > > It seems that the uploaded artifacts look good.
> > >
> > > We have noticed that the Python artifacts are
> > still missing
> > > Python wheel files (compare [3] and [4]). JB,
> > could you please
> > > add the wheel files? Boyuan and I can try to help
> > you prepare
> > > them / upload them if necessary. Please let us know.
> > >
> > > Thanks again!
> > > -P.
> > >
> > >
> > [1] https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=152451807
> > > [2] https://issues.apache.org/jira/browse/BEAM-4534
> > > [3] https://dist.apache.org/repos/dist/dev/beam/2.4.0/
> > > [4] https://dist.apache.org/repos/dist/dev/beam/2.5.0/
> > >
> > > On Mon, Jun 11, 2018 at 12:37 PM Alan Myrvold
> > > <amyrvold@google.com <ma...@google.com>
> > <mailto:amyrvold@google.com <ma...@google.com>>>
> > wrote:
> > >
> > > +1 (non-binding)
> > >
> > > tested some of the quickstarts
> > >
> > > On Sun, Jun 10, 2018 at 1:39 AM Tim
> > > <timrobertson100@gmail.com
> > <ma...@gmail.com>
> > > <mailto:timrobertson100@gmail.com
> > <ma...@gmail.com>>> wrote:
> > >
> > > Tested by our team:
> > > - mvn inclusion
> > > - Avro, ES, Hadoop IF IO
> > > - Pipelines run on Spark (Cloudera 5.12.0
> > YARN cluster)
> > > - Reviewed release notes
> > >
> > > +1
> > >
> > > Thanks also to everyone who helped get
> > over the gradle
> > > hurdle and in particular to JB.
> > >
> > > Tim
> > >
> > > > On 9 Jun 2018, at 05:56, Jean-Baptiste
> > Onofré
> > > <jb@nanthrax.net <ma...@nanthrax.net>
> > <mailto:jb@nanthrax.net <ma...@nanthrax.net>>> wrote:
> > > >
> > > > No problem Pablo.
> > > >
> > > > The vote period is a minimum, it can be
> > extended as
> > > requested or if we
> > > > don't have the minimum of 3 binding votes.
> > > >
> > > > Regards
> > > > JB
> > > >
> > > >> On 09/06/2018 01:54, Pablo Estrada wrote:
> > > >> Hello all,
> > > >> I'd like to request an extension of the
> > voting period
> > > until Monday
> > > >> evening (US time, so later in other
> > geographical
> > > regions). This is
> > > >> because we were only now able to
> > publish Dataflow
> > > Workers, and have not
> > > >> had the chance to run release
> > validation tests on
> > > them. The extension
> > > >> will allow us to validate and vote by
> > Monday.
> > > >> Is this acceptable to the community?
> > > >>
> > > >> Best
> > > >> -P.
> > > >>
> > > >> On Fri, Jun 8, 2018 at 6:20 AM Alexey
> > Romanenko
> > > >> <aromanenko.dev@gmail.com
> > <ma...@gmail.com>
> > > <mailto:aromanenko.dev@gmail.com
> > <ma...@gmail.com>>
> > > <mailto:aromanenko.dev@gmail.com
> > <ma...@gmail.com>
> > > <mailto:aromanenko.dev@gmail.com
> > <ma...@gmail.com>>>> wrote:
> > > >>
> > > >> Thank you JB for your work!
> > > >>
> > > >> I tested running simple streaming
> > (/KafkaIO/) and
> > > batch (/TextIO /
> > > >> HDFS/) pipelines with SparkRunner on
> > YARN cluster
> > > - it works fine.
> > > >>
> > > >> WBR,
> > > >> Alexey
> > > >>
> > > >>
> > > >>> On 8 Jun 2018, at 10:00, Etienne
> > Chauchot
> > > <echauchot@apache.org
> > <ma...@apache.org> <mailto:echauchot@apache.org
> > <ma...@apache.org>>
> > > >>> <mailto:echauchot@apache.org
> > <ma...@apache.org>
> > > <mailto:echauchot@apache.org
> > <ma...@apache.org>>>> wrote:
> > > >>>
> > > >>> I forgot to vote:
> > > >>> +1 (non binding).
> > > >>> What I tested:
> > > >>> - no functional or performance
> > regression
> > > comparing to v2.4
> > > >>> - dependencies in the poms are ok
> > > >>>
> > > >>> Etienne
> > > >>>> Le vendredi 08 juin 2018 à 08:27
> > +0200, Romain
> > > Manni-Bucau a écrit :
> > > >>>> +1 (non-binding), mainstream usage
> > is not broken
> > > by the pom
> > > >>>> changes and runtime has no known
> > regression
> > > compared to the 2.4.0
> > > >>>>
> > > >>>> (side note: kudo to JB for this
> > build tool
> > > change release, I know
> > > >>>> how it can hurt ;))
> > > >>>>
> > > >>>> Romain Manni-Bucau
> > > >>>> @rmannibucau
> > <https://twitter.com/rmannibucau>
> > > | Blog
> > > >>>>
> > <https://rmannibucau.metawerx.net/> | Old Blog
> > > >>>>
> > <http://rmannibucau.wordpress.com/> | Github
> > > >>>> <https://github.com/rmannibucau> |
> > LinkedIn
> > > >>>>
> > <https://www.linkedin.com/in/rmannibucau> | Book
> > > >>>>
> > >
> > <https://www.packtpub.com/application-development/java-ee-8-high-performance>
> > > >>>>
> > > >>>>
> > > >>>> Le jeu. 7 juin 2018 à 16:17,
> > Jean-Baptiste Onofré
> > > >>>> <jb@nanthrax.net
> > <ma...@nanthrax.net> <mailto:jb@nanthrax.net
> > <ma...@nanthrax.net>>
> > > <mailto:jb@nanthrax.net
> > <ma...@nanthrax.net> <mailto:jb@nanthrax.net
> > <ma...@nanthrax.net>>>> a écrit :
> > > >>>>> Thanks for the details Etienne !
> > > >>>>>
> > > >>>>> The good news is that the
> > artifacts seem OK and
> > > the overall Nexmark
> > > >>>>> results are consistent with the
> > 2.4.0 release ones.
> > > >>>>>
> > > >>>>> I'm starting a complete review
> > using the
> > > beam-samples as well.
> > > >>>>>
> > > >>>>> Regards
> > > >>>>> JB
> > > >>>>>
> > > >>>>>> On 07/06/2018 16:14, Etienne
> > Chauchot wrote:
> > > >>>>>> Hi,
> > > >>>>>> I've just run the nexmark queries
> > on v2.5.0-RC1 tag
> > > >>>>>> What we can notice:
> > > >>>>>> - query 3 (exercises CoGroupByKey,
> > state and
> > > timer) shows
> > > >>>>> different
> > > >>>>>> output with DR between batch and
> > streaming and
> > > with the other
> > > >>>>> runners =>
> > > >>>>>> I compared with v2.4 there were
> > still these
> > > differences but with
> > > >>>>>> different output size numbers
> > > >>>>>>
> > > >>>>>> - query 6 (exercises specialized
> > combiner) shows
> > > different output
> > > >>>>>> between the runners => the correct
> > output is 401.
> > > strange that
> > > >>>>> in batch
> > > >>>>>> mode some runners output les
> > Sellers. I compared
> > > with v2.4
> > > >>>>> same output
> > > >>>>>>
> > > >>>>>> - response time of query 7
> > (exercices Max
> > > transform, fanout
> > > >>>>> and side
> > > >>>>>> input) is very slow on DR => I
> > compared with v2.4
> > > , comparable
> > > >>>>> execution
> > > >>>>>> times
> > > >>>>>>
> > > >>>>>> I'm not comparing q10 because it is
> > a write to
> > > GCS so it is
> > > >>>>> very specific.
> > > >>>>>>
> > > >>>>>> => Basically no regression
> > comparing to v2.4
> > > >>>>>>
> > > >>>>>> For the record here is the output
> > (waiting for
> > > ongoing perfkit
> > > >>>>> integration):
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> 1. DR batch
> > > >>>>>>
> > > >>>>>> Performance:
> > > >>>>>>
> > > >>>>>>
> > > >>>>> Conf Runtime(sec) (Baseline)
> > Events(/sec)
> > > (Baseline) Results (Baseline)
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0000 5,8
> > 17283,1
> > > 100000
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0001 3,2
> > 31104,2
> > > 92000
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0002 1,2
> > 82918,7
> > > 351
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0003 2,2
> > 46210,7
> > > 458
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0004 1,2
> > 8503,4
> > > 40
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0005 4,0
> > 25220,7
> > > 12
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0006 0,9
> > 11148,3
> > > 401
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0007 13,2
> > 7580,9
> > > 1
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0008 1,5
> > 67340,1
> > > 6000
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0009 0,7
> > 14025,2
> > > 298
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0010 12,8
> > 7793,0
> > > 1
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0011 2,4
> > 42319,1
> > > 1919
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0012 1,6
> > 61462,8
> > > 1919
> > > >>>>>>
> > > >>>>>
> > >
> > ==========================================================================================
> > > >>>>>>
> > > >>>>>> 2. DR streaming
> > > >>>>>>
> > > >>>>>> Performance:
> > > >>>>>>
> > > >>>>>>
> > > >>>>> Conf Runtime(sec) (Baseline)
> > Events(/sec)
> > > (Baseline) Results (Baseline)
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0000 6,5
> > 15285,8
> > > 100000
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0001 3,7
> > 27397,3
> > > 92000
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0002 1,4
> > 69108,5
> > > 351
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0003 3,2
> > 31181,8
> > > 447
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0004 1,2
> > 8361,2
> > > 40
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0005 5,3
> > 18903,6
> > > 12
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0006 0,9
> > 11111,1
> > > 401
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0007 82,5
> > 1212,2
> > > 1
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0008 2,0
> > 51072,5
> > > 6000
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0009 0,8
> > 12903,2
> > > 298
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0010 49,5
> > 2021,8
> > > 1
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0011 3,9
> > 25667,4
> > > 1919
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0012 2,4
> > 41067,8
> > > 1919
> > > >>>>>>
> > > >>>>>
> > >
> > ==========================================================================================
> > > >>>>>>
> > > >>>>>> 3. Flink batch
> > > >>>>>> Performance:
> > > >>>>>>
> > > >>>>>>
> > > >>>>> Conf Runtime(sec) (Baseline)
> > Events(/sec)
> > > (Baseline) Results (Baseline)
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0000 1,0
> > 97656,3
> > > 100000
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0001 0,7
> > 141643,1
> > > 92000
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0002 0,4
> > 228310,5
> > > 351
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0003 1,6
> > 64020,5
> > > 580
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0004 0,7
> > 13831,3
> > > 40
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0005 1,4
> > 72939,5
> > > 12
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0006 0,5
> > 20491,8
> > > 103
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0007 1,3
> > 74239,0
> > > 1
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0008 0,8
> > 121506,7
> > > 6000
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0009 0,6
> > 17953,3
> > > 298
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0010 1,3
> > 74682,6
> > > 1
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0011 1,1
> > 92936,8
> > > 1919
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0012 0,8
> > 123001,2
> > > 1919
> > > >>>>>>
> > > >>>>>
> > >
> > ==========================================================================================
> > > >>>>>>
> > > >>>>>> 4. Flink streaming
> > > >>>>>> Performance:
> > > >>>>>>
> > > >>>>>>
> > > >>>>> Conf Runtime(sec) (Baseline)
> > Events(/sec)
> > > (Baseline) Results (Baseline)
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0000 5,4
> > 18677,6
> > > 100000
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0001 2,8
> > 35511,4
> > > 92000
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0002 1,8
> > 54318,3
> > > 351
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0003 2,4
> > 41614,6
> > > 580
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0004 1,0
> > 10341,3
> > > 40
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0005 3,4
> > 29568,3
> > > 12
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0006 0,7
> > 13369,0
> > > 401
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0007 2,8
> > 36192,5
> > > 1
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0008 1,8
> > 54854,6
> > > 6000
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0009 0,7
> > 13369,0
> > > 298
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0010 3,4
> > 29841,8
> > > 2
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0011 5,0
> > 19932,2
> > > 1919
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0012 2,6
> > 38835,0
> > > 1919
> > > >>>>>>
> > > >>>>>
> > >
> > ==========================================================================================
> > > >>>>>>
> > > >>>>>> 5. Spark batch
> > > >>>>>> Performance:
> > > >>>>>>
> > > >>>>>>
> > > >>>>> Conf Runtime(sec) (Baseline)
> > Events(/sec)
> > > (Baseline) Results (Baseline)
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0000 1,5
> > 65445,0
> > > 100000
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0001 1,3
> > 79491,3
> > > 92000
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0002 0,9
> > 112107,6
> > > 351
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0003 2,0
> > 48804,3
> > > 580
> > > >>
Re: [CANCEL][VOTE] Apache Beam, version 2.5.0, release candidate #1
Posted by Eugene Kirpichov <ki...@google.com>.
FWIW I have a fix to the flaky test in
https://github.com/apache/beam/pull/5585 (open)
On Wed, Jun 13, 2018 at 5:26 PM Udi Meiri <eh...@google.com> wrote:
> +1 to ignoring flaky test.
>
> FYI there's a fourth cherrypick: https://github.com/apache/beam/pull/5624
>
> On Wed, Jun 13, 2018 at 3:45 PM Pablo Estrada <pa...@google.com> wrote:
>
>> Sent out https://github.com/apache/beam/pull/5640 to ignore the flaky
>> test. As JB is the release manager, I'l let him make the call on what to do
>> about it.
>> Best
>> -P.
>>
>> On Wed, Jun 13, 2018 at 3:34 PM Ahmet Altay <al...@google.com> wrote:
>>
>>> I would vote for second option, not a release blocker and disable the
>>> test in the release branch. My reasoning is:
>>> - ReferenceRunner is not yet the official alternative to existing direct
>>> runners.
>>> - It is bad to have flaky tests on the release branch, and we would not
>>> get good signal during validation.
>>>
>>> On Wed, Jun 13, 2018 at 3:14 PM, Pablo Estrada <pa...@google.com>
>>> wrote:
>>>
>>>> Hello all,
>>>> cherrypicks for the release branch seem to be going well, but thanks to
>>>> them we were able to surface a flaky test in the release branch. JIRA is
>>>> filed: https://issues.apache.org/jira/projects/BEAM/issues/BEAM-4558
>>>>
>>>> Given that test issue, I see the following options:
>>>> - Consider that this test is not a release blocker. Go ahead with RC2
>>>> after cherrypicks are brought in, or
>>>> - Consider that this test is not a release blocker, so we disable it
>>>> before cutting RC2.
>>>> - Consider this test a release blocker, and triage the bug for fixing.
>>>>
>>>> What do you think?
>>>>
>>>> Best
>>>> -P.
>>>>
>>>> On Wed, Jun 13, 2018 at 9:54 AM Pablo Estrada <pa...@google.com>
>>>> wrote:
>>>>
>>>>> Precommits for PR https://github.com/apache/beam/pull/5609 are now
>>>>> passing. For now I've simply set failOnWarning to false to cherrypick into
>>>>> the release, and fix in master later on.
>>>>> Best
>>>>> -P.
>>>>>
>>>>> On Wed, Jun 13, 2018 at 9:08 AM Scott Wegner <sw...@google.com>
>>>>> wrote:
>>>>>
>>>>>> From my understanding, the @SuppressFBWarnings usage is in a
>>>>>> dependency (ByteBuddy) rather than directly in our code; so we're not able
>>>>>> to modify the usage.
>>>>>>
>>>>>> Pablo, feel free to disable failOnWarning for the sdks-java-core
>>>>>> project temporarily. This isn't a major regression since we've only
>>>>>> recently made the change to enable it [1]. We can work separately on
>>>>>> figuring out how to resolve the warnings.
>>>>>>
>>>>>> [1] https://github.com/apache/beam/pull/5319
>>>>>>
>>>>>> On Tue, Jun 12, 2018 at 11:57 PM Tim Robertson <
>>>>>> timrobertson100@gmail.com> wrote:
>>>>>>
>>>>>>> Hi Pablo,
>>>>>>>
>>>>>>> I'm afraid I couldn't find one either... there is an issue about it
>>>>>>> [1] which is old so it doesn't look likely to be resolved either.
>>>>>>>
>>>>>>> If you have time (sorry I am a bit busy) could you please verify the
>>>>>>> version does work if you install that version locally? I know the maven
>>>>>>> version of that [2] but not sure on the gradle equivalent. If we know it
>>>>>>> works, we can then find a repository that fits ok with Apache/Beam policy.
>>>>>>>
>>>>>>> Alternatively, we could consider using a fully qualified reference
>>>>>>> (i.e. @edu.umd.cs.findbugs.annotations.SuppressWarnings) to the deprecated
>>>>>>> version and leave the dependency at the 1.3.9-1. I believe our general
>>>>>>> direction is to remove findbugs when errorprone covers all aspects so I
>>>>>>> *expect* this should be considered reasonable.
>>>>>>>
>>>>>>> I hope this helps,
>>>>>>> Tim
>>>>>>>
>>>>>>> [1] https://github.com/stephenc/findbugs-annotations/issues/4
>>>>>>> [2]
>>>>>>> https://maven.apache.org/guides/mini/guide-3rd-party-jars-local.html
>>>>>>>
>>>>>>> On Wed, Jun 13, 2018 at 8:39 AM, Pablo Estrada <pa...@google.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Tim,
>>>>>>>> you're right. Thanks for pointing that out. There's just one
>>>>>>>> problem that I'm running into now: The 3.0.1-1 version does not seem to be
>>>>>>>> available in Maven Central[1]. Looking at the website, I am not quite sure
>>>>>>>> if there's another repository where they do stage the newer versions?[2]
>>>>>>>>
>>>>>>>> -P
>>>>>>>>
>>>>>>>> [1]
>>>>>>>> https://repo.maven.apache.org/maven2/com/github/stephenc/findbugs/findbugs-annotations
>>>>>>>> /
>>>>>>>> [2] http://stephenc.github.io/findbugs-annotations/
>>>>>>>>
>>>>>>>> On Tue, Jun 12, 2018 at 11:10 PM Tim Robertson <
>>>>>>>> timrobertson100@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi Pablo,
>>>>>>>>>
>>>>>>>>> I took only a quick look.
>>>>>>>>>
>>>>>>>>> "- The JAR from the non-LGPL findbugs does not contain the
>>>>>>>>> SuppressFBWarnings annotation"
>>>>>>>>>
>>>>>>>>> Unless I misunderstand you it looks like SuppressFBWarnings was
>>>>>>>>> added in Stephen's version in this commit [1] which was
>>>>>>>>> introduced in version 2.0.3-1 - I've checked is in the 3.0.1-1 build [2]
>>>>>>>>> I notice in your commits [1] you've been exploring version 3.0.0
>>>>>>>>> already though... what happens when you use 3.0.1-1? It sounds like the
>>>>>>>>> wrong version is coming in rather than the annotation being missing.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Tim
>>>>>>>>>
>>>>>>>>> [1]
>>>>>>>>> https://github.com/stephenc/findbugs-annotations/commits/master/src/main/java/edu/umd/cs/findbugs/annotations/SuppressWarnings.java
>>>>>>>>> [2] https://github.com/stephenc/findbugs-annotations/releases
>>>>>>>>> [3]
>>>>>>>>> https://github.com/apache/beam/pull/5609/commits/32c7df706e970557f154ff6bc521b2e00f9d09ab
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Jun 13, 2018 at 2:37 AM, Pablo Estrada <pabloem@google.com
>>>>>>>>> > wrote:
>>>>>>>>>
>>>>>>>>>> Hi all,
>>>>>>>>>> I'll humbly declare that after wrestling with he build to stop
>>>>>>>>>> depending on the wrong findbugs_annotations, I feel somewhat lost. The
>>>>>>>>>> issue is actually quite small:
>>>>>>>>>>
>>>>>>>>>> - The JAR from the non-LGPL findbugs does not contain the
>>>>>>>>>> SuppressFBWarnings annotation. This means that when building, ByteBuddy
>>>>>>>>>> produces a few warnings (nothing critical).
>>>>>>>>>> - The easiest way to avoid this failure is to call
>>>>>>>>>> applyJavaNature(failOnWarning: false), but this would be bad, since we want
>>>>>>>>>> to keep a high standard for tasks like ErrorProne and FindBugs itself.
>>>>>>>>>> - So I find myself lost: How do we suppress trivial warnings
>>>>>>>>>> coming from missing annotations, and honor warnings coming from other
>>>>>>>>>> plugins?
>>>>>>>>>>
>>>>>>>>>> Any help / a PR from someone more capable would be appreciated.
>>>>>>>>>> Best
>>>>>>>>>> -P.
>>>>>>>>>>
>>>>>>>>>> On Tue, Jun 12, 2018 at 3:02 PM Ismaël Mejía <ie...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Yes, ok I was not aware it was already being addressed, nice.
>>>>>>>>>>> On Tue, Jun 12, 2018 at 11:56 PM Ahmet Altay <al...@google.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>> >
>>>>>>>>>>> > Ismaël,
>>>>>>>>>>> >
>>>>>>>>>>> > I believe Pablo's https://github.com/apache/beam/pull/5609 is
>>>>>>>>>>> fixing the issue by changing the findbugs back to
>>>>>>>>>>> "com.github.stephenc.findbugs". Is this what you are referring to?
>>>>>>>>>>> >
>>>>>>>>>>> > Ahmet
>>>>>>>>>>> >
>>>>>>>>>>> > On Tue, Jun 12, 2018 at 2:51 PM, Boyuan Zhang <
>>>>>>>>>>> boyuanz@google.com> wrote:
>>>>>>>>>>> >>
>>>>>>>>>>> >> Hey JB,
>>>>>>>>>>> >>
>>>>>>>>>>> >> I added some instructions about how to create python wheels
>>>>>>>>>>> in this PR: https://github.com/apache/beam-site/pull/467 . Hope
>>>>>>>>>>> it would be helpful.
>>>>>>>>>>> >>
>>>>>>>>>>> >> Boyuan
>>>>>>>>>>> >>
>>>>>>>>>>> >
>>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Got feedback? go/pabloem-feedback
>>>>>>>>>> <https://goto.google.com/pabloem-feedback>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>> Got feedback? go/pabloem-feedback
>>>>>>>> <https://goto.google.com/pabloem-feedback>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>> Got feedback? go/pabloem-feedback
>>>>> <https://goto.google.com/pabloem-feedback>
>>>>>
>>>> --
>>>> Got feedback? go/pabloem-feedback
>>>> <https://goto.google.com/pabloem-feedback>
>>>>
>>>
>>> --
>> Got feedback? go/pabloem-feedback
>> <https://goto.google.com/pabloem-feedback>
>>
>
Re: [CANCEL][VOTE] Apache Beam, version 2.5.0, release candidate #1
Posted by Udi Meiri <eh...@google.com>.
+1 to ignoring flaky test.
FYI there's a fourth cherrypick: https://github.com/apache/beam/pull/5624
On Wed, Jun 13, 2018 at 3:45 PM Pablo Estrada <pa...@google.com> wrote:
> Sent out https://github.com/apache/beam/pull/5640 to ignore the flaky
> test. As JB is the release manager, I'l let him make the call on what to do
> about it.
> Best
> -P.
>
> On Wed, Jun 13, 2018 at 3:34 PM Ahmet Altay <al...@google.com> wrote:
>
>> I would vote for second option, not a release blocker and disable the
>> test in the release branch. My reasoning is:
>> - ReferenceRunner is not yet the official alternative to existing direct
>> runners.
>> - It is bad to have flaky tests on the release branch, and we would not
>> get good signal during validation.
>>
>> On Wed, Jun 13, 2018 at 3:14 PM, Pablo Estrada <pa...@google.com>
>> wrote:
>>
>>> Hello all,
>>> cherrypicks for the release branch seem to be going well, but thanks to
>>> them we were able to surface a flaky test in the release branch. JIRA is
>>> filed: https://issues.apache.org/jira/projects/BEAM/issues/BEAM-4558
>>>
>>> Given that test issue, I see the following options:
>>> - Consider that this test is not a release blocker. Go ahead with RC2
>>> after cherrypicks are brought in, or
>>> - Consider that this test is not a release blocker, so we disable it
>>> before cutting RC2.
>>> - Consider this test a release blocker, and triage the bug for fixing.
>>>
>>> What do you think?
>>>
>>> Best
>>> -P.
>>>
>>> On Wed, Jun 13, 2018 at 9:54 AM Pablo Estrada <pa...@google.com>
>>> wrote:
>>>
>>>> Precommits for PR https://github.com/apache/beam/pull/5609 are now
>>>> passing. For now I've simply set failOnWarning to false to cherrypick into
>>>> the release, and fix in master later on.
>>>> Best
>>>> -P.
>>>>
>>>> On Wed, Jun 13, 2018 at 9:08 AM Scott Wegner <sw...@google.com>
>>>> wrote:
>>>>
>>>>> From my understanding, the @SuppressFBWarnings usage is in a
>>>>> dependency (ByteBuddy) rather than directly in our code; so we're not able
>>>>> to modify the usage.
>>>>>
>>>>> Pablo, feel free to disable failOnWarning for the sdks-java-core
>>>>> project temporarily. This isn't a major regression since we've only
>>>>> recently made the change to enable it [1]. We can work separately on
>>>>> figuring out how to resolve the warnings.
>>>>>
>>>>> [1] https://github.com/apache/beam/pull/5319
>>>>>
>>>>> On Tue, Jun 12, 2018 at 11:57 PM Tim Robertson <
>>>>> timrobertson100@gmail.com> wrote:
>>>>>
>>>>>> Hi Pablo,
>>>>>>
>>>>>> I'm afraid I couldn't find one either... there is an issue about it
>>>>>> [1] which is old so it doesn't look likely to be resolved either.
>>>>>>
>>>>>> If you have time (sorry I am a bit busy) could you please verify the
>>>>>> version does work if you install that version locally? I know the maven
>>>>>> version of that [2] but not sure on the gradle equivalent. If we know it
>>>>>> works, we can then find a repository that fits ok with Apache/Beam policy.
>>>>>>
>>>>>> Alternatively, we could consider using a fully qualified reference
>>>>>> (i.e. @edu.umd.cs.findbugs.annotations.SuppressWarnings) to the deprecated
>>>>>> version and leave the dependency at the 1.3.9-1. I believe our general
>>>>>> direction is to remove findbugs when errorprone covers all aspects so I
>>>>>> *expect* this should be considered reasonable.
>>>>>>
>>>>>> I hope this helps,
>>>>>> Tim
>>>>>>
>>>>>> [1] https://github.com/stephenc/findbugs-annotations/issues/4
>>>>>> [2]
>>>>>> https://maven.apache.org/guides/mini/guide-3rd-party-jars-local.html
>>>>>>
>>>>>> On Wed, Jun 13, 2018 at 8:39 AM, Pablo Estrada <pa...@google.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Tim,
>>>>>>> you're right. Thanks for pointing that out. There's just one problem
>>>>>>> that I'm running into now: The 3.0.1-1 version does not seem to be
>>>>>>> available in Maven Central[1]. Looking at the website, I am not quite sure
>>>>>>> if there's another repository where they do stage the newer versions?[2]
>>>>>>>
>>>>>>> -P
>>>>>>>
>>>>>>> [1]
>>>>>>> https://repo.maven.apache.org/maven2/com/github/stephenc/findbugs/findbugs-annotations
>>>>>>> /
>>>>>>> [2] http://stephenc.github.io/findbugs-annotations/
>>>>>>>
>>>>>>> On Tue, Jun 12, 2018 at 11:10 PM Tim Robertson <
>>>>>>> timrobertson100@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi Pablo,
>>>>>>>>
>>>>>>>> I took only a quick look.
>>>>>>>>
>>>>>>>> "- The JAR from the non-LGPL findbugs does not contain the
>>>>>>>> SuppressFBWarnings annotation"
>>>>>>>>
>>>>>>>> Unless I misunderstand you it looks like SuppressFBWarnings was
>>>>>>>> added in Stephen's version in this commit [1] which was introduced
>>>>>>>> in version 2.0.3-1 - I've checked is in the 3.0.1-1 build [2]
>>>>>>>> I notice in your commits [1] you've been exploring version 3.0.0
>>>>>>>> already though... what happens when you use 3.0.1-1? It sounds like the
>>>>>>>> wrong version is coming in rather than the annotation being missing.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Tim
>>>>>>>>
>>>>>>>> [1]
>>>>>>>> https://github.com/stephenc/findbugs-annotations/commits/master/src/main/java/edu/umd/cs/findbugs/annotations/SuppressWarnings.java
>>>>>>>> [2] https://github.com/stephenc/findbugs-annotations/releases
>>>>>>>> [3]
>>>>>>>> https://github.com/apache/beam/pull/5609/commits/32c7df706e970557f154ff6bc521b2e00f9d09ab
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Jun 13, 2018 at 2:37 AM, Pablo Estrada <pa...@google.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi all,
>>>>>>>>> I'll humbly declare that after wrestling with he build to stop
>>>>>>>>> depending on the wrong findbugs_annotations, I feel somewhat lost. The
>>>>>>>>> issue is actually quite small:
>>>>>>>>>
>>>>>>>>> - The JAR from the non-LGPL findbugs does not contain the
>>>>>>>>> SuppressFBWarnings annotation. This means that when building, ByteBuddy
>>>>>>>>> produces a few warnings (nothing critical).
>>>>>>>>> - The easiest way to avoid this failure is to call
>>>>>>>>> applyJavaNature(failOnWarning: false), but this would be bad, since we want
>>>>>>>>> to keep a high standard for tasks like ErrorProne and FindBugs itself.
>>>>>>>>> - So I find myself lost: How do we suppress trivial warnings
>>>>>>>>> coming from missing annotations, and honor warnings coming from other
>>>>>>>>> plugins?
>>>>>>>>>
>>>>>>>>> Any help / a PR from someone more capable would be appreciated.
>>>>>>>>> Best
>>>>>>>>> -P.
>>>>>>>>>
>>>>>>>>> On Tue, Jun 12, 2018 at 3:02 PM Ismaël Mejía <ie...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Yes, ok I was not aware it was already being addressed, nice.
>>>>>>>>>> On Tue, Jun 12, 2018 at 11:56 PM Ahmet Altay <al...@google.com>
>>>>>>>>>> wrote:
>>>>>>>>>> >
>>>>>>>>>> > Ismaël,
>>>>>>>>>> >
>>>>>>>>>> > I believe Pablo's https://github.com/apache/beam/pull/5609 is
>>>>>>>>>> fixing the issue by changing the findbugs back to
>>>>>>>>>> "com.github.stephenc.findbugs". Is this what you are referring to?
>>>>>>>>>> >
>>>>>>>>>> > Ahmet
>>>>>>>>>> >
>>>>>>>>>> > On Tue, Jun 12, 2018 at 2:51 PM, Boyuan Zhang <
>>>>>>>>>> boyuanz@google.com> wrote:
>>>>>>>>>> >>
>>>>>>>>>> >> Hey JB,
>>>>>>>>>> >>
>>>>>>>>>> >> I added some instructions about how to create python wheels in
>>>>>>>>>> this PR: https://github.com/apache/beam-site/pull/467 . Hope it
>>>>>>>>>> would be helpful.
>>>>>>>>>> >>
>>>>>>>>>> >> Boyuan
>>>>>>>>>> >>
>>>>>>>>>> >
>>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Got feedback? go/pabloem-feedback
>>>>>>>>> <https://goto.google.com/pabloem-feedback>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>> Got feedback? go/pabloem-feedback
>>>>>>> <https://goto.google.com/pabloem-feedback>
>>>>>>>
>>>>>>
>>>>>> --
>>>> Got feedback? go/pabloem-feedback
>>>> <https://goto.google.com/pabloem-feedback>
>>>>
>>> --
>>> Got feedback? go/pabloem-feedback
>>> <https://goto.google.com/pabloem-feedback>
>>>
>>
>> --
> Got feedback? go/pabloem-feedback
> <https://goto.google.com/pabloem-feedback>
>
Re: [CANCEL][VOTE] Apache Beam, version 2.5.0, release candidate #1
Posted by Pablo Estrada <pa...@google.com>.
Sent out https://github.com/apache/beam/pull/5640 to ignore the flaky test.
As JB is the release manager, I'l let him make the call on what to do about
it.
Best
-P.
On Wed, Jun 13, 2018 at 3:34 PM Ahmet Altay <al...@google.com> wrote:
> I would vote for second option, not a release blocker and disable the test
> in the release branch. My reasoning is:
> - ReferenceRunner is not yet the official alternative to existing direct
> runners.
> - It is bad to have flaky tests on the release branch, and we would not
> get good signal during validation.
>
> On Wed, Jun 13, 2018 at 3:14 PM, Pablo Estrada <pa...@google.com> wrote:
>
>> Hello all,
>> cherrypicks for the release branch seem to be going well, but thanks to
>> them we were able to surface a flaky test in the release branch. JIRA is
>> filed: https://issues.apache.org/jira/projects/BEAM/issues/BEAM-4558
>>
>> Given that test issue, I see the following options:
>> - Consider that this test is not a release blocker. Go ahead with RC2
>> after cherrypicks are brought in, or
>> - Consider that this test is not a release blocker, so we disable it
>> before cutting RC2.
>> - Consider this test a release blocker, and triage the bug for fixing.
>>
>> What do you think?
>>
>> Best
>> -P.
>>
>> On Wed, Jun 13, 2018 at 9:54 AM Pablo Estrada <pa...@google.com> wrote:
>>
>>> Precommits for PR https://github.com/apache/beam/pull/5609 are now
>>> passing. For now I've simply set failOnWarning to false to cherrypick into
>>> the release, and fix in master later on.
>>> Best
>>> -P.
>>>
>>> On Wed, Jun 13, 2018 at 9:08 AM Scott Wegner <sw...@google.com> wrote:
>>>
>>>> From my understanding, the @SuppressFBWarnings usage is in a dependency
>>>> (ByteBuddy) rather than directly in our code; so we're not able to modify
>>>> the usage.
>>>>
>>>> Pablo, feel free to disable failOnWarning for the sdks-java-core
>>>> project temporarily. This isn't a major regression since we've only
>>>> recently made the change to enable it [1]. We can work separately on
>>>> figuring out how to resolve the warnings.
>>>>
>>>> [1] https://github.com/apache/beam/pull/5319
>>>>
>>>> On Tue, Jun 12, 2018 at 11:57 PM Tim Robertson <
>>>> timrobertson100@gmail.com> wrote:
>>>>
>>>>> Hi Pablo,
>>>>>
>>>>> I'm afraid I couldn't find one either... there is an issue about it
>>>>> [1] which is old so it doesn't look likely to be resolved either.
>>>>>
>>>>> If you have time (sorry I am a bit busy) could you please verify the
>>>>> version does work if you install that version locally? I know the maven
>>>>> version of that [2] but not sure on the gradle equivalent. If we know it
>>>>> works, we can then find a repository that fits ok with Apache/Beam policy.
>>>>>
>>>>> Alternatively, we could consider using a fully qualified reference
>>>>> (i.e. @edu.umd.cs.findbugs.annotations.SuppressWarnings) to the deprecated
>>>>> version and leave the dependency at the 1.3.9-1. I believe our general
>>>>> direction is to remove findbugs when errorprone covers all aspects so I
>>>>> *expect* this should be considered reasonable.
>>>>>
>>>>> I hope this helps,
>>>>> Tim
>>>>>
>>>>> [1] https://github.com/stephenc/findbugs-annotations/issues/4
>>>>> [2]
>>>>> https://maven.apache.org/guides/mini/guide-3rd-party-jars-local.html
>>>>>
>>>>> On Wed, Jun 13, 2018 at 8:39 AM, Pablo Estrada <pa...@google.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Tim,
>>>>>> you're right. Thanks for pointing that out. There's just one problem
>>>>>> that I'm running into now: The 3.0.1-1 version does not seem to be
>>>>>> available in Maven Central[1]. Looking at the website, I am not quite sure
>>>>>> if there's another repository where they do stage the newer versions?[2]
>>>>>>
>>>>>> -P
>>>>>>
>>>>>> [1]
>>>>>> https://repo.maven.apache.org/maven2/com/github/stephenc/findbugs/findbugs-annotations
>>>>>> /
>>>>>> [2] http://stephenc.github.io/findbugs-annotations/
>>>>>>
>>>>>> On Tue, Jun 12, 2018 at 11:10 PM Tim Robertson <
>>>>>> timrobertson100@gmail.com> wrote:
>>>>>>
>>>>>>> Hi Pablo,
>>>>>>>
>>>>>>> I took only a quick look.
>>>>>>>
>>>>>>> "- The JAR from the non-LGPL findbugs does not contain the
>>>>>>> SuppressFBWarnings annotation"
>>>>>>>
>>>>>>> Unless I misunderstand you it looks like SuppressFBWarnings was
>>>>>>> added in Stephen's version in this commit [1] which was introduced
>>>>>>> in version 2.0.3-1 - I've checked is in the 3.0.1-1 build [2]
>>>>>>> I notice in your commits [1] you've been exploring version 3.0.0
>>>>>>> already though... what happens when you use 3.0.1-1? It sounds like the
>>>>>>> wrong version is coming in rather than the annotation being missing.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Tim
>>>>>>>
>>>>>>> [1]
>>>>>>> https://github.com/stephenc/findbugs-annotations/commits/master/src/main/java/edu/umd/cs/findbugs/annotations/SuppressWarnings.java
>>>>>>> [2] https://github.com/stephenc/findbugs-annotations/releases
>>>>>>> [3]
>>>>>>> https://github.com/apache/beam/pull/5609/commits/32c7df706e970557f154ff6bc521b2e00f9d09ab
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Jun 13, 2018 at 2:37 AM, Pablo Estrada <pa...@google.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi all,
>>>>>>>> I'll humbly declare that after wrestling with he build to stop
>>>>>>>> depending on the wrong findbugs_annotations, I feel somewhat lost. The
>>>>>>>> issue is actually quite small:
>>>>>>>>
>>>>>>>> - The JAR from the non-LGPL findbugs does not contain the
>>>>>>>> SuppressFBWarnings annotation. This means that when building, ByteBuddy
>>>>>>>> produces a few warnings (nothing critical).
>>>>>>>> - The easiest way to avoid this failure is to call
>>>>>>>> applyJavaNature(failOnWarning: false), but this would be bad, since we want
>>>>>>>> to keep a high standard for tasks like ErrorProne and FindBugs itself.
>>>>>>>> - So I find myself lost: How do we suppress trivial warnings coming
>>>>>>>> from missing annotations, and honor warnings coming from other plugins?
>>>>>>>>
>>>>>>>> Any help / a PR from someone more capable would be appreciated.
>>>>>>>> Best
>>>>>>>> -P.
>>>>>>>>
>>>>>>>> On Tue, Jun 12, 2018 at 3:02 PM Ismaël Mejía <ie...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Yes, ok I was not aware it was already being addressed, nice.
>>>>>>>>> On Tue, Jun 12, 2018 at 11:56 PM Ahmet Altay <al...@google.com>
>>>>>>>>> wrote:
>>>>>>>>> >
>>>>>>>>> > Ismaël,
>>>>>>>>> >
>>>>>>>>> > I believe Pablo's https://github.com/apache/beam/pull/5609 is
>>>>>>>>> fixing the issue by changing the findbugs back to
>>>>>>>>> "com.github.stephenc.findbugs". Is this what you are referring to?
>>>>>>>>> >
>>>>>>>>> > Ahmet
>>>>>>>>> >
>>>>>>>>> > On Tue, Jun 12, 2018 at 2:51 PM, Boyuan Zhang <
>>>>>>>>> boyuanz@google.com> wrote:
>>>>>>>>> >>
>>>>>>>>> >> Hey JB,
>>>>>>>>> >>
>>>>>>>>> >> I added some instructions about how to create python wheels in
>>>>>>>>> this PR: https://github.com/apache/beam-site/pull/467 . Hope it
>>>>>>>>> would be helpful.
>>>>>>>>> >>
>>>>>>>>> >> Boyuan
>>>>>>>>> >>
>>>>>>>>> >
>>>>>>>>>
>>>>>>>> --
>>>>>>>> Got feedback? go/pabloem-feedback
>>>>>>>> <https://goto.google.com/pabloem-feedback>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>> Got feedback? go/pabloem-feedback
>>>>>> <https://goto.google.com/pabloem-feedback>
>>>>>>
>>>>>
>>>>> --
>>> Got feedback? go/pabloem-feedback
>>> <https://goto.google.com/pabloem-feedback>
>>>
>> --
>> Got feedback? go/pabloem-feedback
>> <https://goto.google.com/pabloem-feedback>
>>
>
> --
Got feedback? go/pabloem-feedback
Re: [CANCEL][VOTE] Apache Beam, version 2.5.0, release candidate #1
Posted by Ahmet Altay <al...@google.com>.
I would vote for second option, not a release blocker and disable the test
in the release branch. My reasoning is:
- ReferenceRunner is not yet the official alternative to existing direct
runners.
- It is bad to have flaky tests on the release branch, and we would not get
good signal during validation.
On Wed, Jun 13, 2018 at 3:14 PM, Pablo Estrada <pa...@google.com> wrote:
> Hello all,
> cherrypicks for the release branch seem to be going well, but thanks to
> them we were able to surface a flaky test in the release branch. JIRA is
> filed: https://issues.apache.org/jira/projects/BEAM/issues/BEAM-4558
>
> Given that test issue, I see the following options:
> - Consider that this test is not a release blocker. Go ahead with RC2
> after cherrypicks are brought in, or
> - Consider that this test is not a release blocker, so we disable it
> before cutting RC2.
> - Consider this test a release blocker, and triage the bug for fixing.
>
> What do you think?
>
> Best
> -P.
>
> On Wed, Jun 13, 2018 at 9:54 AM Pablo Estrada <pa...@google.com> wrote:
>
>> Precommits for PR https://github.com/apache/beam/pull/5609 are now
>> passing. For now I've simply set failOnWarning to false to cherrypick into
>> the release, and fix in master later on.
>> Best
>> -P.
>>
>> On Wed, Jun 13, 2018 at 9:08 AM Scott Wegner <sw...@google.com> wrote:
>>
>>> From my understanding, the @SuppressFBWarnings usage is in a dependency
>>> (ByteBuddy) rather than directly in our code; so we're not able to modify
>>> the usage.
>>>
>>> Pablo, feel free to disable failOnWarning for the sdks-java-core project
>>> temporarily. This isn't a major regression since we've only recently made
>>> the change to enable it [1]. We can work separately on figuring out how to
>>> resolve the warnings.
>>>
>>> [1] https://github.com/apache/beam/pull/5319
>>>
>>> On Tue, Jun 12, 2018 at 11:57 PM Tim Robertson <
>>> timrobertson100@gmail.com> wrote:
>>>
>>>> Hi Pablo,
>>>>
>>>> I'm afraid I couldn't find one either... there is an issue about it [1]
>>>> which is old so it doesn't look likely to be resolved either.
>>>>
>>>> If you have time (sorry I am a bit busy) could you please verify the
>>>> version does work if you install that version locally? I know the maven
>>>> version of that [2] but not sure on the gradle equivalent. If we know it
>>>> works, we can then find a repository that fits ok with Apache/Beam policy.
>>>>
>>>> Alternatively, we could consider using a fully qualified reference
>>>> (i.e. @edu.umd.cs.findbugs.annotations.SuppressWarnings) to the
>>>> deprecated version and leave the dependency at the 1.3.9-1. I believe our
>>>> general direction is to remove findbugs when errorprone covers all aspects
>>>> so I *expect* this should be considered reasonable.
>>>>
>>>> I hope this helps,
>>>> Tim
>>>>
>>>> [1] https://github.com/stephenc/findbugs-annotations/issues/4
>>>> [2] https://maven.apache.org/guides/mini/guide-3rd-party-
>>>> jars-local.html
>>>>
>>>> On Wed, Jun 13, 2018 at 8:39 AM, Pablo Estrada <pa...@google.com>
>>>> wrote:
>>>>
>>>>> Hi Tim,
>>>>> you're right. Thanks for pointing that out. There's just one problem
>>>>> that I'm running into now: The 3.0.1-1 version does not seem to be
>>>>> available in Maven Central[1]. Looking at the website, I am not quite sure
>>>>> if there's another repository where they do stage the newer versions?[2]
>>>>>
>>>>> -P
>>>>>
>>>>> [1] https://repo.maven.apache.org/maven2/com/github/
>>>>> stephenc/findbugs/findbugs-annotations/
>>>>> [2] http://stephenc.github.io/findbugs-annotations/
>>>>>
>>>>> On Tue, Jun 12, 2018 at 11:10 PM Tim Robertson <
>>>>> timrobertson100@gmail.com> wrote:
>>>>>
>>>>>> Hi Pablo,
>>>>>>
>>>>>> I took only a quick look.
>>>>>>
>>>>>> "- The JAR from the non-LGPL findbugs does not contain the
>>>>>> SuppressFBWarnings annotation"
>>>>>>
>>>>>> Unless I misunderstand you it looks like SuppressFBWarnings was added
>>>>>> in Stephen's version in this commit [1] which was introduced in
>>>>>> version 2.0.3-1 - I've checked is in the 3.0.1-1 build [2]
>>>>>> I notice in your commits [1] you've been exploring version 3.0.0
>>>>>> already though... what happens when you use 3.0.1-1? It sounds like the
>>>>>> wrong version is coming in rather than the annotation being missing.
>>>>>>
>>>>>> Thanks,
>>>>>> Tim
>>>>>>
>>>>>> [1] https://github.com/stephenc/findbugs-annotations/
>>>>>> commits/master/src/main/java/edu/umd/cs/findbugs/
>>>>>> annotations/SuppressWarnings.java
>>>>>> [2] https://github.com/stephenc/findbugs-annotations/releases
>>>>>> [3] https://github.com/apache/beam/pull/5609/commits/
>>>>>> 32c7df706e970557f154ff6bc521b2e00f9d09ab
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Jun 13, 2018 at 2:37 AM, Pablo Estrada <pa...@google.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi all,
>>>>>>> I'll humbly declare that after wrestling with he build to stop
>>>>>>> depending on the wrong findbugs_annotations, I feel somewhat lost. The
>>>>>>> issue is actually quite small:
>>>>>>>
>>>>>>> - The JAR from the non-LGPL findbugs does not contain the
>>>>>>> SuppressFBWarnings annotation. This means that when building, ByteBuddy
>>>>>>> produces a few warnings (nothing critical).
>>>>>>> - The easiest way to avoid this failure is to call
>>>>>>> applyJavaNature(failOnWarning: false), but this would be bad, since we want
>>>>>>> to keep a high standard for tasks like ErrorProne and FindBugs itself.
>>>>>>> - So I find myself lost: How do we suppress trivial warnings coming
>>>>>>> from missing annotations, and honor warnings coming from other plugins?
>>>>>>>
>>>>>>> Any help / a PR from someone more capable would be appreciated.
>>>>>>> Best
>>>>>>> -P.
>>>>>>>
>>>>>>> On Tue, Jun 12, 2018 at 3:02 PM Ismaël Mejía <ie...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Yes, ok I was not aware it was already being addressed, nice.
>>>>>>>> On Tue, Jun 12, 2018 at 11:56 PM Ahmet Altay <al...@google.com>
>>>>>>>> wrote:
>>>>>>>> >
>>>>>>>> > Ismaël,
>>>>>>>> >
>>>>>>>> > I believe Pablo's https://github.com/apache/beam/pull/5609 is
>>>>>>>> fixing the issue by changing the findbugs back to
>>>>>>>> "com.github.stephenc.findbugs". Is this what you are referring to?
>>>>>>>> >
>>>>>>>> > Ahmet
>>>>>>>> >
>>>>>>>> > On Tue, Jun 12, 2018 at 2:51 PM, Boyuan Zhang <bo...@google.com>
>>>>>>>> wrote:
>>>>>>>> >>
>>>>>>>> >> Hey JB,
>>>>>>>> >>
>>>>>>>> >> I added some instructions about how to create python wheels in
>>>>>>>> this PR: https://github.com/apache/beam-site/pull/467 . Hope it
>>>>>>>> would be helpful.
>>>>>>>> >>
>>>>>>>> >> Boyuan
>>>>>>>> >>
>>>>>>>> >
>>>>>>>>
>>>>>>> --
>>>>>>> Got feedback? go/pabloem-feedback
>>>>>>> <https://goto.google.com/pabloem-feedback>
>>>>>>>
>>>>>>
>>>>>> --
>>>>> Got feedback? go/pabloem-feedback
>>>>> <https://goto.google.com/pabloem-feedback>
>>>>>
>>>>
>>>> --
>> Got feedback? go/pabloem-feedback
>> <https://goto.google.com/pabloem-feedback>
>>
> --
> Got feedback? go/pabloem-feedback
> <https://goto.google.com/pabloem-feedback>
>
Re: [CANCEL][VOTE] Apache Beam, version 2.5.0, release candidate #1
Posted by Pablo Estrada <pa...@google.com>.
Hello all,
cherrypicks for the release branch seem to be going well, but thanks to
them we were able to surface a flaky test in the release branch. JIRA is
filed: https://issues.apache.org/jira/projects/BEAM/issues/BEAM-4558
Given that test issue, I see the following options:
- Consider that this test is not a release blocker. Go ahead with RC2 after
cherrypicks are brought in, or
- Consider that this test is not a release blocker, so we disable it before
cutting RC2.
- Consider this test a release blocker, and triage the bug for fixing.
What do you think?
Best
-P.
On Wed, Jun 13, 2018 at 9:54 AM Pablo Estrada <pa...@google.com> wrote:
> Precommits for PR https://github.com/apache/beam/pull/5609 are now
> passing. For now I've simply set failOnWarning to false to cherrypick into
> the release, and fix in master later on.
> Best
> -P.
>
> On Wed, Jun 13, 2018 at 9:08 AM Scott Wegner <sw...@google.com> wrote:
>
>> From my understanding, the @SuppressFBWarnings usage is in a dependency
>> (ByteBuddy) rather than directly in our code; so we're not able to modify
>> the usage.
>>
>> Pablo, feel free to disable failOnWarning for the sdks-java-core project
>> temporarily. This isn't a major regression since we've only recently made
>> the change to enable it [1]. We can work separately on figuring out how to
>> resolve the warnings.
>>
>> [1] https://github.com/apache/beam/pull/5319
>>
>> On Tue, Jun 12, 2018 at 11:57 PM Tim Robertson <ti...@gmail.com>
>> wrote:
>>
>>> Hi Pablo,
>>>
>>> I'm afraid I couldn't find one either... there is an issue about it [1]
>>> which is old so it doesn't look likely to be resolved either.
>>>
>>> If you have time (sorry I am a bit busy) could you please verify the
>>> version does work if you install that version locally? I know the maven
>>> version of that [2] but not sure on the gradle equivalent. If we know it
>>> works, we can then find a repository that fits ok with Apache/Beam policy.
>>>
>>> Alternatively, we could consider using a fully qualified reference (i.e.
>>> @edu.umd.cs.findbugs.annotations.SuppressWarnings) to the deprecated
>>> version and leave the dependency at the 1.3.9-1. I believe our general
>>> direction is to remove findbugs when errorprone covers all aspects so I
>>> *expect* this should be considered reasonable.
>>>
>>> I hope this helps,
>>> Tim
>>>
>>> [1] https://github.com/stephenc/findbugs-annotations/issues/4
>>> [2] https://maven.apache.org/guides/mini/guide-3rd-party-jars-local.html
>>>
>>> On Wed, Jun 13, 2018 at 8:39 AM, Pablo Estrada <pa...@google.com>
>>> wrote:
>>>
>>>> Hi Tim,
>>>> you're right. Thanks for pointing that out. There's just one problem
>>>> that I'm running into now: The 3.0.1-1 version does not seem to be
>>>> available in Maven Central[1]. Looking at the website, I am not quite sure
>>>> if there's another repository where they do stage the newer versions?[2]
>>>>
>>>> -P
>>>>
>>>> [1]
>>>> https://repo.maven.apache.org/maven2/com/github/stephenc/findbugs/findbugs-annotations
>>>> /
>>>> [2] http://stephenc.github.io/findbugs-annotations/
>>>>
>>>> On Tue, Jun 12, 2018 at 11:10 PM Tim Robertson <
>>>> timrobertson100@gmail.com> wrote:
>>>>
>>>>> Hi Pablo,
>>>>>
>>>>> I took only a quick look.
>>>>>
>>>>> "- The JAR from the non-LGPL findbugs does not contain the
>>>>> SuppressFBWarnings annotation"
>>>>>
>>>>> Unless I misunderstand you it looks like SuppressFBWarnings was added
>>>>> in Stephen's version in this commit [1] which was introduced in
>>>>> version 2.0.3-1 - I've checked is in the 3.0.1-1 build [2]
>>>>> I notice in your commits [1] you've been exploring version 3.0.0
>>>>> already though... what happens when you use 3.0.1-1? It sounds like the
>>>>> wrong version is coming in rather than the annotation being missing.
>>>>>
>>>>> Thanks,
>>>>> Tim
>>>>>
>>>>> [1]
>>>>> https://github.com/stephenc/findbugs-annotations/commits/master/src/main/java/edu/umd/cs/findbugs/annotations/SuppressWarnings.java
>>>>> [2] https://github.com/stephenc/findbugs-annotations/releases
>>>>> [3]
>>>>> https://github.com/apache/beam/pull/5609/commits/32c7df706e970557f154ff6bc521b2e00f9d09ab
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Jun 13, 2018 at 2:37 AM, Pablo Estrada <pa...@google.com>
>>>>> wrote:
>>>>>
>>>>>> Hi all,
>>>>>> I'll humbly declare that after wrestling with he build to stop
>>>>>> depending on the wrong findbugs_annotations, I feel somewhat lost. The
>>>>>> issue is actually quite small:
>>>>>>
>>>>>> - The JAR from the non-LGPL findbugs does not contain the
>>>>>> SuppressFBWarnings annotation. This means that when building, ByteBuddy
>>>>>> produces a few warnings (nothing critical).
>>>>>> - The easiest way to avoid this failure is to call
>>>>>> applyJavaNature(failOnWarning: false), but this would be bad, since we want
>>>>>> to keep a high standard for tasks like ErrorProne and FindBugs itself.
>>>>>> - So I find myself lost: How do we suppress trivial warnings coming
>>>>>> from missing annotations, and honor warnings coming from other plugins?
>>>>>>
>>>>>> Any help / a PR from someone more capable would be appreciated.
>>>>>> Best
>>>>>> -P.
>>>>>>
>>>>>> On Tue, Jun 12, 2018 at 3:02 PM Ismaël Mejía <ie...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Yes, ok I was not aware it was already being addressed, nice.
>>>>>>> On Tue, Jun 12, 2018 at 11:56 PM Ahmet Altay <al...@google.com>
>>>>>>> wrote:
>>>>>>> >
>>>>>>> > Ismaël,
>>>>>>> >
>>>>>>> > I believe Pablo's https://github.com/apache/beam/pull/5609 is
>>>>>>> fixing the issue by changing the findbugs back to
>>>>>>> "com.github.stephenc.findbugs". Is this what you are referring to?
>>>>>>> >
>>>>>>> > Ahmet
>>>>>>> >
>>>>>>> > On Tue, Jun 12, 2018 at 2:51 PM, Boyuan Zhang <bo...@google.com>
>>>>>>> wrote:
>>>>>>> >>
>>>>>>> >> Hey JB,
>>>>>>> >>
>>>>>>> >> I added some instructions about how to create python wheels in
>>>>>>> this PR: https://github.com/apache/beam-site/pull/467 . Hope it
>>>>>>> would be helpful.
>>>>>>> >>
>>>>>>> >> Boyuan
>>>>>>> >>
>>>>>>> >
>>>>>>>
>>>>>> --
>>>>>> Got feedback? go/pabloem-feedback
>>>>>> <https://goto.google.com/pabloem-feedback>
>>>>>>
>>>>>
>>>>> --
>>>> Got feedback? go/pabloem-feedback
>>>> <https://goto.google.com/pabloem-feedback>
>>>>
>>>
>>> --
> Got feedback? go/pabloem-feedback
> <https://goto.google.com/pabloem-feedback>
>
--
Got feedback? go/pabloem-feedback
Re: [CANCEL][VOTE] Apache Beam, version 2.5.0, release candidate #1
Posted by Boyuan Zhang <bo...@google.com>.
Hey all,
Currently we have 3 PRs supposed to be cherrypicked into RC2:
Pablo: <goog_1529729135>https://github.com/apache/beam/pull/5609 (merged)
Udi: https://github.com/apache/beam/pull/5607 (open)
Charles: <goog_1529729128>https://github.com/apache/beam/pull/5636 (open)
Boyuan
On Wed, Jun 13, 2018 at 9:54 AM Pablo Estrada <pa...@google.com> wrote:
> Precommits for PR https://github.com/apache/beam/pull/5609 are now
> passing. For now I've simply set failOnWarning to false to cherrypick into
> the release, and fix in master later on.
> Best
> -P.
>
> On Wed, Jun 13, 2018 at 9:08 AM Scott Wegner <sw...@google.com> wrote:
>
>> From my understanding, the @SuppressFBWarnings usage is in a dependency
>> (ByteBuddy) rather than directly in our code; so we're not able to modify
>> the usage.
>>
>> Pablo, feel free to disable failOnWarning for the sdks-java-core project
>> temporarily. This isn't a major regression since we've only recently made
>> the change to enable it [1]. We can work separately on figuring out how to
>> resolve the warnings.
>>
>> [1] https://github.com/apache/beam/pull/5319
>>
>> On Tue, Jun 12, 2018 at 11:57 PM Tim Robertson <ti...@gmail.com>
>> wrote:
>>
>>> Hi Pablo,
>>>
>>> I'm afraid I couldn't find one either... there is an issue about it [1]
>>> which is old so it doesn't look likely to be resolved either.
>>>
>>> If you have time (sorry I am a bit busy) could you please verify the
>>> version does work if you install that version locally? I know the maven
>>> version of that [2] but not sure on the gradle equivalent. If we know it
>>> works, we can then find a repository that fits ok with Apache/Beam policy.
>>>
>>> Alternatively, we could consider using a fully qualified reference (i.e.
>>> @edu.umd.cs.findbugs.annotations.SuppressWarnings) to the deprecated
>>> version and leave the dependency at the 1.3.9-1. I believe our general
>>> direction is to remove findbugs when errorprone covers all aspects so I
>>> *expect* this should be considered reasonable.
>>>
>>> I hope this helps,
>>> Tim
>>>
>>> [1] https://github.com/stephenc/findbugs-annotations/issues/4
>>> [2] https://maven.apache.org/guides/mini/guide-3rd-party-jars-local.html
>>>
>>> On Wed, Jun 13, 2018 at 8:39 AM, Pablo Estrada <pa...@google.com>
>>> wrote:
>>>
>>>> Hi Tim,
>>>> you're right. Thanks for pointing that out. There's just one problem
>>>> that I'm running into now: The 3.0.1-1 version does not seem to be
>>>> available in Maven Central[1]. Looking at the website, I am not quite sure
>>>> if there's another repository where they do stage the newer versions?[2]
>>>>
>>>> -P
>>>>
>>>> [1]
>>>> https://repo.maven.apache.org/maven2/com/github/stephenc/findbugs/findbugs-annotations
>>>> /
>>>> [2] http://stephenc.github.io/findbugs-annotations/
>>>>
>>>> On Tue, Jun 12, 2018 at 11:10 PM Tim Robertson <
>>>> timrobertson100@gmail.com> wrote:
>>>>
>>>>> Hi Pablo,
>>>>>
>>>>> I took only a quick look.
>>>>>
>>>>> "- The JAR from the non-LGPL findbugs does not contain the
>>>>> SuppressFBWarnings annotation"
>>>>>
>>>>> Unless I misunderstand you it looks like SuppressFBWarnings was added
>>>>> in Stephen's version in this commit [1] which was introduced in
>>>>> version 2.0.3-1 - I've checked is in the 3.0.1-1 build [2]
>>>>> I notice in your commits [1] you've been exploring version 3.0.0
>>>>> already though... what happens when you use 3.0.1-1? It sounds like the
>>>>> wrong version is coming in rather than the annotation being missing.
>>>>>
>>>>> Thanks,
>>>>> Tim
>>>>>
>>>>> [1]
>>>>> https://github.com/stephenc/findbugs-annotations/commits/master/src/main/java/edu/umd/cs/findbugs/annotations/SuppressWarnings.java
>>>>> [2] https://github.com/stephenc/findbugs-annotations/releases
>>>>> [3]
>>>>> https://github.com/apache/beam/pull/5609/commits/32c7df706e970557f154ff6bc521b2e00f9d09ab
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Jun 13, 2018 at 2:37 AM, Pablo Estrada <pa...@google.com>
>>>>> wrote:
>>>>>
>>>>>> Hi all,
>>>>>> I'll humbly declare that after wrestling with he build to stop
>>>>>> depending on the wrong findbugs_annotations, I feel somewhat lost. The
>>>>>> issue is actually quite small:
>>>>>>
>>>>>> - The JAR from the non-LGPL findbugs does not contain the
>>>>>> SuppressFBWarnings annotation. This means that when building, ByteBuddy
>>>>>> produces a few warnings (nothing critical).
>>>>>> - The easiest way to avoid this failure is to call
>>>>>> applyJavaNature(failOnWarning: false), but this would be bad, since we want
>>>>>> to keep a high standard for tasks like ErrorProne and FindBugs itself.
>>>>>> - So I find myself lost: How do we suppress trivial warnings coming
>>>>>> from missing annotations, and honor warnings coming from other plugins?
>>>>>>
>>>>>> Any help / a PR from someone more capable would be appreciated.
>>>>>> Best
>>>>>> -P.
>>>>>>
>>>>>> On Tue, Jun 12, 2018 at 3:02 PM Ismaël Mejía <ie...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Yes, ok I was not aware it was already being addressed, nice.
>>>>>>> On Tue, Jun 12, 2018 at 11:56 PM Ahmet Altay <al...@google.com>
>>>>>>> wrote:
>>>>>>> >
>>>>>>> > Ismaël,
>>>>>>> >
>>>>>>> > I believe Pablo's https://github.com/apache/beam/pull/5609 is
>>>>>>> fixing the issue by changing the findbugs back to
>>>>>>> "com.github.stephenc.findbugs". Is this what you are referring to?
>>>>>>> >
>>>>>>> > Ahmet
>>>>>>> >
>>>>>>> > On Tue, Jun 12, 2018 at 2:51 PM, Boyuan Zhang <bo...@google.com>
>>>>>>> wrote:
>>>>>>> >>
>>>>>>> >> Hey JB,
>>>>>>> >>
>>>>>>> >> I added some instructions about how to create python wheels in
>>>>>>> this PR: https://github.com/apache/beam-site/pull/467 . Hope it
>>>>>>> would be helpful.
>>>>>>> >>
>>>>>>> >> Boyuan
>>>>>>> >>
>>>>>>> >
>>>>>>>
>>>>>> --
>>>>>> Got feedback? go/pabloem-feedback
>>>>>> <https://goto.google.com/pabloem-feedback>
>>>>>>
>>>>>
>>>>> --
>>>> Got feedback? go/pabloem-feedback
>>>> <https://goto.google.com/pabloem-feedback>
>>>>
>>>
>>> --
> Got feedback? go/pabloem-feedback
> <https://goto.google.com/pabloem-feedback>
>
Re: [CANCEL][VOTE] Apache Beam, version 2.5.0, release candidate #1
Posted by Pablo Estrada <pa...@google.com>.
Precommits for PR https://github.com/apache/beam/pull/5609 are now passing.
For now I've simply set failOnWarning to false to cherrypick into the
release, and fix in master later on.
Best
-P.
On Wed, Jun 13, 2018 at 9:08 AM Scott Wegner <sw...@google.com> wrote:
> From my understanding, the @SuppressFBWarnings usage is in a dependency
> (ByteBuddy) rather than directly in our code; so we're not able to modify
> the usage.
>
> Pablo, feel free to disable failOnWarning for the sdks-java-core project
> temporarily. This isn't a major regression since we've only recently made
> the change to enable it [1]. We can work separately on figuring out how to
> resolve the warnings.
>
> [1] https://github.com/apache/beam/pull/5319
>
> On Tue, Jun 12, 2018 at 11:57 PM Tim Robertson <ti...@gmail.com>
> wrote:
>
>> Hi Pablo,
>>
>> I'm afraid I couldn't find one either... there is an issue about it [1]
>> which is old so it doesn't look likely to be resolved either.
>>
>> If you have time (sorry I am a bit busy) could you please verify the
>> version does work if you install that version locally? I know the maven
>> version of that [2] but not sure on the gradle equivalent. If we know it
>> works, we can then find a repository that fits ok with Apache/Beam policy.
>>
>> Alternatively, we could consider using a fully qualified reference (i.e.
>> @edu.umd.cs.findbugs.annotations.SuppressWarnings) to the deprecated
>> version and leave the dependency at the 1.3.9-1. I believe our general
>> direction is to remove findbugs when errorprone covers all aspects so I
>> *expect* this should be considered reasonable.
>>
>> I hope this helps,
>> Tim
>>
>> [1] https://github.com/stephenc/findbugs-annotations/issues/4
>> [2] https://maven.apache.org/guides/mini/guide-3rd-party-jars-local.html
>>
>> On Wed, Jun 13, 2018 at 8:39 AM, Pablo Estrada <pa...@google.com>
>> wrote:
>>
>>> Hi Tim,
>>> you're right. Thanks for pointing that out. There's just one problem
>>> that I'm running into now: The 3.0.1-1 version does not seem to be
>>> available in Maven Central[1]. Looking at the website, I am not quite sure
>>> if there's another repository where they do stage the newer versions?[2]
>>>
>>> -P
>>>
>>> [1]
>>> https://repo.maven.apache.org/maven2/com/github/stephenc/findbugs/findbugs-annotations
>>> /
>>> [2] http://stephenc.github.io/findbugs-annotations/
>>>
>>> On Tue, Jun 12, 2018 at 11:10 PM Tim Robertson <
>>> timrobertson100@gmail.com> wrote:
>>>
>>>> Hi Pablo,
>>>>
>>>> I took only a quick look.
>>>>
>>>> "- The JAR from the non-LGPL findbugs does not contain the
>>>> SuppressFBWarnings annotation"
>>>>
>>>> Unless I misunderstand you it looks like SuppressFBWarnings was added
>>>> in Stephen's version in this commit [1] which was introduced in
>>>> version 2.0.3-1 - I've checked is in the 3.0.1-1 build [2]
>>>> I notice in your commits [1] you've been exploring version 3.0.0
>>>> already though... what happens when you use 3.0.1-1? It sounds like the
>>>> wrong version is coming in rather than the annotation being missing.
>>>>
>>>> Thanks,
>>>> Tim
>>>>
>>>> [1]
>>>> https://github.com/stephenc/findbugs-annotations/commits/master/src/main/java/edu/umd/cs/findbugs/annotations/SuppressWarnings.java
>>>> [2] https://github.com/stephenc/findbugs-annotations/releases
>>>> [3]
>>>> https://github.com/apache/beam/pull/5609/commits/32c7df706e970557f154ff6bc521b2e00f9d09ab
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Wed, Jun 13, 2018 at 2:37 AM, Pablo Estrada <pa...@google.com>
>>>> wrote:
>>>>
>>>>> Hi all,
>>>>> I'll humbly declare that after wrestling with he build to stop
>>>>> depending on the wrong findbugs_annotations, I feel somewhat lost. The
>>>>> issue is actually quite small:
>>>>>
>>>>> - The JAR from the non-LGPL findbugs does not contain the
>>>>> SuppressFBWarnings annotation. This means that when building, ByteBuddy
>>>>> produces a few warnings (nothing critical).
>>>>> - The easiest way to avoid this failure is to call
>>>>> applyJavaNature(failOnWarning: false), but this would be bad, since we want
>>>>> to keep a high standard for tasks like ErrorProne and FindBugs itself.
>>>>> - So I find myself lost: How do we suppress trivial warnings coming
>>>>> from missing annotations, and honor warnings coming from other plugins?
>>>>>
>>>>> Any help / a PR from someone more capable would be appreciated.
>>>>> Best
>>>>> -P.
>>>>>
>>>>> On Tue, Jun 12, 2018 at 3:02 PM Ismaël Mejía <ie...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Yes, ok I was not aware it was already being addressed, nice.
>>>>>> On Tue, Jun 12, 2018 at 11:56 PM Ahmet Altay <al...@google.com>
>>>>>> wrote:
>>>>>> >
>>>>>> > Ismaël,
>>>>>> >
>>>>>> > I believe Pablo's https://github.com/apache/beam/pull/5609 is
>>>>>> fixing the issue by changing the findbugs back to
>>>>>> "com.github.stephenc.findbugs". Is this what you are referring to?
>>>>>> >
>>>>>> > Ahmet
>>>>>> >
>>>>>> > On Tue, Jun 12, 2018 at 2:51 PM, Boyuan Zhang <bo...@google.com>
>>>>>> wrote:
>>>>>> >>
>>>>>> >> Hey JB,
>>>>>> >>
>>>>>> >> I added some instructions about how to create python wheels in
>>>>>> this PR: https://github.com/apache/beam-site/pull/467 . Hope it
>>>>>> would be helpful.
>>>>>> >>
>>>>>> >> Boyuan
>>>>>> >>
>>>>>> >
>>>>>>
>>>>> --
>>>>> Got feedback? go/pabloem-feedback
>>>>> <https://goto.google.com/pabloem-feedback>
>>>>>
>>>>
>>>> --
>>> Got feedback? go/pabloem-feedback
>>> <https://goto.google.com/pabloem-feedback>
>>>
>>
>> --
Got feedback? go/pabloem-feedback
Re: [CANCEL][VOTE] Apache Beam, version 2.5.0, release candidate #1
Posted by Scott Wegner <sw...@google.com>.
From my understanding, the @SuppressFBWarnings usage is in a dependency
(ByteBuddy) rather than directly in our code; so we're not able to modify
the usage.
Pablo, feel free to disable failOnWarning for the sdks-java-core project
temporarily. This isn't a major regression since we've only recently made
the change to enable it [1]. We can work separately on figuring out how to
resolve the warnings.
[1] https://github.com/apache/beam/pull/5319
On Tue, Jun 12, 2018 at 11:57 PM Tim Robertson <ti...@gmail.com>
wrote:
> Hi Pablo,
>
> I'm afraid I couldn't find one either... there is an issue about it [1]
> which is old so it doesn't look likely to be resolved either.
>
> If you have time (sorry I am a bit busy) could you please verify the
> version does work if you install that version locally? I know the maven
> version of that [2] but not sure on the gradle equivalent. If we know it
> works, we can then find a repository that fits ok with Apache/Beam policy.
>
> Alternatively, we could consider using a fully qualified reference (i.e.
> @edu.umd.cs.findbugs.annotations.SuppressWarnings) to the deprecated
> version and leave the dependency at the 1.3.9-1. I believe our general
> direction is to remove findbugs when errorprone covers all aspects so I
> *expect* this should be considered reasonable.
>
> I hope this helps,
> Tim
>
> [1] https://github.com/stephenc/findbugs-annotations/issues/4
> [2] https://maven.apache.org/guides/mini/guide-3rd-party-jars-local.html
>
> On Wed, Jun 13, 2018 at 8:39 AM, Pablo Estrada <pa...@google.com> wrote:
>
>> Hi Tim,
>> you're right. Thanks for pointing that out. There's just one problem that
>> I'm running into now: The 3.0.1-1 version does not seem to be available in
>> Maven Central[1]. Looking at the website, I am not quite sure if there's
>> another repository where they do stage the newer versions?[2]
>>
>> -P
>>
>> [1]
>> https://repo.maven.apache.org/maven2/com/github/stephenc/findbugs/findbugs-annotations
>> /
>> [2] http://stephenc.github.io/findbugs-annotations/
>>
>> On Tue, Jun 12, 2018 at 11:10 PM Tim Robertson <ti...@gmail.com>
>> wrote:
>>
>>> Hi Pablo,
>>>
>>> I took only a quick look.
>>>
>>> "- The JAR from the non-LGPL findbugs does not contain the
>>> SuppressFBWarnings annotation"
>>>
>>> Unless I misunderstand you it looks like SuppressFBWarnings was added in
>>> Stephen's version in this commit [1] which was introduced in version
>>> 2.0.3-1 - I've checked is in the 3.0.1-1 build [2]
>>> I notice in your commits [1] you've been exploring version 3.0.0 already
>>> though... what happens when you use 3.0.1-1? It sounds like the wrong
>>> version is coming in rather than the annotation being missing.
>>>
>>> Thanks,
>>> Tim
>>>
>>> [1]
>>> https://github.com/stephenc/findbugs-annotations/commits/master/src/main/java/edu/umd/cs/findbugs/annotations/SuppressWarnings.java
>>> [2] https://github.com/stephenc/findbugs-annotations/releases
>>> [3]
>>> https://github.com/apache/beam/pull/5609/commits/32c7df706e970557f154ff6bc521b2e00f9d09ab
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Wed, Jun 13, 2018 at 2:37 AM, Pablo Estrada <pa...@google.com>
>>> wrote:
>>>
>>>> Hi all,
>>>> I'll humbly declare that after wrestling with he build to stop
>>>> depending on the wrong findbugs_annotations, I feel somewhat lost. The
>>>> issue is actually quite small:
>>>>
>>>> - The JAR from the non-LGPL findbugs does not contain the
>>>> SuppressFBWarnings annotation. This means that when building, ByteBuddy
>>>> produces a few warnings (nothing critical).
>>>> - The easiest way to avoid this failure is to call
>>>> applyJavaNature(failOnWarning: false), but this would be bad, since we want
>>>> to keep a high standard for tasks like ErrorProne and FindBugs itself.
>>>> - So I find myself lost: How do we suppress trivial warnings coming
>>>> from missing annotations, and honor warnings coming from other plugins?
>>>>
>>>> Any help / a PR from someone more capable would be appreciated.
>>>> Best
>>>> -P.
>>>>
>>>> On Tue, Jun 12, 2018 at 3:02 PM Ismaël Mejía <ie...@gmail.com> wrote:
>>>>
>>>>> Yes, ok I was not aware it was already being addressed, nice.
>>>>> On Tue, Jun 12, 2018 at 11:56 PM Ahmet Altay <al...@google.com> wrote:
>>>>> >
>>>>> > Ismaël,
>>>>> >
>>>>> > I believe Pablo's https://github.com/apache/beam/pull/5609 is
>>>>> fixing the issue by changing the findbugs back to
>>>>> "com.github.stephenc.findbugs". Is this what you are referring to?
>>>>> >
>>>>> > Ahmet
>>>>> >
>>>>> > On Tue, Jun 12, 2018 at 2:51 PM, Boyuan Zhang <bo...@google.com>
>>>>> wrote:
>>>>> >>
>>>>> >> Hey JB,
>>>>> >>
>>>>> >> I added some instructions about how to create python wheels in this
>>>>> PR: https://github.com/apache/beam-site/pull/467 . Hope it would be
>>>>> helpful.
>>>>> >>
>>>>> >> Boyuan
>>>>> >>
>>>>> >
>>>>>
>>>> --
>>>> Got feedback? go/pabloem-feedback
>>>> <https://goto.google.com/pabloem-feedback>
>>>>
>>>
>>> --
>> Got feedback? go/pabloem-feedback
>> <https://goto.google.com/pabloem-feedback>
>>
>
>
Re: [CANCEL][VOTE] Apache Beam, version 2.5.0, release candidate #1
Posted by Tim Robertson <ti...@gmail.com>.
Hi Pablo,
I'm afraid I couldn't find one either... there is an issue about it [1]
which is old so it doesn't look likely to be resolved either.
If you have time (sorry I am a bit busy) could you please verify the
version does work if you install that version locally? I know the maven
version of that [2] but not sure on the gradle equivalent. If we know it
works, we can then find a repository that fits ok with Apache/Beam policy.
Alternatively, we could consider using a fully qualified reference (i.e.
@edu.umd.cs.findbugs.annotations.SuppressWarnings) to the deprecated
version and leave the dependency at the 1.3.9-1. I believe our general
direction is to remove findbugs when errorprone covers all aspects so I
*expect* this should be considered reasonable.
I hope this helps,
Tim
[1] https://github.com/stephenc/findbugs-annotations/issues/4
[2] https://maven.apache.org/guides/mini/guide-3rd-party-jars-local.html
On Wed, Jun 13, 2018 at 8:39 AM, Pablo Estrada <pa...@google.com> wrote:
> Hi Tim,
> you're right. Thanks for pointing that out. There's just one problem that
> I'm running into now: The 3.0.1-1 version does not seem to be available in
> Maven Central[1]. Looking at the website, I am not quite sure if there's
> another repository where they do stage the newer versions?[2]
>
> -P
>
> [1] https://repo.maven.apache.org/maven2/com/github/
> stephenc/findbugs/findbugs-annotations/
> [2] http://stephenc.github.io/findbugs-annotations/
>
> On Tue, Jun 12, 2018 at 11:10 PM Tim Robertson <ti...@gmail.com>
> wrote:
>
>> Hi Pablo,
>>
>> I took only a quick look.
>>
>> "- The JAR from the non-LGPL findbugs does not contain the
>> SuppressFBWarnings annotation"
>>
>> Unless I misunderstand you it looks like SuppressFBWarnings was added in
>> Stephen's version in this commit [1] which was introduced in version
>> 2.0.3-1 - I've checked is in the 3.0.1-1 build [2]
>> I notice in your commits [1] you've been exploring version 3.0.0 already
>> though... what happens when you use 3.0.1-1? It sounds like the wrong
>> version is coming in rather than the annotation being missing.
>>
>> Thanks,
>> Tim
>>
>> [1] https://github.com/stephenc/findbugs-annotations/
>> commits/master/src/main/java/edu/umd/cs/findbugs/
>> annotations/SuppressWarnings.java
>> [2] https://github.com/stephenc/findbugs-annotations/releases
>> [3] https://github.com/apache/beam/pull/5609/commits/
>> 32c7df706e970557f154ff6bc521b2e00f9d09ab
>>
>>
>>
>>
>>
>>
>>
>> On Wed, Jun 13, 2018 at 2:37 AM, Pablo Estrada <pa...@google.com>
>> wrote:
>>
>>> Hi all,
>>> I'll humbly declare that after wrestling with he build to stop depending
>>> on the wrong findbugs_annotations, I feel somewhat lost. The issue is
>>> actually quite small:
>>>
>>> - The JAR from the non-LGPL findbugs does not contain the
>>> SuppressFBWarnings annotation. This means that when building, ByteBuddy
>>> produces a few warnings (nothing critical).
>>> - The easiest way to avoid this failure is to call
>>> applyJavaNature(failOnWarning: false), but this would be bad, since we want
>>> to keep a high standard for tasks like ErrorProne and FindBugs itself.
>>> - So I find myself lost: How do we suppress trivial warnings coming from
>>> missing annotations, and honor warnings coming from other plugins?
>>>
>>> Any help / a PR from someone more capable would be appreciated.
>>> Best
>>> -P.
>>>
>>> On Tue, Jun 12, 2018 at 3:02 PM Ismaël Mejía <ie...@gmail.com> wrote:
>>>
>>>> Yes, ok I was not aware it was already being addressed, nice.
>>>> On Tue, Jun 12, 2018 at 11:56 PM Ahmet Altay <al...@google.com> wrote:
>>>> >
>>>> > Ismaël,
>>>> >
>>>> > I believe Pablo's https://github.com/apache/beam/pull/5609 is fixing
>>>> the issue by changing the findbugs back to "com.github.stephenc.findbugs".
>>>> Is this what you are referring to?
>>>> >
>>>> > Ahmet
>>>> >
>>>> > On Tue, Jun 12, 2018 at 2:51 PM, Boyuan Zhang <bo...@google.com>
>>>> wrote:
>>>> >>
>>>> >> Hey JB,
>>>> >>
>>>> >> I added some instructions about how to create python wheels in this
>>>> PR: https://github.com/apache/beam-site/pull/467 . Hope it would be
>>>> helpful.
>>>> >>
>>>> >> Boyuan
>>>> >>
>>>> >
>>>>
>>> --
>>> Got feedback? go/pabloem-feedback
>>> <https://goto.google.com/pabloem-feedback>
>>>
>>
>> --
> Got feedback? go/pabloem-feedback
>
Re: [CANCEL][VOTE] Apache Beam, version 2.5.0, release candidate #1
Posted by Pablo Estrada <pa...@google.com>.
Hi Tim,
you're right. Thanks for pointing that out. There's just one problem that
I'm running into now: The 3.0.1-1 version does not seem to be available in
Maven Central[1]. Looking at the website, I am not quite sure if there's
another repository where they do stage the newer versions?[2]
-P
[1]
https://repo.maven.apache.org/maven2/com/github/stephenc/findbugs/findbugs-annotations
/
[2] http://stephenc.github.io/findbugs-annotations/
On Tue, Jun 12, 2018 at 11:10 PM Tim Robertson <ti...@gmail.com>
wrote:
> Hi Pablo,
>
> I took only a quick look.
>
> "- The JAR from the non-LGPL findbugs does not contain the
> SuppressFBWarnings annotation"
>
> Unless I misunderstand you it looks like SuppressFBWarnings was added in
> Stephen's version in this commit [1] which was introduced in version
> 2.0.3-1 - I've checked is in the 3.0.1-1 build [2]
> I notice in your commits [1] you've been exploring version 3.0.0 already
> though... what happens when you use 3.0.1-1? It sounds like the wrong
> version is coming in rather than the annotation being missing.
>
> Thanks,
> Tim
>
> [1]
> https://github.com/stephenc/findbugs-annotations/commits/master/src/main/java/edu/umd/cs/findbugs/annotations/SuppressWarnings.java
> [2] https://github.com/stephenc/findbugs-annotations/releases
> [3]
> https://github.com/apache/beam/pull/5609/commits/32c7df706e970557f154ff6bc521b2e00f9d09ab
>
>
>
>
>
>
>
> On Wed, Jun 13, 2018 at 2:37 AM, Pablo Estrada <pa...@google.com> wrote:
>
>> Hi all,
>> I'll humbly declare that after wrestling with he build to stop depending
>> on the wrong findbugs_annotations, I feel somewhat lost. The issue is
>> actually quite small:
>>
>> - The JAR from the non-LGPL findbugs does not contain the
>> SuppressFBWarnings annotation. This means that when building, ByteBuddy
>> produces a few warnings (nothing critical).
>> - The easiest way to avoid this failure is to call
>> applyJavaNature(failOnWarning: false), but this would be bad, since we want
>> to keep a high standard for tasks like ErrorProne and FindBugs itself.
>> - So I find myself lost: How do we suppress trivial warnings coming from
>> missing annotations, and honor warnings coming from other plugins?
>>
>> Any help / a PR from someone more capable would be appreciated.
>> Best
>> -P.
>>
>> On Tue, Jun 12, 2018 at 3:02 PM Ismaël Mejía <ie...@gmail.com> wrote:
>>
>>> Yes, ok I was not aware it was already being addressed, nice.
>>> On Tue, Jun 12, 2018 at 11:56 PM Ahmet Altay <al...@google.com> wrote:
>>> >
>>> > Ismaël,
>>> >
>>> > I believe Pablo's https://github.com/apache/beam/pull/5609 is fixing
>>> the issue by changing the findbugs back to "com.github.stephenc.findbugs".
>>> Is this what you are referring to?
>>> >
>>> > Ahmet
>>> >
>>> > On Tue, Jun 12, 2018 at 2:51 PM, Boyuan Zhang <bo...@google.com>
>>> wrote:
>>> >>
>>> >> Hey JB,
>>> >>
>>> >> I added some instructions about how to create python wheels in this
>>> PR: https://github.com/apache/beam-site/pull/467 . Hope it would be
>>> helpful.
>>> >>
>>> >> Boyuan
>>> >>
>>> >
>>>
>> --
>> Got feedback? go/pabloem-feedback
>> <https://goto.google.com/pabloem-feedback>
>>
>
> --
Got feedback? go/pabloem-feedback
Re: [CANCEL][VOTE] Apache Beam, version 2.5.0, release candidate #1
Posted by Tim Robertson <ti...@gmail.com>.
Hi Pablo,
I took only a quick look.
"- The JAR from the non-LGPL findbugs does not contain the
SuppressFBWarnings annotation"
Unless I misunderstand you it looks like SuppressFBWarnings was added in
Stephen's version in this commit [1] which was introduced in version
2.0.3-1 - I've checked is in the 3.0.1-1 build [2]
I notice in your commits [1] you've been exploring version 3.0.0 already
though... what happens when you use 3.0.1-1? It sounds like the wrong
version is coming in rather than the annotation being missing.
Thanks,
Tim
[1]
https://github.com/stephenc/findbugs-annotations/commits/master/src/main/java/edu/umd/cs/findbugs/annotations/SuppressWarnings.java
[2] https://github.com/stephenc/findbugs-annotations/releases
[3]
https://github.com/apache/beam/pull/5609/commits/32c7df706e970557f154ff6bc521b2e00f9d09ab
On Wed, Jun 13, 2018 at 2:37 AM, Pablo Estrada <pa...@google.com> wrote:
> Hi all,
> I'll humbly declare that after wrestling with he build to stop depending
> on the wrong findbugs_annotations, I feel somewhat lost. The issue is
> actually quite small:
>
> - The JAR from the non-LGPL findbugs does not contain the
> SuppressFBWarnings annotation. This means that when building, ByteBuddy
> produces a few warnings (nothing critical).
> - The easiest way to avoid this failure is to call
> applyJavaNature(failOnWarning: false), but this would be bad, since we want
> to keep a high standard for tasks like ErrorProne and FindBugs itself.
> - So I find myself lost: How do we suppress trivial warnings coming from
> missing annotations, and honor warnings coming from other plugins?
>
> Any help / a PR from someone more capable would be appreciated.
> Best
> -P.
>
> On Tue, Jun 12, 2018 at 3:02 PM Ismaël Mejía <ie...@gmail.com> wrote:
>
>> Yes, ok I was not aware it was already being addressed, nice.
>> On Tue, Jun 12, 2018 at 11:56 PM Ahmet Altay <al...@google.com> wrote:
>> >
>> > Ismaël,
>> >
>> > I believe Pablo's https://github.com/apache/beam/pull/5609 is fixing
>> the issue by changing the findbugs back to "com.github.stephenc.findbugs".
>> Is this what you are referring to?
>> >
>> > Ahmet
>> >
>> > On Tue, Jun 12, 2018 at 2:51 PM, Boyuan Zhang <bo...@google.com>
>> wrote:
>> >>
>> >> Hey JB,
>> >>
>> >> I added some instructions about how to create python wheels in this
>> PR: https://github.com/apache/beam-site/pull/467 . Hope it would be
>> helpful.
>> >>
>> >> Boyuan
>> >>
>> >
>>
> --
> Got feedback? go/pabloem-feedback
>
Re: [CANCEL][VOTE] Apache Beam, version 2.5.0, release candidate #1
Posted by Pablo Estrada <pa...@google.com>.
Hi all,
I'll humbly declare that after wrestling with he build to stop depending on
the wrong findbugs_annotations, I feel somewhat lost. The issue is actually
quite small:
- The JAR from the non-LGPL findbugs does not contain the
SuppressFBWarnings annotation. This means that when building, ByteBuddy
produces a few warnings (nothing critical).
- The easiest way to avoid this failure is to call
applyJavaNature(failOnWarning: false), but this would be bad, since we want
to keep a high standard for tasks like ErrorProne and FindBugs itself.
- So I find myself lost: How do we suppress trivial warnings coming from
missing annotations, and honor warnings coming from other plugins?
Any help / a PR from someone more capable would be appreciated.
Best
-P.
On Tue, Jun 12, 2018 at 3:02 PM Ismaël Mejía <ie...@gmail.com> wrote:
> Yes, ok I was not aware it was already being addressed, nice.
> On Tue, Jun 12, 2018 at 11:56 PM Ahmet Altay <al...@google.com> wrote:
> >
> > Ismaël,
> >
> > I believe Pablo's https://github.com/apache/beam/pull/5609 is fixing
> the issue by changing the findbugs back to "com.github.stephenc.findbugs".
> Is this what you are referring to?
> >
> > Ahmet
> >
> > On Tue, Jun 12, 2018 at 2:51 PM, Boyuan Zhang <bo...@google.com>
> wrote:
> >>
> >> Hey JB,
> >>
> >> I added some instructions about how to create python wheels in this PR:
> https://github.com/apache/beam-site/pull/467 . Hope it would be helpful.
> >>
> >> Boyuan
> >>
> >
>
--
Got feedback? go/pabloem-feedback
Re: [CANCEL][VOTE] Apache Beam, version 2.5.0, release candidate #1
Posted by Ismaël Mejía <ie...@gmail.com>.
Yes, ok I was not aware it was already being addressed, nice.
On Tue, Jun 12, 2018 at 11:56 PM Ahmet Altay <al...@google.com> wrote:
>
> Ismaël,
>
> I believe Pablo's https://github.com/apache/beam/pull/5609 is fixing the issue by changing the findbugs back to "com.github.stephenc.findbugs". Is this what you are referring to?
>
> Ahmet
>
> On Tue, Jun 12, 2018 at 2:51 PM, Boyuan Zhang <bo...@google.com> wrote:
>>
>> Hey JB,
>>
>> I added some instructions about how to create python wheels in this PR: https://github.com/apache/beam-site/pull/467 . Hope it would be helpful.
>>
>> Boyuan
>>
>
Re: [CANCEL][VOTE] Apache Beam, version 2.5.0, release candidate #1
Posted by Ahmet Altay <al...@google.com>.
Ismaël,
I believe Pablo's https://github.com/apache/beam/pull/5609 is fixing the
issue by changing the findbugs back to "com.github.stephenc.findbugs". Is
this what you are referring to?
Ahmet
On Tue, Jun 12, 2018 at 2:51 PM, Boyuan Zhang <bo...@google.com> wrote:
> Hey JB,
>
> I added some instructions about how to create python wheels in this PR:
> https://github.com/apache/beam-site/pull/467 . Hope it would be helpful.
>
> Boyuan
>
>
Re: [CANCEL][VOTE] Apache Beam, version 2.5.0, release candidate #1
Posted by Boyuan Zhang <bo...@google.com>.
Hey JB,
I added some instructions about how to create python wheels in this PR:
https://github.com/apache/beam-site/pull/467 . Hope it would be helpful.
Boyuan
On Tue, Jun 12, 2018 at 12:33 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> Hi all,
>
> due to issues found by Udi and Pablo, I cancel RC1 to prepare a new
> release.
>
> Thanks Udi and Pablo for catching up.
>
> I will prepare a RC2 after cherry pick/fixed the found issues (tomorrow
> morning my time).
>
> Regards
> JB
>
>
> On 12/06/2018 21:04, Pablo Estrada wrote:
> > Hello all,
> > I've found another release blocker: The current findbugs dependency in
> > build_rules.gradle has a GPL license. Here's the PR to depend on the
> > correct findbugs: https://github.com/apache/beam/pull/5609
> >
> > Best
> > -P.
> >
> > On Tue, Jun 12, 2018 at 11:03 AM Udi Meiri <ehudm@google.com
> > <ma...@google.com>> wrote:
> >
> > Cherrypick created: https://github.com/apache/beam/pull/5607
> > Tests still running.
> >
> > On Tue, Jun 12, 2018 at 9:59 AM Udi Meiri <ehudm@google.com
> > <ma...@google.com>> wrote:
> >
> > -1: Would like to cherry pick a fix
> > for https://issues.apache.org/jira/browse/BEAM-4536
> >
> > On Tue, Jun 12, 2018 at 9:48 AM Jean-Baptiste Onofré
> > <jb@nanthrax.net <ma...@nanthrax.net>> wrote:
> >
> > Any update about your vote and fix ?
> >
> > Thanks !
> > Regards
> > JB
> >
> > On 12/06/2018 04:02, Udi Meiri wrote:
> > > Another bug: reading from PubSub with_attributes=True is
> > broken on
> > > Python with Dataflow.
> > > https://issues.apache.org/jira/browse/BEAM-4536
> > >
> > > JB, I'm making a PR that removes this keyword and I'd like
> > to propose it
> > > as a cherrypick to 2.5.0.
> > > (feature should be fixed in the next release)
> > >
> > > On Mon, Jun 11, 2018 at 6:19 PM Chamikara Jayalath
> > <chamikara@google.com <ma...@google.com>
> > > <mailto:chamikara@google.com
> > <ma...@google.com>>> wrote:
> > >
> > > FYI: looks like Python tests are failing for Windows.
> JIRA
> > > is https://issues.apache.org/jira/browse/BEAM-4535.
> > >
> > > I don't think this is a release blocker but this
> > should probably go
> > > in release notes (for any user that tries to run tests
> > on Python
> > > source build). And we should try to incorporate a fix
> > if we happen
> > > to cut another release candidate for some reason.
> > >
> > > Thanks,
> > > Cham
> > >
> > > On Mon, Jun 11, 2018 at 5:46 PM Pablo Estrada
> > <pabloem@google.com <ma...@google.com>
> > > <mailto:pabloem@google.com
> > <ma...@google.com>>> wrote:
> > >
> > > Thanks everyone who has pitched in to validate the
> > release!
> > >
> > > Boyuan Zhang and I have also run a few pipelines,
> > and verified
> > > that they work properly (see release validation
> > spreadsheet[1]).
> > >
> > > We have also found that the Game Stats pipeline is
> > failing in
> > > Python Streaming Dataflow. I have filed
> > BEAM-4534[2]. This is
> > > not a blocker, since Python streaming is not yet
> > fully supported.
> > >
> > > It seems that the uploaded artifacts look good.
> > >
> > > We have noticed that the Python artifacts are
> > still missing
> > > Python wheel files (compare [3] and [4]). JB,
> > could you please
> > > add the wheel files? Boyuan and I can try to help
> > you prepare
> > > them / upload them if necessary. Please let us
> know.
> > >
> > > Thanks again!
> > > -P.
> > >
> > >
> > [1]
> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=152451807
> > > [2]
> https://issues.apache.org/jira/browse/BEAM-4534
> > > [3]
> https://dist.apache.org/repos/dist/dev/beam/2.4.0/
> > > [4]
> https://dist.apache.org/repos/dist/dev/beam/2.5.0/
> > >
> > > On Mon, Jun 11, 2018 at 12:37 PM Alan Myrvold
> > > <amyrvold@google.com <ma...@google.com>
> > <mailto:amyrvold@google.com <ma...@google.com>>>
> > wrote:
> > >
> > > +1 (non-binding)
> > >
> > > tested some of the quickstarts
> > >
> > > On Sun, Jun 10, 2018 at 1:39 AM Tim
> > > <timrobertson100@gmail.com
> > <ma...@gmail.com>
> > > <mailto:timrobertson100@gmail.com
> > <ma...@gmail.com>>> wrote:
> > >
> > > Tested by our team:
> > > - mvn inclusion
> > > - Avro, ES, Hadoop IF IO
> > > - Pipelines run on Spark (Cloudera 5.12.0
> > YARN cluster)
> > > - Reviewed release notes
> > >
> > > +1
> > >
> > > Thanks also to everyone who helped get
> > over the gradle
> > > hurdle and in particular to JB.
> > >
> > > Tim
> > >
> > > > On 9 Jun 2018, at 05:56, Jean-Baptiste
> > Onofré
> > > <jb@nanthrax.net <ma...@nanthrax.net>
> > <mailto:jb@nanthrax.net <ma...@nanthrax.net>>> wrote:
> > > >
> > > > No problem Pablo.
> > > >
> > > > The vote period is a minimum, it can be
> > extended as
> > > requested or if we
> > > > don't have the minimum of 3 binding
> votes.
> > > >
> > > > Regards
> > > > JB
> > > >
> > > >> On 09/06/2018 01:54, Pablo Estrada
> wrote:
> > > >> Hello all,
> > > >> I'd like to request an extension of the
> > voting period
> > > until Monday
> > > >> evening (US time, so later in other
> > geographical
> > > regions). This is
> > > >> because we were only now able to
> > publish Dataflow
> > > Workers, and have not
> > > >> had the chance to run release
> > validation tests on
> > > them. The extension
> > > >> will allow us to validate and vote by
> > Monday.
> > > >> Is this acceptable to the community?
> > > >>
> > > >> Best
> > > >> -P.
> > > >>
> > > >> On Fri, Jun 8, 2018 at 6:20 AM Alexey
> > Romanenko
> > > >> <aromanenko.dev@gmail.com
> > <ma...@gmail.com>
> > > <mailto:aromanenko.dev@gmail.com
> > <ma...@gmail.com>>
> > > <mailto:aromanenko.dev@gmail.com
> > <ma...@gmail.com>
> > > <mailto:aromanenko.dev@gmail.com
> > <ma...@gmail.com>>>> wrote:
> > > >>
> > > >> Thank you JB for your work!
> > > >>
> > > >> I tested running simple streaming
> > (/KafkaIO/) and
> > > batch (/TextIO /
> > > >> HDFS/) pipelines with SparkRunner on
> > YARN cluster
> > > - it works fine.
> > > >>
> > > >> WBR,
> > > >> Alexey
> > > >>
> > > >>
> > > >>> On 8 Jun 2018, at 10:00, Etienne
> > Chauchot
> > > <echauchot@apache.org
> > <ma...@apache.org> <mailto:echauchot@apache.org
> > <ma...@apache.org>>
> > > >>> <mailto:echauchot@apache.org
> > <ma...@apache.org>
> > > <mailto:echauchot@apache.org
> > <ma...@apache.org>>>> wrote:
> > > >>>
> > > >>> I forgot to vote:
> > > >>> +1 (non binding).
> > > >>> What I tested:
> > > >>> - no functional or performance
> > regression
> > > comparing to v2.4
> > > >>> - dependencies in the poms are ok
> > > >>>
> > > >>> Etienne
> > > >>>> Le vendredi 08 juin 2018 à 08:27
> > +0200, Romain
> > > Manni-Bucau a écrit :
> > > >>>> +1 (non-binding), mainstream usage
> > is not broken
> > > by the pom
> > > >>>> changes and runtime has no known
> > regression
> > > compared to the 2.4.0
> > > >>>>
> > > >>>> (side note: kudo to JB for this
> > build tool
> > > change release, I know
> > > >>>> how it can hurt ;))
> > > >>>>
> > > >>>> Romain Manni-Bucau
> > > >>>> @rmannibucau
> > <https://twitter.com/rmannibucau>
> > > | Blog
> > > >>>>
> > <https://rmannibucau.metawerx.net/> | Old Blog
> > > >>>>
> > <http://rmannibucau.wordpress.com/> | Github
> > > >>>> <https://github.com/rmannibucau> |
> > LinkedIn
> > > >>>>
> > <https://www.linkedin.com/in/rmannibucau> | Book
> > > >>>>
> > >
> > <
> https://www.packtpub.com/application-development/java-ee-8-high-performance
> >
> > > >>>>
> > > >>>>
> > > >>>> Le jeu. 7 juin 2018 à 16:17,
> > Jean-Baptiste Onofré
> > > >>>> <jb@nanthrax.net
> > <ma...@nanthrax.net> <mailto:jb@nanthrax.net
> > <ma...@nanthrax.net>>
> > > <mailto:jb@nanthrax.net
> > <ma...@nanthrax.net> <mailto:jb@nanthrax.net
> > <ma...@nanthrax.net>>>> a écrit :
> > > >>>>> Thanks for the details Etienne !
> > > >>>>>
> > > >>>>> The good news is that the
> > artifacts seem OK and
> > > the overall Nexmark
> > > >>>>> results are consistent with the
> > 2.4.0 release ones.
> > > >>>>>
> > > >>>>> I'm starting a complete review
> > using the
> > > beam-samples as well.
> > > >>>>>
> > > >>>>> Regards
> > > >>>>> JB
> > > >>>>>
> > > >>>>>> On 07/06/2018 16:14, Etienne
> > Chauchot wrote:
> > > >>>>>> Hi,
> > > >>>>>> I've just run the nexmark queries
> > on v2.5.0-RC1 tag
> > > >>>>>> What we can notice:
> > > >>>>>> - query 3 (exercises CoGroupByKey,
> > state and
> > > timer) shows
> > > >>>>> different
> > > >>>>>> output with DR between batch and
> > streaming and
> > > with the other
> > > >>>>> runners =>
> > > >>>>>> I compared with v2.4 there were
> > still these
> > > differences but with
> > > >>>>>> different output size numbers
> > > >>>>>>
> > > >>>>>> - query 6 (exercises specialized
> > combiner) shows
> > > different output
> > > >>>>>> between the runners => the correct
> > output is 401.
> > > strange that
> > > >>>>> in batch
> > > >>>>>> mode some runners output les
> > Sellers. I compared
> > > with v2.4
> > > >>>>> same output
> > > >>>>>>
> > > >>>>>> - response time of query 7
> > (exercices Max
> > > transform, fanout
> > > >>>>> and side
> > > >>>>>> input) is very slow on DR => I
> > compared with v2.4
> > > , comparable
> > > >>>>> execution
> > > >>>>>> times
> > > >>>>>>
> > > >>>>>> I'm not comparing q10 because it is
> > a write to
> > > GCS so it is
> > > >>>>> very specific.
> > > >>>>>>
> > > >>>>>> => Basically no regression
> > comparing to v2.4
> > > >>>>>>
> > > >>>>>> For the record here is the output
> > (waiting for
> > > ongoing perfkit
> > > >>>>> integration):
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> 1. DR batch
> > > >>>>>>
> > > >>>>>> Performance:
> > > >>>>>>
> > > >>>>>>
> > > >>>>> Conf Runtime(sec) (Baseline)
> > Events(/sec)
> > > (Baseline) Results (Baseline)
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0000 5,8
> > 17283,1
> > > 100000
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0001 3,2
> > 31104,2
> > > 92000
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0002 1,2
> > 82918,7
> > > 351
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0003 2,2
> > 46210,7
> > > 458
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0004 1,2
> > 8503,4
> > > 40
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0005 4,0
> > 25220,7
> > > 12
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0006 0,9
> > 11148,3
> > > 401
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0007 13,2
> > 7580,9
> > > 1
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0008 1,5
> > 67340,1
> > > 6000
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0009 0,7
> > 14025,2
> > > 298
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0010 12,8
> > 7793,0
> > > 1
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0011 2,4
> > 42319,1
> > > 1919
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0012 1,6
> > 61462,8
> > > 1919
> > > >>>>>>
> > > >>>>>
> > >
> >
> ==========================================================================================
> > > >>>>>>
> > > >>>>>> 2. DR streaming
> > > >>>>>>
> > > >>>>>> Performance:
> > > >>>>>>
> > > >>>>>>
> > > >>>>> Conf Runtime(sec) (Baseline)
> > Events(/sec)
> > > (Baseline) Results (Baseline)
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0000 6,5
> > 15285,8
> > > 100000
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0001 3,7
> > 27397,3
> > > 92000
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0002 1,4
> > 69108,5
> > > 351
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0003 3,2
> > 31181,8
> > > 447
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0004 1,2
> > 8361,2
> > > 40
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0005 5,3
> > 18903,6
> > > 12
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0006 0,9
> > 11111,1
> > > 401
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0007 82,5
> > 1212,2
> > > 1
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0008 2,0
> > 51072,5
> > > 6000
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0009 0,8
> > 12903,2
> > > 298
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0010 49,5
> > 2021,8
> > > 1
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0011 3,9
> > 25667,4
> > > 1919
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0012 2,4
> > 41067,8
> > > 1919
> > > >>>>>>
> > > >>>>>
> > >
> >
> ==========================================================================================
> > > >>>>>>
> > > >>>>>> 3. Flink batch
> > > >>>>>> Performance:
> > > >>>>>>
> > > >>>>>>
> > > >>>>> Conf Runtime(sec) (Baseline)
> > Events(/sec)
> > > (Baseline) Results (Baseline)
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0000 1,0
> > 97656,3
> > > 100000
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0001 0,7
> > 141643,1
> > > 92000
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0002 0,4
> > 228310,5
> > > 351
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0003 1,6
> > 64020,5
> > > 580
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0004 0,7
> > 13831,3
> > > 40
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0005 1,4
> > 72939,5
> > > 12
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0006 0,5
> > 20491,8
> > > 103
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0007 1,3
> > 74239,0
> > > 1
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0008 0,8
> > 121506,7
> > > 6000
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0009 0,6
> > 17953,3
> > > 298
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0010 1,3
> > 74682,6
> > > 1
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0011 1,1
> > 92936,8
> > > 1919
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0012 0,8
> > 123001,2
> > > 1919
> > > >>>>>>
> > > >>>>>
> > >
> >
> ==========================================================================================
> > > >>>>>>
> > > >>>>>> 4. Flink streaming
> > > >>>>>> Performance:
> > > >>>>>>
> > > >>>>>>
> > > >>>>> Conf Runtime(sec) (Baseline)
> > Events(/sec)
> > > (Baseline) Results (Baseline)
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0000 5,4
> > 18677,6
> > > 100000
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0001 2,8
> > 35511,4
> > > 92000
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0002 1,8
> > 54318,3
> > > 351
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0003 2,4
> > 41614,6
> > > 580
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0004 1,0
> > 10341,3
> > > 40
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0005 3,4
> > 29568,3
> > > 12
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0006 0,7
> > 13369,0
> > > 401
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0007 2,8
> > 36192,5
> > > 1
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0008 1,8
> > 54854,6
> > > 6000
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0009 0,7
> > 13369,0
> > > 298
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0010 3,4
> > 29841,8
> > > 2
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0011 5,0
> > 19932,2
> > > 1919
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0012 2,6
> > 38835,0
> > > 1919
> > > >>>>>>
> > > >>>>>
> > >
> >
> ==========================================================================================
> > > >>>>>>
> > > >>>>>> 5. Spark batch
> > > >>>>>> Performance:
> > > >>>>>>
> > > >>>>>>
> > > >>>>> Conf Runtime(sec) (Baseline)
> > Events(/sec)
> > > (Baseline) Results (Baseline)
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0000 1,5
> > 65445,0
> > > 100000
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0001 1,3
> > 79491,3
> > > 92000
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0002 0,9
> > 112107,6
> > > 351
> > > >>>>>>
> > > >>>>>>
> > > >>>>> 0003 2,0
> > 48804,3
> > > 580
> > > >>
[CANCEL][VOTE] Apache Beam, version 2.5.0, release candidate #1
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi all,
due to issues found by Udi and Pablo, I cancel RC1 to prepare a new release.
Thanks Udi and Pablo for catching up.
I will prepare a RC2 after cherry pick/fixed the found issues (tomorrow
morning my time).
Regards
JB
On 12/06/2018 21:04, Pablo Estrada wrote:
> Hello all,
> I've found another release blocker: The current findbugs dependency in
> build_rules.gradle has a GPL license. Here's the PR to depend on the
> correct findbugs: https://github.com/apache/beam/pull/5609
>
> Best
> -P.
>
> On Tue, Jun 12, 2018 at 11:03 AM Udi Meiri <ehudm@google.com
> <ma...@google.com>> wrote:
>
> Cherrypick created: https://github.com/apache/beam/pull/5607
> Tests still running.
>
> On Tue, Jun 12, 2018 at 9:59 AM Udi Meiri <ehudm@google.com
> <ma...@google.com>> wrote:
>
> -1: Would like to cherry pick a fix
> for https://issues.apache.org/jira/browse/BEAM-4536
>
> On Tue, Jun 12, 2018 at 9:48 AM Jean-Baptiste Onofré
> <jb@nanthrax.net <ma...@nanthrax.net>> wrote:
>
> Any update about your vote and fix ?
>
> Thanks !
> Regards
> JB
>
> On 12/06/2018 04:02, Udi Meiri wrote:
> > Another bug: reading from PubSub with_attributes=True is
> broken on
> > Python with Dataflow.
> > https://issues.apache.org/jira/browse/BEAM-4536
> >
> > JB, I'm making a PR that removes this keyword and I'd like
> to propose it
> > as a cherrypick to 2.5.0.
> > (feature should be fixed in the next release)
> >
> > On Mon, Jun 11, 2018 at 6:19 PM Chamikara Jayalath
> <chamikara@google.com <ma...@google.com>
> > <mailto:chamikara@google.com
> <ma...@google.com>>> wrote:
> >
> > FYI: looks like Python tests are failing for Windows. JIRA
> > is https://issues.apache.org/jira/browse/BEAM-4535.
> >
> > I don't think this is a release blocker but this
> should probably go
> > in release notes (for any user that tries to run tests
> on Python
> > source build). And we should try to incorporate a fix
> if we happen
> > to cut another release candidate for some reason.
> >
> > Thanks,
> > Cham
> >
> > On Mon, Jun 11, 2018 at 5:46 PM Pablo Estrada
> <pabloem@google.com <ma...@google.com>
> > <mailto:pabloem@google.com
> <ma...@google.com>>> wrote:
> >
> > Thanks everyone who has pitched in to validate the
> release!
> >
> > Boyuan Zhang and I have also run a few pipelines,
> and verified
> > that they work properly (see release validation
> spreadsheet[1]).
> >
> > We have also found that the Game Stats pipeline is
> failing in
> > Python Streaming Dataflow. I have filed
> BEAM-4534[2]. This is
> > not a blocker, since Python streaming is not yet
> fully supported.
> >
> > It seems that the uploaded artifacts look good.
> >
> > We have noticed that the Python artifacts are
> still missing
> > Python wheel files (compare [3] and [4]). JB,
> could you please
> > add the wheel files? Boyuan and I can try to help
> you prepare
> > them / upload them if necessary. Please let us know.
> >
> > Thanks again!
> > -P.
> >
> >
> [1] https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=152451807
> > [2] https://issues.apache.org/jira/browse/BEAM-4534
> > [3] https://dist.apache.org/repos/dist/dev/beam/2.4.0/
> > [4] https://dist.apache.org/repos/dist/dev/beam/2.5.0/
> >
> > On Mon, Jun 11, 2018 at 12:37 PM Alan Myrvold
> > <amyrvold@google.com <ma...@google.com>
> <mailto:amyrvold@google.com <ma...@google.com>>>
> wrote:
> >
> > +1 (non-binding)
> >
> > tested some of the quickstarts
> >
> > On Sun, Jun 10, 2018 at 1:39 AM Tim
> > <timrobertson100@gmail.com
> <ma...@gmail.com>
> > <mailto:timrobertson100@gmail.com
> <ma...@gmail.com>>> wrote:
> >
> > Tested by our team:
> > - mvn inclusion
> > - Avro, ES, Hadoop IF IO
> > - Pipelines run on Spark (Cloudera 5.12.0
> YARN cluster)
> > - Reviewed release notes
> >
> > +1
> >
> > Thanks also to everyone who helped get
> over the gradle
> > hurdle and in particular to JB.
> >
> > Tim
> >
> > > On 9 Jun 2018, at 05:56, Jean-Baptiste
> Onofré
> > <jb@nanthrax.net <ma...@nanthrax.net>
> <mailto:jb@nanthrax.net <ma...@nanthrax.net>>> wrote:
> > >
> > > No problem Pablo.
> > >
> > > The vote period is a minimum, it can be
> extended as
> > requested or if we
> > > don't have the minimum of 3 binding votes.
> > >
> > > Regards
> > > JB
> > >
> > >> On 09/06/2018 01:54, Pablo Estrada wrote:
> > >> Hello all,
> > >> I'd like to request an extension of the
> voting period
> > until Monday
> > >> evening (US time, so later in other
> geographical
> > regions). This is
> > >> because we were only now able to
> publish Dataflow
> > Workers, and have not
> > >> had the chance to run release
> validation tests on
> > them. The extension
> > >> will allow us to validate and vote by
> Monday.
> > >> Is this acceptable to the community?
> > >>
> > >> Best
> > >> -P.
> > >>
> > >> On Fri, Jun 8, 2018 at 6:20 AM Alexey
> Romanenko
> > >> <aromanenko.dev@gmail.com
> <ma...@gmail.com>
> > <mailto:aromanenko.dev@gmail.com
> <ma...@gmail.com>>
> > <mailto:aromanenko.dev@gmail.com
> <ma...@gmail.com>
> > <mailto:aromanenko.dev@gmail.com
> <ma...@gmail.com>>>> wrote:
> > >>
> > >> Thank you JB for your work!
> > >>
> > >> I tested running simple streaming
> (/KafkaIO/) and
> > batch (/TextIO /
> > >> HDFS/) pipelines with SparkRunner on
> YARN cluster
> > - it works fine.
> > >>
> > >> WBR,
> > >> Alexey
> > >>
> > >>
> > >>> On 8 Jun 2018, at 10:00, Etienne
> Chauchot
> > <echauchot@apache.org
> <ma...@apache.org> <mailto:echauchot@apache.org
> <ma...@apache.org>>
> > >>> <mailto:echauchot@apache.org
> <ma...@apache.org>
> > <mailto:echauchot@apache.org
> <ma...@apache.org>>>> wrote:
> > >>>
> > >>> I forgot to vote:
> > >>> +1 (non binding).
> > >>> What I tested:
> > >>> - no functional or performance
> regression
> > comparing to v2.4
> > >>> - dependencies in the poms are ok
> > >>>
> > >>> Etienne
> > >>>> Le vendredi 08 juin 2018 à 08:27
> +0200, Romain
> > Manni-Bucau a écrit :
> > >>>> +1 (non-binding), mainstream usage
> is not broken
> > by the pom
> > >>>> changes and runtime has no known
> regression
> > compared to the 2.4.0
> > >>>>
> > >>>> (side note: kudo to JB for this
> build tool
> > change release, I know
> > >>>> how it can hurt ;))
> > >>>>
> > >>>> Romain Manni-Bucau
> > >>>> @rmannibucau
> <https://twitter.com/rmannibucau>
> > | Blog
> > >>>>
> <https://rmannibucau.metawerx.net/> | Old Blog
> > >>>>
> <http://rmannibucau.wordpress.com/> | Github
> > >>>> <https://github.com/rmannibucau> |
> LinkedIn
> > >>>>
> <https://www.linkedin.com/in/rmannibucau> | Book
> > >>>>
> >
> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
> > >>>>
> > >>>>
> > >>>> Le jeu. 7 juin 2018 à 16:17,
> Jean-Baptiste Onofré
> > >>>> <jb@nanthrax.net
> <ma...@nanthrax.net> <mailto:jb@nanthrax.net
> <ma...@nanthrax.net>>
> > <mailto:jb@nanthrax.net
> <ma...@nanthrax.net> <mailto:jb@nanthrax.net
> <ma...@nanthrax.net>>>> a écrit :
> > >>>>> Thanks for the details Etienne !
> > >>>>>
> > >>>>> The good news is that the
> artifacts seem OK and
> > the overall Nexmark
> > >>>>> results are consistent with the
> 2.4.0 release ones.
> > >>>>>
> > >>>>> I'm starting a complete review
> using the
> > beam-samples as well.
> > >>>>>
> > >>>>> Regards
> > >>>>> JB
> > >>>>>
> > >>>>>> On 07/06/2018 16:14, Etienne
> Chauchot wrote:
> > >>>>>> Hi,
> > >>>>>> I've just run the nexmark queries
> on v2.5.0-RC1 tag
> > >>>>>> What we can notice:
> > >>>>>> - query 3 (exercises CoGroupByKey,
> state and
> > timer) shows
> > >>>>> different
> > >>>>>> output with DR between batch and
> streaming and
> > with the other
> > >>>>> runners =>
> > >>>>>> I compared with v2.4 there were
> still these
> > differences but with
> > >>>>>> different output size numbers
> > >>>>>>
> > >>>>>> - query 6 (exercises specialized
> combiner) shows
> > different output
> > >>>>>> between the runners => the correct
> output is 401.
> > strange that
> > >>>>> in batch
> > >>>>>> mode some runners output les
> Sellers. I compared
> > with v2.4
> > >>>>> same output
> > >>>>>>
> > >>>>>> - response time of query 7
> (exercices Max
> > transform, fanout
> > >>>>> and side
> > >>>>>> input) is very slow on DR => I
> compared with v2.4
> > , comparable
> > >>>>> execution
> > >>>>>> times
> > >>>>>>
> > >>>>>> I'm not comparing q10 because it is
> a write to
> > GCS so it is
> > >>>>> very specific.
> > >>>>>>
> > >>>>>> => Basically no regression
> comparing to v2.4
> > >>>>>>
> > >>>>>> For the record here is the output
> (waiting for
> > ongoing perfkit
> > >>>>> integration):
> > >>>>>>
> > >>>>>>
> > >>>>>> 1. DR batch
> > >>>>>>
> > >>>>>> Performance:
> > >>>>>>
> > >>>>>>
> > >>>>> Conf Runtime(sec) (Baseline)
> Events(/sec)
> > (Baseline) Results (Baseline)
> > >>>>>>
> > >>>>>>
> > >>>>> 0000 5,8
> 17283,1
> > 100000
> > >>>>>>
> > >>>>>>
> > >>>>> 0001 3,2
> 31104,2
> > 92000
> > >>>>>>
> > >>>>>>
> > >>>>> 0002 1,2
> 82918,7
> > 351
> > >>>>>>
> > >>>>>>
> > >>>>> 0003 2,2
> 46210,7
> > 458
> > >>>>>>
> > >>>>>>
> > >>>>> 0004 1,2
> 8503,4
> > 40
> > >>>>>>
> > >>>>>>
> > >>>>> 0005 4,0
> 25220,7
> > 12
> > >>>>>>
> > >>>>>>
> > >>>>> 0006 0,9
> 11148,3
> > 401
> > >>>>>>
> > >>>>>>
> > >>>>> 0007 13,2
> 7580,9
> > 1
> > >>>>>>
> > >>>>>>
> > >>>>> 0008 1,5
> 67340,1
> > 6000
> > >>>>>>
> > >>>>>>
> > >>>>> 0009 0,7
> 14025,2
> > 298
> > >>>>>>
> > >>>>>>
> > >>>>> 0010 12,8
> 7793,0
> > 1
> > >>>>>>
> > >>>>>>
> > >>>>> 0011 2,4
> 42319,1
> > 1919
> > >>>>>>
> > >>>>>>
> > >>>>> 0012 1,6
> 61462,8
> > 1919
> > >>>>>>
> > >>>>>
> >
> ==========================================================================================
> > >>>>>>
> > >>>>>> 2. DR streaming
> > >>>>>>
> > >>>>>> Performance:
> > >>>>>>
> > >>>>>>
> > >>>>> Conf Runtime(sec) (Baseline)
> Events(/sec)
> > (Baseline) Results (Baseline)
> > >>>>>>
> > >>>>>>
> > >>>>> 0000 6,5
> 15285,8
> > 100000
> > >>>>>>
> > >>>>>>
> > >>>>> 0001 3,7
> 27397,3
> > 92000
> > >>>>>>
> > >>>>>>
> > >>>>> 0002 1,4
> 69108,5
> > 351
> > >>>>>>
> > >>>>>>
> > >>>>> 0003 3,2
> 31181,8
> > 447
> > >>>>>>
> > >>>>>>
> > >>>>> 0004 1,2
> 8361,2
> > 40
> > >>>>>>
> > >>>>>>
> > >>>>> 0005 5,3
> 18903,6
> > 12
> > >>>>>>
> > >>>>>>
> > >>>>> 0006 0,9
> 11111,1
> > 401
> > >>>>>>
> > >>>>>>
> > >>>>> 0007 82,5
> 1212,2
> > 1
> > >>>>>>
> > >>>>>>
> > >>>>> 0008 2,0
> 51072,5
> > 6000
> > >>>>>>
> > >>>>>>
> > >>>>> 0009 0,8
> 12903,2
> > 298
> > >>>>>>
> > >>>>>>
> > >>>>> 0010 49,5
> 2021,8
> > 1
> > >>>>>>
> > >>>>>>
> > >>>>> 0011 3,9
> 25667,4
> > 1919
> > >>>>>>
> > >>>>>>
> > >>>>> 0012 2,4
> 41067,8
> > 1919
> > >>>>>>
> > >>>>>
> >
> ==========================================================================================
> > >>>>>>
> > >>>>>> 3. Flink batch
> > >>>>>> Performance:
> > >>>>>>
> > >>>>>>
> > >>>>> Conf Runtime(sec) (Baseline)
> Events(/sec)
> > (Baseline) Results (Baseline)
> > >>>>>>
> > >>>>>>
> > >>>>> 0000 1,0
> 97656,3
> > 100000
> > >>>>>>
> > >>>>>>
> > >>>>> 0001 0,7
> 141643,1
> > 92000
> > >>>>>>
> > >>>>>>
> > >>>>> 0002 0,4
> 228310,5
> > 351
> > >>>>>>
> > >>>>>>
> > >>>>> 0003 1,6
> 64020,5
> > 580
> > >>>>>>
> > >>>>>>
> > >>>>> 0004 0,7
> 13831,3
> > 40
> > >>>>>>
> > >>>>>>
> > >>>>> 0005 1,4
> 72939,5
> > 12
> > >>>>>>
> > >>>>>>
> > >>>>> 0006 0,5
> 20491,8
> > 103
> > >>>>>>
> > >>>>>>
> > >>>>> 0007 1,3
> 74239,0
> > 1
> > >>>>>>
> > >>>>>>
> > >>>>> 0008 0,8
> 121506,7
> > 6000
> > >>>>>>
> > >>>>>>
> > >>>>> 0009 0,6
> 17953,3
> > 298
> > >>>>>>
> > >>>>>>
> > >>>>> 0010 1,3
> 74682,6
> > 1
> > >>>>>>
> > >>>>>>
> > >>>>> 0011 1,1
> 92936,8
> > 1919
> > >>>>>>
> > >>>>>>
> > >>>>> 0012 0,8
> 123001,2
> > 1919
> > >>>>>>
> > >>>>>
> >
> ==========================================================================================
> > >>>>>>
> > >>>>>> 4. Flink streaming
> > >>>>>> Performance:
> > >>>>>>
> > >>>>>>
> > >>>>> Conf Runtime(sec) (Baseline)
> Events(/sec)
> > (Baseline) Results (Baseline)
> > >>>>>>
> > >>>>>>
> > >>>>> 0000 5,4
> 18677,6
> > 100000
> > >>>>>>
> > >>>>>>
> > >>>>> 0001 2,8
> 35511,4
> > 92000
> > >>>>>>
> > >>>>>>
> > >>>>> 0002 1,8
> 54318,3
> > 351
> > >>>>>>
> > >>>>>>
> > >>>>> 0003 2,4
> 41614,6
> > 580
> > >>>>>>
> > >>>>>>
> > >>>>> 0004 1,0
> 10341,3
> > 40
> > >>>>>>
> > >>>>>>
> > >>>>> 0005 3,4
> 29568,3
> > 12
> > >>>>>>
> > >>>>>>
> > >>>>> 0006 0,7
> 13369,0
> > 401
> > >>>>>>
> > >>>>>>
> > >>>>> 0007 2,8
> 36192,5
> > 1
> > >>>>>>
> > >>>>>>
> > >>>>> 0008 1,8
> 54854,6
> > 6000
> > >>>>>>
> > >>>>>>
> > >>>>> 0009 0,7
> 13369,0
> > 298
> > >>>>>>
> > >>>>>>
> > >>>>> 0010 3,4
> 29841,8
> > 2
> > >>>>>>
> > >>>>>>
> > >>>>> 0011 5,0
> 19932,2
> > 1919
> > >>>>>>
> > >>>>>>
> > >>>>> 0012 2,6
> 38835,0
> > 1919
> > >>>>>>
> > >>>>>
> >
> ==========================================================================================
> > >>>>>>
> > >>>>>> 5. Spark batch
> > >>>>>> Performance:
> > >>>>>>
> > >>>>>>
> > >>>>> Conf Runtime(sec) (Baseline)
> Events(/sec)
> > (Baseline) Results (Baseline)
> > >>>>>>
> > >>>>>>
> > >>>>> 0000 1,5
> 65445,0
> > 100000
> > >>>>>>
> > >>>>>>
> > >>>>> 0001 1,3
> 79491,3
> > 92000
> > >>>>>>
> > >>>>>>
> > >>>>> 0002 0,9
> 112107,6
> > 351
> > >>>>>>
> > >>>>>>
> > >>>>> 0003 2,0
> 48804,3
> > 580
> > >>>>>>
> > >>>>>>
> > >>>>> 0004 1,2
> 8382,2
> > 40
> > >>>>>>
> > >>>>>>
> > >>>>> 0005 2,0
> 50838,8
> > 12
> > >>>>>>
> > >>>>>>
> > >>>>> 0006 1,0
> 9699,3
> > 103
> > >>>>>>
> > >>>>>>
> > >>>>> 0007 2,3
> 43308,8
> > 1
> > >>>>>>
> > >>>>>>
> > >>>>> 0008 2,1
> 46794,6
> > 6000
> > >>>>>>
> > >>>>>>
> > >>>>> 0009 1,1
> 8976,7
> > 298
> > >>>>>>
> > >>>>>>
> > >>>>> 0010 1,6
> 62111,8
> > 1
> > >>>>>>
> > >>>>>>
> > >>>>> 0011 2,1
> 46598,3
> > 1919
> > >>>>>>
> > >>>>>>
> > >>>>> 0012 2,3
> 43687,2
> > 1919
> > >>>>>>
> > >>>>>
> >
> ==========================================================================================
> > >>>>>>
> > >>>>>>> Le mercredi 06 juin 2018 à 10:50
> +0200, Etienne
> > Chauchot a écrit :
> > >>>>>>> Thanks JB for all your work ! I
> believe doing
> > the first
> > >>>>> gradle release
> > >>>>>>> must have been hard.
> > >>>>>>> I'll run Nexmark on the release
> and keep you posted.
> > >>>>>>>
> > >>>>>>> Best
> > >>>>>>> Etienne
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> Le mercredi 06 juin 2018 à 10:44
> +0200,
> > Jean-Baptiste Onofré
> > >>>>> a écrit :
> > >>>>>>>> Hi everyone,
> > >>>>>>>>
> > >>>>>>>> Please review and vote on the
> release candidate
> > #1 for the
> > >>>>> version
> > >>>>>>>> 2.5.0, as follows:
> > >>>>>>>>
> > >>>>>>>> [ ] +1, Approve the release
> > >>>>>>>> [ ] -1, Do not approve the
> release (please
> > provide specific
> > >>>>> comments)
> > >>>>>>>>
> > >>>>>>>> NB: this is the first release
> using Gradle, so
> > don't be too
> > >>>>> harsh ;) A
> > >>>>>>>> PR about the release guide will
> follow thanks
> > to this release.
> > >>>>>>>>
> > >>>>>>>> The complete staging area is
> available for your
> > review,
> > >>>>> which includes:
> > >>>>>>>> * JIRA release notes [1],
> > >>>>>>>> * the official Apache source
> release to be
> > deployed to
> > >>>>> dist.apache.org
> <http://dist.apache.org> <http://dist.apache.org>
> > <http://dist.apache.org/>
> > >>>>>>>> [2], which is signed with the key
> with
> > fingerprint C8282E76 [3],
> > >>>>>>>> * all artifacts to be deployed to
> the Maven Central
> > >>>>> Repository [4],
> > >>>>>>>> * source code tag "v2.5.0-RC1" [5],
> > >>>>>>>> * website pull request listing
> the release and
> > publishing
> > >>>>> the API
> > >>>>>>>> reference manual [6].
> > >>>>>>>> * Java artifacts were built with
> Gradle 4.7
> > (wrapper) and
> > >>>>> OpenJDK/Oracle
> > >>>>>>>> JDK 1.8.0_172 (Oracle Corporation
> 25.172-b11).
> > >>>>>>>> * Python artifacts are deployed
> along with the
> > source
> > >>>>> release to the
> > >>>>>>>> dist.apache.org
> <http://dist.apache.org> <http://dist.apache.org>
> > <http://dist.apache.org/> [2].
> > >>>>>>>>
> > >>>>>>>> The vote will be open for at
> least 72 hours. It
> > is adopted
> > >>>>> by majority
> > >>>>>>>> approval, with at least 3 PMC
> affirmative votes.
> > >>>>>>>>
> > >>>>>>>> Thanks,
> > >>>>>>>> JB
> > >>>>>>>>
> > >>>>>>>> [1]
> > >>>>>>>>
> > >>>>>
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12342847
> > >>>>>>>> [2]
> >
> https://dist.apache.org/repos/dist/dev/beam/2.5.0/
> > >>>>>>>> [3]
> >
> https://dist.apache.org/repos/dist/release/beam/KEYS
> > >>>>>>>> [4]
> > >>>>>
> >
> https://repository.apache.org/content/repositories/orgapachebeam-1041/
> > >>>>>>>> [5]
> https://github.com/apache/beam/tree/v2.5.0-RC1
> > >>>>>>>> [6]
> https://github.com/apache/beam-site/pull/463
> > >>>>>>>>
> > >>>>>
> > >>
> > >> --
> > >> Got feedback? go/pabloem-feedback
> <https://goto.google.com/pabloem-feedback>
> > <https://goto.google.com/pabloem-feedback>
> > >
> > > --
> > > Jean-Baptiste Onofré
> > > jbonofre@apache.org
> <ma...@apache.org> <mailto:jbonofre@apache.org
> <ma...@apache.org>>
> > > http://blog.nanthrax.net
> > > Talend - http://www.talend.com
> >
> > --
> > Got feedback? go/pabloem-feedback
> <https://goto.google.com/pabloem-feedback>
> > <https://goto.google.com/pabloem-feedback>
> >
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org <ma...@apache.org>
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
> --
> Got feedback? go/pabloem-feedback
Re: [VOTE] Apache Beam, version 2.5.0, release candidate #1
Posted by Pablo Estrada <pa...@google.com>.
Hello all,
I've found another release blocker: The current findbugs dependency in
build_rules.gradle has a GPL license. Here's the PR to depend on the
correct findbugs: https://github.com/apache/beam/pull/5609
Best
-P.
On Tue, Jun 12, 2018 at 11:03 AM Udi Meiri <eh...@google.com> wrote:
> Cherrypick created: https://github.com/apache/beam/pull/5607
> Tests still running.
>
> On Tue, Jun 12, 2018 at 9:59 AM Udi Meiri <eh...@google.com> wrote:
>
>> -1: Would like to cherry pick a fix for
>> https://issues.apache.org/jira/browse/BEAM-4536
>>
>> On Tue, Jun 12, 2018 at 9:48 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
>> wrote:
>>
>>> Any update about your vote and fix ?
>>>
>>> Thanks !
>>> Regards
>>> JB
>>>
>>> On 12/06/2018 04:02, Udi Meiri wrote:
>>> > Another bug: reading from PubSub with_attributes=True is broken on
>>> > Python with Dataflow.
>>> > https://issues.apache.org/jira/browse/BEAM-4536
>>> >
>>> > JB, I'm making a PR that removes this keyword and I'd like to propose
>>> it
>>> > as a cherrypick to 2.5.0.
>>> > (feature should be fixed in the next release)
>>> >
>>> > On Mon, Jun 11, 2018 at 6:19 PM Chamikara Jayalath <
>>> chamikara@google.com
>>> > <ma...@google.com>> wrote:
>>> >
>>> > FYI: looks like Python tests are failing for Windows. JIRA
>>> > is https://issues.apache.org/jira/browse/BEAM-4535.
>>> >
>>> > I don't think this is a release blocker but this should probably go
>>> > in release notes (for any user that tries to run tests on Python
>>> > source build). And we should try to incorporate a fix if we happen
>>> > to cut another release candidate for some reason.
>>> >
>>> > Thanks,
>>> > Cham
>>> >
>>> > On Mon, Jun 11, 2018 at 5:46 PM Pablo Estrada <pabloem@google.com
>>> > <ma...@google.com>> wrote:
>>> >
>>> > Thanks everyone who has pitched in to validate the release!
>>> >
>>> > Boyuan Zhang and I have also run a few pipelines, and verified
>>> > that they work properly (see release validation
>>> spreadsheet[1]).
>>> >
>>> > We have also found that the Game Stats pipeline is failing in
>>> > Python Streaming Dataflow. I have filed BEAM-4534[2]. This is
>>> > not a blocker, since Python streaming is not yet fully
>>> supported.
>>> >
>>> > It seems that the uploaded artifacts look good.
>>> >
>>> > We have noticed that the Python artifacts are still missing
>>> > Python wheel files (compare [3] and [4]). JB, could you please
>>> > add the wheel files? Boyuan and I can try to help you prepare
>>> > them / upload them if necessary. Please let us know.
>>> >
>>> > Thanks again!
>>> > -P.
>>> >
>>> > [1]
>>> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=152451807
>>> > [2] https://issues.apache.org/jira/browse/BEAM-4534
>>> > [3] https://dist.apache.org/repos/dist/dev/beam/2.4.0/
>>> > [4] https://dist.apache.org/repos/dist/dev/beam/2.5.0/
>>> >
>>> > On Mon, Jun 11, 2018 at 12:37 PM Alan Myrvold
>>> > <amyrvold@google.com <ma...@google.com>> wrote:
>>> >
>>> > +1 (non-binding)
>>> >
>>> > tested some of the quickstarts
>>> >
>>> > On Sun, Jun 10, 2018 at 1:39 AM Tim
>>> > <timrobertson100@gmail.com
>>> > <ma...@gmail.com>> wrote:
>>> >
>>> > Tested by our team:
>>> > - mvn inclusion
>>> > - Avro, ES, Hadoop IF IO
>>> > - Pipelines run on Spark (Cloudera 5.12.0 YARN cluster)
>>> > - Reviewed release notes
>>> >
>>> > +1
>>> >
>>> > Thanks also to everyone who helped get over the gradle
>>> > hurdle and in particular to JB.
>>> >
>>> > Tim
>>> >
>>> > > On 9 Jun 2018, at 05:56, Jean-Baptiste Onofré
>>> > <jb@nanthrax.net <ma...@nanthrax.net>> wrote:
>>> > >
>>> > > No problem Pablo.
>>> > >
>>> > > The vote period is a minimum, it can be extended as
>>> > requested or if we
>>> > > don't have the minimum of 3 binding votes.
>>> > >
>>> > > Regards
>>> > > JB
>>> > >
>>> > >> On 09/06/2018 01:54, Pablo Estrada wrote:
>>> > >> Hello all,
>>> > >> I'd like to request an extension of the voting
>>> period
>>> > until Monday
>>> > >> evening (US time, so later in other geographical
>>> > regions). This is
>>> > >> because we were only now able to publish Dataflow
>>> > Workers, and have not
>>> > >> had the chance to run release validation tests on
>>> > them. The extension
>>> > >> will allow us to validate and vote by Monday.
>>> > >> Is this acceptable to the community?
>>> > >>
>>> > >> Best
>>> > >> -P.
>>> > >>
>>> > >> On Fri, Jun 8, 2018 at 6:20 AM Alexey Romanenko
>>> > >> <aromanenko.dev@gmail.com
>>> > <ma...@gmail.com>
>>> > <mailto:aromanenko.dev@gmail.com
>>> > <ma...@gmail.com>>> wrote:
>>> > >>
>>> > >> Thank you JB for your work!
>>> > >>
>>> > >> I tested running simple streaming (/KafkaIO/) and
>>> > batch (/TextIO /
>>> > >> HDFS/) pipelines with SparkRunner on YARN cluster
>>> > - it works fine.
>>> > >>
>>> > >> WBR,
>>> > >> Alexey
>>> > >>
>>> > >>
>>> > >>> On 8 Jun 2018, at 10:00, Etienne Chauchot
>>> > <echauchot@apache.org <ma...@apache.org>
>>> > >>> <mailto:echauchot@apache.org
>>> > <ma...@apache.org>>> wrote:
>>> > >>>
>>> > >>> I forgot to vote:
>>> > >>> +1 (non binding).
>>> > >>> What I tested:
>>> > >>> - no functional or performance regression
>>> > comparing to v2.4
>>> > >>> - dependencies in the poms are ok
>>> > >>>
>>> > >>> Etienne
>>> > >>>> Le vendredi 08 juin 2018 à 08:27 +0200, Romain
>>> > Manni-Bucau a écrit :
>>> > >>>> +1 (non-binding), mainstream usage is not
>>> broken
>>> > by the pom
>>> > >>>> changes and runtime has no known regression
>>> > compared to the 2.4.0
>>> > >>>>
>>> > >>>> (side note: kudo to JB for this build tool
>>> > change release, I know
>>> > >>>> how it can hurt ;))
>>> > >>>>
>>> > >>>> Romain Manni-Bucau
>>> > >>>> @rmannibucau <https://twitter.com/rmannibucau>
>>> > | Blog
>>> > >>>> <https://rmannibucau.metawerx.net/> | Old Blog
>>> > >>>> <http://rmannibucau.wordpress.com/> | Github
>>> > >>>> <https://github.com/rmannibucau> | LinkedIn
>>> > >>>> <https://www.linkedin.com/in/rmannibucau> |
>>> Book
>>> > >>>>
>>> > <
>>> https://www.packtpub.com/application-development/java-ee-8-high-performance
>>> >
>>> > >>>>
>>> > >>>>
>>> > >>>> Le jeu. 7 juin 2018 à 16:17, Jean-Baptiste
>>> Onofré
>>> > >>>> <jb@nanthrax.net <ma...@nanthrax.net>
>>> > <mailto:jb@nanthrax.net <ma...@nanthrax.net>>> a
>>> écrit :
>>> > >>>>> Thanks for the details Etienne !
>>> > >>>>>
>>> > >>>>> The good news is that the artifacts seem OK
>>> and
>>> > the overall Nexmark
>>> > >>>>> results are consistent with the 2.4.0 release
>>> ones.
>>> > >>>>>
>>> > >>>>> I'm starting a complete review using the
>>> > beam-samples as well.
>>> > >>>>>
>>> > >>>>> Regards
>>> > >>>>> JB
>>> > >>>>>
>>> > >>>>>> On 07/06/2018 16:14, Etienne Chauchot wrote:
>>> > >>>>>> Hi,
>>> > >>>>>> I've just run the nexmark queries on v2.5.0-RC1
>>> tag
>>> > >>>>>> What we can notice:
>>> > >>>>>> - query 3 (exercises CoGroupByKey, state and
>>> > timer) shows
>>> > >>>>> different
>>> > >>>>>> output with DR between batch and streaming and
>>> > with the other
>>> > >>>>> runners =>
>>> > >>>>>> I compared with v2.4 there were still these
>>> > differences but with
>>> > >>>>>> different output size numbers
>>> > >>>>>>
>>> > >>>>>> - query 6 (exercises specialized combiner) shows
>>> > different output
>>> > >>>>>> between the runners => the correct output is
>>> 401.
>>> > strange that
>>> > >>>>> in batch
>>> > >>>>>> mode some runners output les Sellers. I compared
>>> > with v2.4
>>> > >>>>> same output
>>> > >>>>>>
>>> > >>>>>> - response time of query 7 (exercices Max
>>> > transform, fanout
>>> > >>>>> and side
>>> > >>>>>> input) is very slow on DR => I compared with
>>> v2.4
>>> > , comparable
>>> > >>>>> execution
>>> > >>>>>> times
>>> > >>>>>>
>>> > >>>>>> I'm not comparing q10 because it is a write to
>>> > GCS so it is
>>> > >>>>> very specific.
>>> > >>>>>>
>>> > >>>>>> => Basically no regression comparing to v2.4
>>> > >>>>>>
>>> > >>>>>> For the record here is the output (waiting for
>>> > ongoing perfkit
>>> > >>>>> integration):
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>>> 1. DR batch
>>> > >>>>>>
>>> > >>>>>> Performance:
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> Conf Runtime(sec) (Baseline)
>>> Events(/sec)
>>> > (Baseline) Results (Baseline)
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0000 5,8
>>> 17283,1
>>> > 100000
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0001 3,2
>>> 31104,2
>>> > 92000
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0002 1,2
>>> 82918,7
>>> > 351
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0003 2,2
>>> 46210,7
>>> > 458
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0004 1,2
>>> 8503,4
>>> > 40
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0005 4,0
>>> 25220,7
>>> > 12
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0006 0,9
>>> 11148,3
>>> > 401
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0007 13,2
>>> 7580,9
>>> > 1
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0008 1,5
>>> 67340,1
>>> > 6000
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0009 0,7
>>> 14025,2
>>> > 298
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0010 12,8
>>> 7793,0
>>> > 1
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0011 2,4
>>> 42319,1
>>> > 1919
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0012 1,6
>>> 61462,8
>>> > 1919
>>> > >>>>>>
>>> > >>>>>
>>> >
>>> ==========================================================================================
>>> > >>>>>>
>>> > >>>>>> 2. DR streaming
>>> > >>>>>>
>>> > >>>>>> Performance:
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> Conf Runtime(sec) (Baseline)
>>> Events(/sec)
>>> > (Baseline) Results (Baseline)
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0000 6,5
>>> 15285,8
>>> > 100000
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0001 3,7
>>> 27397,3
>>> > 92000
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0002 1,4
>>> 69108,5
>>> > 351
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0003 3,2
>>> 31181,8
>>> > 447
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0004 1,2
>>> 8361,2
>>> > 40
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0005 5,3
>>> 18903,6
>>> > 12
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0006 0,9
>>> 11111,1
>>> > 401
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0007 82,5
>>> 1212,2
>>> > 1
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0008 2,0
>>> 51072,5
>>> > 6000
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0009 0,8
>>> 12903,2
>>> > 298
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0010 49,5
>>> 2021,8
>>> > 1
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0011 3,9
>>> 25667,4
>>> > 1919
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0012 2,4
>>> 41067,8
>>> > 1919
>>> > >>>>>>
>>> > >>>>>
>>> >
>>> ==========================================================================================
>>> > >>>>>>
>>> > >>>>>> 3. Flink batch
>>> > >>>>>> Performance:
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> Conf Runtime(sec) (Baseline)
>>> Events(/sec)
>>> > (Baseline) Results (Baseline)
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0000 1,0
>>> 97656,3
>>> > 100000
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0001 0,7
>>> 141643,1
>>> > 92000
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0002 0,4
>>> 228310,5
>>> > 351
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0003 1,6
>>> 64020,5
>>> > 580
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0004 0,7
>>> 13831,3
>>> > 40
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0005 1,4
>>> 72939,5
>>> > 12
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0006 0,5
>>> 20491,8
>>> > 103
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0007 1,3
>>> 74239,0
>>> > 1
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0008 0,8
>>> 121506,7
>>> > 6000
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0009 0,6
>>> 17953,3
>>> > 298
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0010 1,3
>>> 74682,6
>>> > 1
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0011 1,1
>>> 92936,8
>>> > 1919
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0012 0,8
>>> 123001,2
>>> > 1919
>>> > >>>>>>
>>> > >>>>>
>>> >
>>> ==========================================================================================
>>> > >>>>>>
>>> > >>>>>> 4. Flink streaming
>>> > >>>>>> Performance:
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> Conf Runtime(sec) (Baseline)
>>> Events(/sec)
>>> > (Baseline) Results (Baseline)
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0000 5,4
>>> 18677,6
>>> > 100000
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0001 2,8
>>> 35511,4
>>> > 92000
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0002 1,8
>>> 54318,3
>>> > 351
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0003 2,4
>>> 41614,6
>>> > 580
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0004 1,0
>>> 10341,3
>>> > 40
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0005 3,4
>>> 29568,3
>>> > 12
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0006 0,7
>>> 13369,0
>>> > 401
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0007 2,8
>>> 36192,5
>>> > 1
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0008 1,8
>>> 54854,6
>>> > 6000
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0009 0,7
>>> 13369,0
>>> > 298
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0010 3,4
>>> 29841,8
>>> > 2
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0011 5,0
>>> 19932,2
>>> > 1919
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0012 2,6
>>> 38835,0
>>> > 1919
>>> > >>>>>>
>>> > >>>>>
>>> >
>>> ==========================================================================================
>>> > >>>>>>
>>> > >>>>>> 5. Spark batch
>>> > >>>>>> Performance:
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> Conf Runtime(sec) (Baseline)
>>> Events(/sec)
>>> > (Baseline) Results (Baseline)
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0000 1,5
>>> 65445,0
>>> > 100000
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0001 1,3
>>> 79491,3
>>> > 92000
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0002 0,9
>>> 112107,6
>>> > 351
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0003 2,0
>>> 48804,3
>>> > 580
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0004 1,2
>>> 8382,2
>>> > 40
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0005 2,0
>>> 50838,8
>>> > 12
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0006 1,0
>>> 9699,3
>>> > 103
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0007 2,3
>>> 43308,8
>>> > 1
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0008 2,1
>>> 46794,6
>>> > 6000
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0009 1,1
>>> 8976,7
>>> > 298
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0010 1,6
>>> 62111,8
>>> > 1
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0011 2,1
>>> 46598,3
>>> > 1919
>>> > >>>>>>
>>> > >>>>>>
>>> > >>>>> 0012 2,3
>>> 43687,2
>>> > 1919
>>> > >>>>>>
>>> > >>>>>
>>> >
>>> ==========================================================================================
>>> > >>>>>>
>>> > >>>>>>> Le mercredi 06 juin 2018 à 10:50 +0200, Etienne
>>> > Chauchot a écrit :
>>> > >>>>>>> Thanks JB for all your work ! I believe doing
>>> > the first
>>> > >>>>> gradle release
>>> > >>>>>>> must have been hard.
>>> > >>>>>>> I'll run Nexmark on the release and keep you
>>> posted.
>>> > >>>>>>>
>>> > >>>>>>> Best
>>> > >>>>>>> Etienne
>>> > >>>>>>>
>>> > >>>>>>>
>>> > >>>>>>> Le mercredi 06 juin 2018 à 10:44 +0200,
>>> > Jean-Baptiste Onofré
>>> > >>>>> a écrit :
>>> > >>>>>>>> Hi everyone,
>>> > >>>>>>>>
>>> > >>>>>>>> Please review and vote on the release
>>> candidate
>>> > #1 for the
>>> > >>>>> version
>>> > >>>>>>>> 2.5.0, as follows:
>>> > >>>>>>>>
>>> > >>>>>>>> [ ] +1, Approve the release
>>> > >>>>>>>> [ ] -1, Do not approve the release (please
>>> > provide specific
>>> > >>>>> comments)
>>> > >>>>>>>>
>>> > >>>>>>>> NB: this is the first release using Gradle, so
>>> > don't be too
>>> > >>>>> harsh ;) A
>>> > >>>>>>>> PR about the release guide will follow thanks
>>> > to this release.
>>> > >>>>>>>>
>>> > >>>>>>>> The complete staging area is available for
>>> your
>>> > review,
>>> > >>>>> which includes:
>>> > >>>>>>>> * JIRA release notes [1],
>>> > >>>>>>>> * the official Apache source release to be
>>> > deployed to
>>> > >>>>> dist.apache.org <http://dist.apache.org>
>>> > <http://dist.apache.org/>
>>> > >>>>>>>> [2], which is signed with the key with
>>> > fingerprint C8282E76 [3],
>>> > >>>>>>>> * all artifacts to be deployed to the Maven
>>> Central
>>> > >>>>> Repository [4],
>>> > >>>>>>>> * source code tag "v2.5.0-RC1" [5],
>>> > >>>>>>>> * website pull request listing the release and
>>> > publishing
>>> > >>>>> the API
>>> > >>>>>>>> reference manual [6].
>>> > >>>>>>>> * Java artifacts were built with Gradle 4.7
>>> > (wrapper) and
>>> > >>>>> OpenJDK/Oracle
>>> > >>>>>>>> JDK 1.8.0_172 (Oracle Corporation 25.172-b11).
>>> > >>>>>>>> * Python artifacts are deployed along with the
>>> > source
>>> > >>>>> release to the
>>> > >>>>>>>> dist.apache.org <http://dist.apache.org>
>>> > <http://dist.apache.org/> [2].
>>> > >>>>>>>>
>>> > >>>>>>>> The vote will be open for at least 72 hours.
>>> It
>>> > is adopted
>>> > >>>>> by majority
>>> > >>>>>>>> approval, with at least 3 PMC affirmative
>>> votes.
>>> > >>>>>>>>
>>> > >>>>>>>> Thanks,
>>> > >>>>>>>> JB
>>> > >>>>>>>>
>>> > >>>>>>>> [1]
>>> > >>>>>>>>
>>> > >>>>>
>>> >
>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12342847
>>> > >>>>>>>> [2]
>>> > https://dist.apache.org/repos/dist/dev/beam/2.5.0/
>>> > >>>>>>>> [3]
>>> > https://dist.apache.org/repos/dist/release/beam/KEYS
>>> > >>>>>>>> [4]
>>> > >>>>>
>>> >
>>> https://repository.apache.org/content/repositories/orgapachebeam-1041/
>>> > >>>>>>>> [5]
>>> https://github.com/apache/beam/tree/v2.5.0-RC1
>>> > >>>>>>>> [6]
>>> https://github.com/apache/beam-site/pull/463
>>> > >>>>>>>>
>>> > >>>>>
>>> > >>
>>> > >> --
>>> > >> Got feedback? go/pabloem-feedback
>>> <https://goto.google.com/pabloem-feedback>
>>> > <https://goto.google.com/pabloem-feedback>
>>> > >
>>> > > --
>>> > > Jean-Baptiste Onofré
>>> > > jbonofre@apache.org <ma...@apache.org>
>>> > > http://blog.nanthrax.net
>>> > > Talend - http://www.talend.com
>>> >
>>> > --
>>> > Got feedback? go/pabloem-feedback
>>> <https://goto.google.com/pabloem-feedback>
>>> > <https://goto.google.com/pabloem-feedback>
>>> >
>>>
>>> --
>>> Jean-Baptiste Onofré
>>> jbonofre@apache.org
>>> http://blog.nanthrax.net
>>> Talend - http://www.talend.com
>>>
>> --
Got feedback? go/pabloem-feedback
Re: [VOTE] Apache Beam, version 2.5.0, release candidate #1
Posted by Udi Meiri <eh...@google.com>.
Cherrypick created: https://github.com/apache/beam/pull/5607
Tests still running.
On Tue, Jun 12, 2018 at 9:59 AM Udi Meiri <eh...@google.com> wrote:
> -1: Would like to cherry pick a fix for
> https://issues.apache.org/jira/browse/BEAM-4536
>
> On Tue, Jun 12, 2018 at 9:48 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
>> Any update about your vote and fix ?
>>
>> Thanks !
>> Regards
>> JB
>>
>> On 12/06/2018 04:02, Udi Meiri wrote:
>> > Another bug: reading from PubSub with_attributes=True is broken on
>> > Python with Dataflow.
>> > https://issues.apache.org/jira/browse/BEAM-4536
>> >
>> > JB, I'm making a PR that removes this keyword and I'd like to propose it
>> > as a cherrypick to 2.5.0.
>> > (feature should be fixed in the next release)
>> >
>> > On Mon, Jun 11, 2018 at 6:19 PM Chamikara Jayalath <
>> chamikara@google.com
>> > <ma...@google.com>> wrote:
>> >
>> > FYI: looks like Python tests are failing for Windows. JIRA
>> > is https://issues.apache.org/jira/browse/BEAM-4535.
>> >
>> > I don't think this is a release blocker but this should probably go
>> > in release notes (for any user that tries to run tests on Python
>> > source build). And we should try to incorporate a fix if we happen
>> > to cut another release candidate for some reason.
>> >
>> > Thanks,
>> > Cham
>> >
>> > On Mon, Jun 11, 2018 at 5:46 PM Pablo Estrada <pabloem@google.com
>> > <ma...@google.com>> wrote:
>> >
>> > Thanks everyone who has pitched in to validate the release!
>> >
>> > Boyuan Zhang and I have also run a few pipelines, and verified
>> > that they work properly (see release validation spreadsheet[1]).
>> >
>> > We have also found that the Game Stats pipeline is failing in
>> > Python Streaming Dataflow. I have filed BEAM-4534[2]. This is
>> > not a blocker, since Python streaming is not yet fully
>> supported.
>> >
>> > It seems that the uploaded artifacts look good.
>> >
>> > We have noticed that the Python artifacts are still missing
>> > Python wheel files (compare [3] and [4]). JB, could you please
>> > add the wheel files? Boyuan and I can try to help you prepare
>> > them / upload them if necessary. Please let us know.
>> >
>> > Thanks again!
>> > -P.
>> >
>> > [1]
>> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=152451807
>> > [2] https://issues.apache.org/jira/browse/BEAM-4534
>> > [3] https://dist.apache.org/repos/dist/dev/beam/2.4.0/
>> > [4] https://dist.apache.org/repos/dist/dev/beam/2.5.0/
>> >
>> > On Mon, Jun 11, 2018 at 12:37 PM Alan Myrvold
>> > <amyrvold@google.com <ma...@google.com>> wrote:
>> >
>> > +1 (non-binding)
>> >
>> > tested some of the quickstarts
>> >
>> > On Sun, Jun 10, 2018 at 1:39 AM Tim
>> > <timrobertson100@gmail.com
>> > <ma...@gmail.com>> wrote:
>> >
>> > Tested by our team:
>> > - mvn inclusion
>> > - Avro, ES, Hadoop IF IO
>> > - Pipelines run on Spark (Cloudera 5.12.0 YARN cluster)
>> > - Reviewed release notes
>> >
>> > +1
>> >
>> > Thanks also to everyone who helped get over the gradle
>> > hurdle and in particular to JB.
>> >
>> > Tim
>> >
>> > > On 9 Jun 2018, at 05:56, Jean-Baptiste Onofré
>> > <jb@nanthrax.net <ma...@nanthrax.net>> wrote:
>> > >
>> > > No problem Pablo.
>> > >
>> > > The vote period is a minimum, it can be extended as
>> > requested or if we
>> > > don't have the minimum of 3 binding votes.
>> > >
>> > > Regards
>> > > JB
>> > >
>> > >> On 09/06/2018 01:54, Pablo Estrada wrote:
>> > >> Hello all,
>> > >> I'd like to request an extension of the voting period
>> > until Monday
>> > >> evening (US time, so later in other geographical
>> > regions). This is
>> > >> because we were only now able to publish Dataflow
>> > Workers, and have not
>> > >> had the chance to run release validation tests on
>> > them. The extension
>> > >> will allow us to validate and vote by Monday.
>> > >> Is this acceptable to the community?
>> > >>
>> > >> Best
>> > >> -P.
>> > >>
>> > >> On Fri, Jun 8, 2018 at 6:20 AM Alexey Romanenko
>> > >> <aromanenko.dev@gmail.com
>> > <ma...@gmail.com>
>> > <mailto:aromanenko.dev@gmail.com
>> > <ma...@gmail.com>>> wrote:
>> > >>
>> > >> Thank you JB for your work!
>> > >>
>> > >> I tested running simple streaming (/KafkaIO/) and
>> > batch (/TextIO /
>> > >> HDFS/) pipelines with SparkRunner on YARN cluster
>> > - it works fine.
>> > >>
>> > >> WBR,
>> > >> Alexey
>> > >>
>> > >>
>> > >>> On 8 Jun 2018, at 10:00, Etienne Chauchot
>> > <echauchot@apache.org <ma...@apache.org>
>> > >>> <mailto:echauchot@apache.org
>> > <ma...@apache.org>>> wrote:
>> > >>>
>> > >>> I forgot to vote:
>> > >>> +1 (non binding).
>> > >>> What I tested:
>> > >>> - no functional or performance regression
>> > comparing to v2.4
>> > >>> - dependencies in the poms are ok
>> > >>>
>> > >>> Etienne
>> > >>>> Le vendredi 08 juin 2018 à 08:27 +0200, Romain
>> > Manni-Bucau a écrit :
>> > >>>> +1 (non-binding), mainstream usage is not broken
>> > by the pom
>> > >>>> changes and runtime has no known regression
>> > compared to the 2.4.0
>> > >>>>
>> > >>>> (side note: kudo to JB for this build tool
>> > change release, I know
>> > >>>> how it can hurt ;))
>> > >>>>
>> > >>>> Romain Manni-Bucau
>> > >>>> @rmannibucau <https://twitter.com/rmannibucau>
>> > | Blog
>> > >>>> <https://rmannibucau.metawerx.net/> | Old Blog
>> > >>>> <http://rmannibucau.wordpress.com/> | Github
>> > >>>> <https://github.com/rmannibucau> | LinkedIn
>> > >>>> <https://www.linkedin.com/in/rmannibucau> |
>> Book
>> > >>>>
>> > <
>> https://www.packtpub.com/application-development/java-ee-8-high-performance
>> >
>> > >>>>
>> > >>>>
>> > >>>> Le jeu. 7 juin 2018 à 16:17, Jean-Baptiste
>> Onofré
>> > >>>> <jb@nanthrax.net <ma...@nanthrax.net>
>> > <mailto:jb@nanthrax.net <ma...@nanthrax.net>>> a
>> écrit :
>> > >>>>> Thanks for the details Etienne !
>> > >>>>>
>> > >>>>> The good news is that the artifacts seem OK and
>> > the overall Nexmark
>> > >>>>> results are consistent with the 2.4.0 release
>> ones.
>> > >>>>>
>> > >>>>> I'm starting a complete review using the
>> > beam-samples as well.
>> > >>>>>
>> > >>>>> Regards
>> > >>>>> JB
>> > >>>>>
>> > >>>>>> On 07/06/2018 16:14, Etienne Chauchot wrote:
>> > >>>>>> Hi,
>> > >>>>>> I've just run the nexmark queries on v2.5.0-RC1
>> tag
>> > >>>>>> What we can notice:
>> > >>>>>> - query 3 (exercises CoGroupByKey, state and
>> > timer) shows
>> > >>>>> different
>> > >>>>>> output with DR between batch and streaming and
>> > with the other
>> > >>>>> runners =>
>> > >>>>>> I compared with v2.4 there were still these
>> > differences but with
>> > >>>>>> different output size numbers
>> > >>>>>>
>> > >>>>>> - query 6 (exercises specialized combiner) shows
>> > different output
>> > >>>>>> between the runners => the correct output is 401.
>> > strange that
>> > >>>>> in batch
>> > >>>>>> mode some runners output les Sellers. I compared
>> > with v2.4
>> > >>>>> same output
>> > >>>>>>
>> > >>>>>> - response time of query 7 (exercices Max
>> > transform, fanout
>> > >>>>> and side
>> > >>>>>> input) is very slow on DR => I compared with v2.4
>> > , comparable
>> > >>>>> execution
>> > >>>>>> times
>> > >>>>>>
>> > >>>>>> I'm not comparing q10 because it is a write to
>> > GCS so it is
>> > >>>>> very specific.
>> > >>>>>>
>> > >>>>>> => Basically no regression comparing to v2.4
>> > >>>>>>
>> > >>>>>> For the record here is the output (waiting for
>> > ongoing perfkit
>> > >>>>> integration):
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> 1. DR batch
>> > >>>>>>
>> > >>>>>> Performance:
>> > >>>>>>
>> > >>>>>>
>> > >>>>> Conf Runtime(sec) (Baseline) Events(/sec)
>> > (Baseline) Results (Baseline)
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0000 5,8 17283,1
>> > 100000
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0001 3,2 31104,2
>> > 92000
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0002 1,2 82918,7
>> > 351
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0003 2,2 46210,7
>> > 458
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0004 1,2 8503,4
>> > 40
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0005 4,0 25220,7
>> > 12
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0006 0,9 11148,3
>> > 401
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0007 13,2 7580,9
>> > 1
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0008 1,5 67340,1
>> > 6000
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0009 0,7 14025,2
>> > 298
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0010 12,8 7793,0
>> > 1
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0011 2,4 42319,1
>> > 1919
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0012 1,6 61462,8
>> > 1919
>> > >>>>>>
>> > >>>>>
>> >
>> ==========================================================================================
>> > >>>>>>
>> > >>>>>> 2. DR streaming
>> > >>>>>>
>> > >>>>>> Performance:
>> > >>>>>>
>> > >>>>>>
>> > >>>>> Conf Runtime(sec) (Baseline) Events(/sec)
>> > (Baseline) Results (Baseline)
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0000 6,5 15285,8
>> > 100000
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0001 3,7 27397,3
>> > 92000
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0002 1,4 69108,5
>> > 351
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0003 3,2 31181,8
>> > 447
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0004 1,2 8361,2
>> > 40
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0005 5,3 18903,6
>> > 12
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0006 0,9 11111,1
>> > 401
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0007 82,5 1212,2
>> > 1
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0008 2,0 51072,5
>> > 6000
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0009 0,8 12903,2
>> > 298
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0010 49,5 2021,8
>> > 1
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0011 3,9 25667,4
>> > 1919
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0012 2,4 41067,8
>> > 1919
>> > >>>>>>
>> > >>>>>
>> >
>> ==========================================================================================
>> > >>>>>>
>> > >>>>>> 3. Flink batch
>> > >>>>>> Performance:
>> > >>>>>>
>> > >>>>>>
>> > >>>>> Conf Runtime(sec) (Baseline) Events(/sec)
>> > (Baseline) Results (Baseline)
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0000 1,0 97656,3
>> > 100000
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0001 0,7 141643,1
>> > 92000
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0002 0,4 228310,5
>> > 351
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0003 1,6 64020,5
>> > 580
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0004 0,7 13831,3
>> > 40
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0005 1,4 72939,5
>> > 12
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0006 0,5 20491,8
>> > 103
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0007 1,3 74239,0
>> > 1
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0008 0,8 121506,7
>> > 6000
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0009 0,6 17953,3
>> > 298
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0010 1,3 74682,6
>> > 1
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0011 1,1 92936,8
>> > 1919
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0012 0,8 123001,2
>> > 1919
>> > >>>>>>
>> > >>>>>
>> >
>> ==========================================================================================
>> > >>>>>>
>> > >>>>>> 4. Flink streaming
>> > >>>>>> Performance:
>> > >>>>>>
>> > >>>>>>
>> > >>>>> Conf Runtime(sec) (Baseline) Events(/sec)
>> > (Baseline) Results (Baseline)
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0000 5,4 18677,6
>> > 100000
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0001 2,8 35511,4
>> > 92000
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0002 1,8 54318,3
>> > 351
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0003 2,4 41614,6
>> > 580
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0004 1,0 10341,3
>> > 40
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0005 3,4 29568,3
>> > 12
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0006 0,7 13369,0
>> > 401
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0007 2,8 36192,5
>> > 1
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0008 1,8 54854,6
>> > 6000
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0009 0,7 13369,0
>> > 298
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0010 3,4 29841,8
>> > 2
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0011 5,0 19932,2
>> > 1919
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0012 2,6 38835,0
>> > 1919
>> > >>>>>>
>> > >>>>>
>> >
>> ==========================================================================================
>> > >>>>>>
>> > >>>>>> 5. Spark batch
>> > >>>>>> Performance:
>> > >>>>>>
>> > >>>>>>
>> > >>>>> Conf Runtime(sec) (Baseline) Events(/sec)
>> > (Baseline) Results (Baseline)
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0000 1,5 65445,0
>> > 100000
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0001 1,3 79491,3
>> > 92000
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0002 0,9 112107,6
>> > 351
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0003 2,0 48804,3
>> > 580
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0004 1,2 8382,2
>> > 40
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0005 2,0 50838,8
>> > 12
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0006 1,0 9699,3
>> > 103
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0007 2,3 43308,8
>> > 1
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0008 2,1 46794,6
>> > 6000
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0009 1,1 8976,7
>> > 298
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0010 1,6 62111,8
>> > 1
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0011 2,1 46598,3
>> > 1919
>> > >>>>>>
>> > >>>>>>
>> > >>>>> 0012 2,3 43687,2
>> > 1919
>> > >>>>>>
>> > >>>>>
>> >
>> ==========================================================================================
>> > >>>>>>
>> > >>>>>>> Le mercredi 06 juin 2018 à 10:50 +0200, Etienne
>> > Chauchot a écrit :
>> > >>>>>>> Thanks JB for all your work ! I believe doing
>> > the first
>> > >>>>> gradle release
>> > >>>>>>> must have been hard.
>> > >>>>>>> I'll run Nexmark on the release and keep you
>> posted.
>> > >>>>>>>
>> > >>>>>>> Best
>> > >>>>>>> Etienne
>> > >>>>>>>
>> > >>>>>>>
>> > >>>>>>> Le mercredi 06 juin 2018 à 10:44 +0200,
>> > Jean-Baptiste Onofré
>> > >>>>> a écrit :
>> > >>>>>>>> Hi everyone,
>> > >>>>>>>>
>> > >>>>>>>> Please review and vote on the release candidate
>> > #1 for the
>> > >>>>> version
>> > >>>>>>>> 2.5.0, as follows:
>> > >>>>>>>>
>> > >>>>>>>> [ ] +1, Approve the release
>> > >>>>>>>> [ ] -1, Do not approve the release (please
>> > provide specific
>> > >>>>> comments)
>> > >>>>>>>>
>> > >>>>>>>> NB: this is the first release using Gradle, so
>> > don't be too
>> > >>>>> harsh ;) A
>> > >>>>>>>> PR about the release guide will follow thanks
>> > to this release.
>> > >>>>>>>>
>> > >>>>>>>> The complete staging area is available for your
>> > review,
>> > >>>>> which includes:
>> > >>>>>>>> * JIRA release notes [1],
>> > >>>>>>>> * the official Apache source release to be
>> > deployed to
>> > >>>>> dist.apache.org <http://dist.apache.org>
>> > <http://dist.apache.org/>
>> > >>>>>>>> [2], which is signed with the key with
>> > fingerprint C8282E76 [3],
>> > >>>>>>>> * all artifacts to be deployed to the Maven
>> Central
>> > >>>>> Repository [4],
>> > >>>>>>>> * source code tag "v2.5.0-RC1" [5],
>> > >>>>>>>> * website pull request listing the release and
>> > publishing
>> > >>>>> the API
>> > >>>>>>>> reference manual [6].
>> > >>>>>>>> * Java artifacts were built with Gradle 4.7
>> > (wrapper) and
>> > >>>>> OpenJDK/Oracle
>> > >>>>>>>> JDK 1.8.0_172 (Oracle Corporation 25.172-b11).
>> > >>>>>>>> * Python artifacts are deployed along with the
>> > source
>> > >>>>> release to the
>> > >>>>>>>> dist.apache.org <http://dist.apache.org>
>> > <http://dist.apache.org/> [2].
>> > >>>>>>>>
>> > >>>>>>>> The vote will be open for at least 72 hours. It
>> > is adopted
>> > >>>>> by majority
>> > >>>>>>>> approval, with at least 3 PMC affirmative
>> votes.
>> > >>>>>>>>
>> > >>>>>>>> Thanks,
>> > >>>>>>>> JB
>> > >>>>>>>>
>> > >>>>>>>> [1]
>> > >>>>>>>>
>> > >>>>>
>> >
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12342847
>> > >>>>>>>> [2]
>> > https://dist.apache.org/repos/dist/dev/beam/2.5.0/
>> > >>>>>>>> [3]
>> > https://dist.apache.org/repos/dist/release/beam/KEYS
>> > >>>>>>>> [4]
>> > >>>>>
>> >
>> https://repository.apache.org/content/repositories/orgapachebeam-1041/
>> > >>>>>>>> [5]
>> https://github.com/apache/beam/tree/v2.5.0-RC1
>> > >>>>>>>> [6]
>> https://github.com/apache/beam-site/pull/463
>> > >>>>>>>>
>> > >>>>>
>> > >>
>> > >> --
>> > >> Got feedback? go/pabloem-feedback
>> <https://goto.google.com/pabloem-feedback>
>> > <https://goto.google.com/pabloem-feedback>
>> > >
>> > > --
>> > > Jean-Baptiste Onofré
>> > > jbonofre@apache.org <ma...@apache.org>
>> > > http://blog.nanthrax.net
>> > > Talend - http://www.talend.com
>> >
>> > --
>> > Got feedback? go/pabloem-feedback
>> <https://goto.google.com/pabloem-feedback>
>> > <https://goto.google.com/pabloem-feedback>
>> >
>>
>> --
>> Jean-Baptiste Onofré
>> jbonofre@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>>
>
Re: [VOTE] Apache Beam, version 2.5.0, release candidate #1
Posted by Udi Meiri <eh...@google.com>.
-1: Would like to cherry pick a fix for
https://issues.apache.org/jira/browse/BEAM-4536
On Tue, Jun 12, 2018 at 9:48 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> Any update about your vote and fix ?
>
> Thanks !
> Regards
> JB
>
> On 12/06/2018 04:02, Udi Meiri wrote:
> > Another bug: reading from PubSub with_attributes=True is broken on
> > Python with Dataflow.
> > https://issues.apache.org/jira/browse/BEAM-4536
> >
> > JB, I'm making a PR that removes this keyword and I'd like to propose it
> > as a cherrypick to 2.5.0.
> > (feature should be fixed in the next release)
> >
> > On Mon, Jun 11, 2018 at 6:19 PM Chamikara Jayalath <chamikara@google.com
> > <ma...@google.com>> wrote:
> >
> > FYI: looks like Python tests are failing for Windows. JIRA
> > is https://issues.apache.org/jira/browse/BEAM-4535.
> >
> > I don't think this is a release blocker but this should probably go
> > in release notes (for any user that tries to run tests on Python
> > source build). And we should try to incorporate a fix if we happen
> > to cut another release candidate for some reason.
> >
> > Thanks,
> > Cham
> >
> > On Mon, Jun 11, 2018 at 5:46 PM Pablo Estrada <pabloem@google.com
> > <ma...@google.com>> wrote:
> >
> > Thanks everyone who has pitched in to validate the release!
> >
> > Boyuan Zhang and I have also run a few pipelines, and verified
> > that they work properly (see release validation spreadsheet[1]).
> >
> > We have also found that the Game Stats pipeline is failing in
> > Python Streaming Dataflow. I have filed BEAM-4534[2]. This is
> > not a blocker, since Python streaming is not yet fully supported.
> >
> > It seems that the uploaded artifacts look good.
> >
> > We have noticed that the Python artifacts are still missing
> > Python wheel files (compare [3] and [4]). JB, could you please
> > add the wheel files? Boyuan and I can try to help you prepare
> > them / upload them if necessary. Please let us know.
> >
> > Thanks again!
> > -P.
> >
> > [1]
> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=152451807
> > [2] https://issues.apache.org/jira/browse/BEAM-4534
> > [3] https://dist.apache.org/repos/dist/dev/beam/2.4.0/
> > [4] https://dist.apache.org/repos/dist/dev/beam/2.5.0/
> >
> > On Mon, Jun 11, 2018 at 12:37 PM Alan Myrvold
> > <amyrvold@google.com <ma...@google.com>> wrote:
> >
> > +1 (non-binding)
> >
> > tested some of the quickstarts
> >
> > On Sun, Jun 10, 2018 at 1:39 AM Tim
> > <timrobertson100@gmail.com
> > <ma...@gmail.com>> wrote:
> >
> > Tested by our team:
> > - mvn inclusion
> > - Avro, ES, Hadoop IF IO
> > - Pipelines run on Spark (Cloudera 5.12.0 YARN cluster)
> > - Reviewed release notes
> >
> > +1
> >
> > Thanks also to everyone who helped get over the gradle
> > hurdle and in particular to JB.
> >
> > Tim
> >
> > > On 9 Jun 2018, at 05:56, Jean-Baptiste Onofré
> > <jb@nanthrax.net <ma...@nanthrax.net>> wrote:
> > >
> > > No problem Pablo.
> > >
> > > The vote period is a minimum, it can be extended as
> > requested or if we
> > > don't have the minimum of 3 binding votes.
> > >
> > > Regards
> > > JB
> > >
> > >> On 09/06/2018 01:54, Pablo Estrada wrote:
> > >> Hello all,
> > >> I'd like to request an extension of the voting period
> > until Monday
> > >> evening (US time, so later in other geographical
> > regions). This is
> > >> because we were only now able to publish Dataflow
> > Workers, and have not
> > >> had the chance to run release validation tests on
> > them. The extension
> > >> will allow us to validate and vote by Monday.
> > >> Is this acceptable to the community?
> > >>
> > >> Best
> > >> -P.
> > >>
> > >> On Fri, Jun 8, 2018 at 6:20 AM Alexey Romanenko
> > >> <aromanenko.dev@gmail.com
> > <ma...@gmail.com>
> > <mailto:aromanenko.dev@gmail.com
> > <ma...@gmail.com>>> wrote:
> > >>
> > >> Thank you JB for your work!
> > >>
> > >> I tested running simple streaming (/KafkaIO/) and
> > batch (/TextIO /
> > >> HDFS/) pipelines with SparkRunner on YARN cluster
> > - it works fine.
> > >>
> > >> WBR,
> > >> Alexey
> > >>
> > >>
> > >>> On 8 Jun 2018, at 10:00, Etienne Chauchot
> > <echauchot@apache.org <ma...@apache.org>
> > >>> <mailto:echauchot@apache.org
> > <ma...@apache.org>>> wrote:
> > >>>
> > >>> I forgot to vote:
> > >>> +1 (non binding).
> > >>> What I tested:
> > >>> - no functional or performance regression
> > comparing to v2.4
> > >>> - dependencies in the poms are ok
> > >>>
> > >>> Etienne
> > >>>> Le vendredi 08 juin 2018 à 08:27 +0200, Romain
> > Manni-Bucau a écrit :
> > >>>> +1 (non-binding), mainstream usage is not broken
> > by the pom
> > >>>> changes and runtime has no known regression
> > compared to the 2.4.0
> > >>>>
> > >>>> (side note: kudo to JB for this build tool
> > change release, I know
> > >>>> how it can hurt ;))
> > >>>>
> > >>>> Romain Manni-Bucau
> > >>>> @rmannibucau <https://twitter.com/rmannibucau>
> > | Blog
> > >>>> <https://rmannibucau.metawerx.net/> | Old Blog
> > >>>> <http://rmannibucau.wordpress.com/> | Github
> > >>>> <https://github.com/rmannibucau> | LinkedIn
> > >>>> <https://www.linkedin.com/in/rmannibucau> | Book
> > >>>>
> > <
> https://www.packtpub.com/application-development/java-ee-8-high-performance
> >
> > >>>>
> > >>>>
> > >>>> Le jeu. 7 juin 2018 à 16:17, Jean-Baptiste Onofré
> > >>>> <jb@nanthrax.net <ma...@nanthrax.net>
> > <mailto:jb@nanthrax.net <ma...@nanthrax.net>>> a
> écrit :
> > >>>>> Thanks for the details Etienne !
> > >>>>>
> > >>>>> The good news is that the artifacts seem OK and
> > the overall Nexmark
> > >>>>> results are consistent with the 2.4.0 release
> ones.
> > >>>>>
> > >>>>> I'm starting a complete review using the
> > beam-samples as well.
> > >>>>>
> > >>>>> Regards
> > >>>>> JB
> > >>>>>
> > >>>>>> On 07/06/2018 16:14, Etienne Chauchot wrote:
> > >>>>>> Hi,
> > >>>>>> I've just run the nexmark queries on v2.5.0-RC1
> tag
> > >>>>>> What we can notice:
> > >>>>>> - query 3 (exercises CoGroupByKey, state and
> > timer) shows
> > >>>>> different
> > >>>>>> output with DR between batch and streaming and
> > with the other
> > >>>>> runners =>
> > >>>>>> I compared with v2.4 there were still these
> > differences but with
> > >>>>>> different output size numbers
> > >>>>>>
> > >>>>>> - query 6 (exercises specialized combiner) shows
> > different output
> > >>>>>> between the runners => the correct output is 401.
> > strange that
> > >>>>> in batch
> > >>>>>> mode some runners output les Sellers. I compared
> > with v2.4
> > >>>>> same output
> > >>>>>>
> > >>>>>> - response time of query 7 (exercices Max
> > transform, fanout
> > >>>>> and side
> > >>>>>> input) is very slow on DR => I compared with v2.4
> > , comparable
> > >>>>> execution
> > >>>>>> times
> > >>>>>>
> > >>>>>> I'm not comparing q10 because it is a write to
> > GCS so it is
> > >>>>> very specific.
> > >>>>>>
> > >>>>>> => Basically no regression comparing to v2.4
> > >>>>>>
> > >>>>>> For the record here is the output (waiting for
> > ongoing perfkit
> > >>>>> integration):
> > >>>>>>
> > >>>>>>
> > >>>>>> 1. DR batch
> > >>>>>>
> > >>>>>> Performance:
> > >>>>>>
> > >>>>>>
> > >>>>> Conf Runtime(sec) (Baseline) Events(/sec)
> > (Baseline) Results (Baseline)
> > >>>>>>
> > >>>>>>
> > >>>>> 0000 5,8 17283,1
> > 100000
> > >>>>>>
> > >>>>>>
> > >>>>> 0001 3,2 31104,2
> > 92000
> > >>>>>>
> > >>>>>>
> > >>>>> 0002 1,2 82918,7
> > 351
> > >>>>>>
> > >>>>>>
> > >>>>> 0003 2,2 46210,7
> > 458
> > >>>>>>
> > >>>>>>
> > >>>>> 0004 1,2 8503,4
> > 40
> > >>>>>>
> > >>>>>>
> > >>>>> 0005 4,0 25220,7
> > 12
> > >>>>>>
> > >>>>>>
> > >>>>> 0006 0,9 11148,3
> > 401
> > >>>>>>
> > >>>>>>
> > >>>>> 0007 13,2 7580,9
> > 1
> > >>>>>>
> > >>>>>>
> > >>>>> 0008 1,5 67340,1
> > 6000
> > >>>>>>
> > >>>>>>
> > >>>>> 0009 0,7 14025,2
> > 298
> > >>>>>>
> > >>>>>>
> > >>>>> 0010 12,8 7793,0
> > 1
> > >>>>>>
> > >>>>>>
> > >>>>> 0011 2,4 42319,1
> > 1919
> > >>>>>>
> > >>>>>>
> > >>>>> 0012 1,6 61462,8
> > 1919
> > >>>>>>
> > >>>>>
> >
> ==========================================================================================
> > >>>>>>
> > >>>>>> 2. DR streaming
> > >>>>>>
> > >>>>>> Performance:
> > >>>>>>
> > >>>>>>
> > >>>>> Conf Runtime(sec) (Baseline) Events(/sec)
> > (Baseline) Results (Baseline)
> > >>>>>>
> > >>>>>>
> > >>>>> 0000 6,5 15285,8
> > 100000
> > >>>>>>
> > >>>>>>
> > >>>>> 0001 3,7 27397,3
> > 92000
> > >>>>>>
> > >>>>>>
> > >>>>> 0002 1,4 69108,5
> > 351
> > >>>>>>
> > >>>>>>
> > >>>>> 0003 3,2 31181,8
> > 447
> > >>>>>>
> > >>>>>>
> > >>>>> 0004 1,2 8361,2
> > 40
> > >>>>>>
> > >>>>>>
> > >>>>> 0005 5,3 18903,6
> > 12
> > >>>>>>
> > >>>>>>
> > >>>>> 0006 0,9 11111,1
> > 401
> > >>>>>>
> > >>>>>>
> > >>>>> 0007 82,5 1212,2
> > 1
> > >>>>>>
> > >>>>>>
> > >>>>> 0008 2,0 51072,5
> > 6000
> > >>>>>>
> > >>>>>>
> > >>>>> 0009 0,8 12903,2
> > 298
> > >>>>>>
> > >>>>>>
> > >>>>> 0010 49,5 2021,8
> > 1
> > >>>>>>
> > >>>>>>
> > >>>>> 0011 3,9 25667,4
> > 1919
> > >>>>>>
> > >>>>>>
> > >>>>> 0012 2,4 41067,8
> > 1919
> > >>>>>>
> > >>>>>
> >
> ==========================================================================================
> > >>>>>>
> > >>>>>> 3. Flink batch
> > >>>>>> Performance:
> > >>>>>>
> > >>>>>>
> > >>>>> Conf Runtime(sec) (Baseline) Events(/sec)
> > (Baseline) Results (Baseline)
> > >>>>>>
> > >>>>>>
> > >>>>> 0000 1,0 97656,3
> > 100000
> > >>>>>>
> > >>>>>>
> > >>>>> 0001 0,7 141643,1
> > 92000
> > >>>>>>
> > >>>>>>
> > >>>>> 0002 0,4 228310,5
> > 351
> > >>>>>>
> > >>>>>>
> > >>>>> 0003 1,6 64020,5
> > 580
> > >>>>>>
> > >>>>>>
> > >>>>> 0004 0,7 13831,3
> > 40
> > >>>>>>
> > >>>>>>
> > >>>>> 0005 1,4 72939,5
> > 12
> > >>>>>>
> > >>>>>>
> > >>>>> 0006 0,5 20491,8
> > 103
> > >>>>>>
> > >>>>>>
> > >>>>> 0007 1,3 74239,0
> > 1
> > >>>>>>
> > >>>>>>
> > >>>>> 0008 0,8 121506,7
> > 6000
> > >>>>>>
> > >>>>>>
> > >>>>> 0009 0,6 17953,3
> > 298
> > >>>>>>
> > >>>>>>
> > >>>>> 0010 1,3 74682,6
> > 1
> > >>>>>>
> > >>>>>>
> > >>>>> 0011 1,1 92936,8
> > 1919
> > >>>>>>
> > >>>>>>
> > >>>>> 0012 0,8 123001,2
> > 1919
> > >>>>>>
> > >>>>>
> >
> ==========================================================================================
> > >>>>>>
> > >>>>>> 4. Flink streaming
> > >>>>>> Performance:
> > >>>>>>
> > >>>>>>
> > >>>>> Conf Runtime(sec) (Baseline) Events(/sec)
> > (Baseline) Results (Baseline)
> > >>>>>>
> > >>>>>>
> > >>>>> 0000 5,4 18677,6
> > 100000
> > >>>>>>
> > >>>>>>
> > >>>>> 0001 2,8 35511,4
> > 92000
> > >>>>>>
> > >>>>>>
> > >>>>> 0002 1,8 54318,3
> > 351
> > >>>>>>
> > >>>>>>
> > >>>>> 0003 2,4 41614,6
> > 580
> > >>>>>>
> > >>>>>>
> > >>>>> 0004 1,0 10341,3
> > 40
> > >>>>>>
> > >>>>>>
> > >>>>> 0005 3,4 29568,3
> > 12
> > >>>>>>
> > >>>>>>
> > >>>>> 0006 0,7 13369,0
> > 401
> > >>>>>>
> > >>>>>>
> > >>>>> 0007 2,8 36192,5
> > 1
> > >>>>>>
> > >>>>>>
> > >>>>> 0008 1,8 54854,6
> > 6000
> > >>>>>>
> > >>>>>>
> > >>>>> 0009 0,7 13369,0
> > 298
> > >>>>>>
> > >>>>>>
> > >>>>> 0010 3,4 29841,8
> > 2
> > >>>>>>
> > >>>>>>
> > >>>>> 0011 5,0 19932,2
> > 1919
> > >>>>>>
> > >>>>>>
> > >>>>> 0012 2,6 38835,0
> > 1919
> > >>>>>>
> > >>>>>
> >
> ==========================================================================================
> > >>>>>>
> > >>>>>> 5. Spark batch
> > >>>>>> Performance:
> > >>>>>>
> > >>>>>>
> > >>>>> Conf Runtime(sec) (Baseline) Events(/sec)
> > (Baseline) Results (Baseline)
> > >>>>>>
> > >>>>>>
> > >>>>> 0000 1,5 65445,0
> > 100000
> > >>>>>>
> > >>>>>>
> > >>>>> 0001 1,3 79491,3
> > 92000
> > >>>>>>
> > >>>>>>
> > >>>>> 0002 0,9 112107,6
> > 351
> > >>>>>>
> > >>>>>>
> > >>>>> 0003 2,0 48804,3
> > 580
> > >>>>>>
> > >>>>>>
> > >>>>> 0004 1,2 8382,2
> > 40
> > >>>>>>
> > >>>>>>
> > >>>>> 0005 2,0 50838,8
> > 12
> > >>>>>>
> > >>>>>>
> > >>>>> 0006 1,0 9699,3
> > 103
> > >>>>>>
> > >>>>>>
> > >>>>> 0007 2,3 43308,8
> > 1
> > >>>>>>
> > >>>>>>
> > >>>>> 0008 2,1 46794,6
> > 6000
> > >>>>>>
> > >>>>>>
> > >>>>> 0009 1,1 8976,7
> > 298
> > >>>>>>
> > >>>>>>
> > >>>>> 0010 1,6 62111,8
> > 1
> > >>>>>>
> > >>>>>>
> > >>>>> 0011 2,1 46598,3
> > 1919
> > >>>>>>
> > >>>>>>
> > >>>>> 0012 2,3 43687,2
> > 1919
> > >>>>>>
> > >>>>>
> >
> ==========================================================================================
> > >>>>>>
> > >>>>>>> Le mercredi 06 juin 2018 à 10:50 +0200, Etienne
> > Chauchot a écrit :
> > >>>>>>> Thanks JB for all your work ! I believe doing
> > the first
> > >>>>> gradle release
> > >>>>>>> must have been hard.
> > >>>>>>> I'll run Nexmark on the release and keep you
> posted.
> > >>>>>>>
> > >>>>>>> Best
> > >>>>>>> Etienne
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> Le mercredi 06 juin 2018 à 10:44 +0200,
> > Jean-Baptiste Onofré
> > >>>>> a écrit :
> > >>>>>>>> Hi everyone,
> > >>>>>>>>
> > >>>>>>>> Please review and vote on the release candidate
> > #1 for the
> > >>>>> version
> > >>>>>>>> 2.5.0, as follows:
> > >>>>>>>>
> > >>>>>>>> [ ] +1, Approve the release
> > >>>>>>>> [ ] -1, Do not approve the release (please
> > provide specific
> > >>>>> comments)
> > >>>>>>>>
> > >>>>>>>> NB: this is the first release using Gradle, so
> > don't be too
> > >>>>> harsh ;) A
> > >>>>>>>> PR about the release guide will follow thanks
> > to this release.
> > >>>>>>>>
> > >>>>>>>> The complete staging area is available for your
> > review,
> > >>>>> which includes:
> > >>>>>>>> * JIRA release notes [1],
> > >>>>>>>> * the official Apache source release to be
> > deployed to
> > >>>>> dist.apache.org <http://dist.apache.org>
> > <http://dist.apache.org/>
> > >>>>>>>> [2], which is signed with the key with
> > fingerprint C8282E76 [3],
> > >>>>>>>> * all artifacts to be deployed to the Maven
> Central
> > >>>>> Repository [4],
> > >>>>>>>> * source code tag "v2.5.0-RC1" [5],
> > >>>>>>>> * website pull request listing the release and
> > publishing
> > >>>>> the API
> > >>>>>>>> reference manual [6].
> > >>>>>>>> * Java artifacts were built with Gradle 4.7
> > (wrapper) and
> > >>>>> OpenJDK/Oracle
> > >>>>>>>> JDK 1.8.0_172 (Oracle Corporation 25.172-b11).
> > >>>>>>>> * Python artifacts are deployed along with the
> > source
> > >>>>> release to the
> > >>>>>>>> dist.apache.org <http://dist.apache.org>
> > <http://dist.apache.org/> [2].
> > >>>>>>>>
> > >>>>>>>> The vote will be open for at least 72 hours. It
> > is adopted
> > >>>>> by majority
> > >>>>>>>> approval, with at least 3 PMC affirmative votes.
> > >>>>>>>>
> > >>>>>>>> Thanks,
> > >>>>>>>> JB
> > >>>>>>>>
> > >>>>>>>> [1]
> > >>>>>>>>
> > >>>>>
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12342847
> > >>>>>>>> [2]
> > https://dist.apache.org/repos/dist/dev/beam/2.5.0/
> > >>>>>>>> [3]
> > https://dist.apache.org/repos/dist/release/beam/KEYS
> > >>>>>>>> [4]
> > >>>>>
> >
> https://repository.apache.org/content/repositories/orgapachebeam-1041/
> > >>>>>>>> [5]
> https://github.com/apache/beam/tree/v2.5.0-RC1
> > >>>>>>>> [6]
> https://github.com/apache/beam-site/pull/463
> > >>>>>>>>
> > >>>>>
> > >>
> > >> --
> > >> Got feedback? go/pabloem-feedback
> <https://goto.google.com/pabloem-feedback>
> > <https://goto.google.com/pabloem-feedback>
> > >
> > > --
> > > Jean-Baptiste Onofré
> > > jbonofre@apache.org <ma...@apache.org>
> > > http://blog.nanthrax.net
> > > Talend - http://www.talend.com
> >
> > --
> > Got feedback? go/pabloem-feedback
> <https://goto.google.com/pabloem-feedback>
> > <https://goto.google.com/pabloem-feedback>
> >
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
Re: [VOTE] Apache Beam, version 2.5.0, release candidate #1
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Any update about your vote and fix ?
Thanks !
Regards
JB
On 12/06/2018 04:02, Udi Meiri wrote:
> Another bug: reading from PubSub with_attributes=True is broken on
> Python with Dataflow.
> https://issues.apache.org/jira/browse/BEAM-4536
>
> JB, I'm making a PR that removes this keyword and I'd like to propose it
> as a cherrypick to 2.5.0.
> (feature should be fixed in the next release)
>
> On Mon, Jun 11, 2018 at 6:19 PM Chamikara Jayalath <chamikara@google.com
> <ma...@google.com>> wrote:
>
> FYI: looks like Python tests are failing for Windows. JIRA
> is https://issues.apache.org/jira/browse/BEAM-4535.
>
> I don't think this is a release blocker but this should probably go
> in release notes (for any user that tries to run tests on Python
> source build). And we should try to incorporate a fix if we happen
> to cut another release candidate for some reason.
>
> Thanks,
> Cham
>
> On Mon, Jun 11, 2018 at 5:46 PM Pablo Estrada <pabloem@google.com
> <ma...@google.com>> wrote:
>
> Thanks everyone who has pitched in to validate the release!
>
> Boyuan Zhang and I have also run a few pipelines, and verified
> that they work properly (see release validation spreadsheet[1]).
>
> We have also found that the Game Stats pipeline is failing in
> Python Streaming Dataflow. I have filed BEAM-4534[2]. This is
> not a blocker, since Python streaming is not yet fully supported.
>
> It seems that the uploaded artifacts look good.
>
> We have noticed that the Python artifacts are still missing
> Python wheel files (compare [3] and [4]). JB, could you please
> add the wheel files? Boyuan and I can try to help you prepare
> them / upload them if necessary. Please let us know.
>
> Thanks again!
> -P.
>
> [1] https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=152451807
> [2] https://issues.apache.org/jira/browse/BEAM-4534
> [3] https://dist.apache.org/repos/dist/dev/beam/2.4.0/
> [4] https://dist.apache.org/repos/dist/dev/beam/2.5.0/
>
> On Mon, Jun 11, 2018 at 12:37 PM Alan Myrvold
> <amyrvold@google.com <ma...@google.com>> wrote:
>
> +1 (non-binding)
>
> tested some of the quickstarts
>
> On Sun, Jun 10, 2018 at 1:39 AM Tim
> <timrobertson100@gmail.com
> <ma...@gmail.com>> wrote:
>
> Tested by our team:
> - mvn inclusion
> - Avro, ES, Hadoop IF IO
> - Pipelines run on Spark (Cloudera 5.12.0 YARN cluster)
> - Reviewed release notes
>
> +1
>
> Thanks also to everyone who helped get over the gradle
> hurdle and in particular to JB.
>
> Tim
>
> > On 9 Jun 2018, at 05:56, Jean-Baptiste Onofré
> <jb@nanthrax.net <ma...@nanthrax.net>> wrote:
> >
> > No problem Pablo.
> >
> > The vote period is a minimum, it can be extended as
> requested or if we
> > don't have the minimum of 3 binding votes.
> >
> > Regards
> > JB
> >
> >> On 09/06/2018 01:54, Pablo Estrada wrote:
> >> Hello all,
> >> I'd like to request an extension of the voting period
> until Monday
> >> evening (US time, so later in other geographical
> regions). This is
> >> because we were only now able to publish Dataflow
> Workers, and have not
> >> had the chance to run release validation tests on
> them. The extension
> >> will allow us to validate and vote by Monday.
> >> Is this acceptable to the community?
> >>
> >> Best
> >> -P.
> >>
> >> On Fri, Jun 8, 2018 at 6:20 AM Alexey Romanenko
> >> <aromanenko.dev@gmail.com
> <ma...@gmail.com>
> <mailto:aromanenko.dev@gmail.com
> <ma...@gmail.com>>> wrote:
> >>
> >> Thank you JB for your work!
> >>
> >> I tested running simple streaming (/KafkaIO/) and
> batch (/TextIO /
> >> HDFS/) pipelines with SparkRunner on YARN cluster
> - it works fine.
> >>
> >> WBR,
> >> Alexey
> >>
> >>
> >>> On 8 Jun 2018, at 10:00, Etienne Chauchot
> <echauchot@apache.org <ma...@apache.org>
> >>> <mailto:echauchot@apache.org
> <ma...@apache.org>>> wrote:
> >>>
> >>> I forgot to vote:
> >>> +1 (non binding).
> >>> What I tested:
> >>> - no functional or performance regression
> comparing to v2.4
> >>> - dependencies in the poms are ok
> >>>
> >>> Etienne
> >>>> Le vendredi 08 juin 2018 à 08:27 +0200, Romain
> Manni-Bucau a écrit :
> >>>> +1 (non-binding), mainstream usage is not broken
> by the pom
> >>>> changes and runtime has no known regression
> compared to the 2.4.0
> >>>>
> >>>> (side note: kudo to JB for this build tool
> change release, I know
> >>>> how it can hurt ;))
> >>>>
> >>>> Romain Manni-Bucau
> >>>> @rmannibucau <https://twitter.com/rmannibucau>
> | Blog
> >>>> <https://rmannibucau.metawerx.net/> | Old Blog
> >>>> <http://rmannibucau.wordpress.com/> | Github
> >>>> <https://github.com/rmannibucau> | LinkedIn
> >>>> <https://www.linkedin.com/in/rmannibucau> | Book
> >>>>
> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
> >>>>
> >>>>
> >>>> Le jeu. 7 juin 2018 à 16:17, Jean-Baptiste Onofré
> >>>> <jb@nanthrax.net <ma...@nanthrax.net>
> <mailto:jb@nanthrax.net <ma...@nanthrax.net>>> a écrit :
> >>>>> Thanks for the details Etienne !
> >>>>>
> >>>>> The good news is that the artifacts seem OK and
> the overall Nexmark
> >>>>> results are consistent with the 2.4.0 release ones.
> >>>>>
> >>>>> I'm starting a complete review using the
> beam-samples as well.
> >>>>>
> >>>>> Regards
> >>>>> JB
> >>>>>
> >>>>>> On 07/06/2018 16:14, Etienne Chauchot wrote:
> >>>>>> Hi,
> >>>>>> I've just run the nexmark queries on v2.5.0-RC1 tag
> >>>>>> What we can notice:
> >>>>>> - query 3 (exercises CoGroupByKey, state and
> timer) shows
> >>>>> different
> >>>>>> output with DR between batch and streaming and
> with the other
> >>>>> runners =>
> >>>>>> I compared with v2.4 there were still these
> differences but with
> >>>>>> different output size numbers
> >>>>>>
> >>>>>> - query 6 (exercises specialized combiner) shows
> different output
> >>>>>> between the runners => the correct output is 401.
> strange that
> >>>>> in batch
> >>>>>> mode some runners output les Sellers. I compared
> with v2.4
> >>>>> same output
> >>>>>>
> >>>>>> - response time of query 7 (exercices Max
> transform, fanout
> >>>>> and side
> >>>>>> input) is very slow on DR => I compared with v2.4
> , comparable
> >>>>> execution
> >>>>>> times
> >>>>>>
> >>>>>> I'm not comparing q10 because it is a write to
> GCS so it is
> >>>>> very specific.
> >>>>>>
> >>>>>> => Basically no regression comparing to v2.4
> >>>>>>
> >>>>>> For the record here is the output (waiting for
> ongoing perfkit
> >>>>> integration):
> >>>>>>
> >>>>>>
> >>>>>> 1. DR batch
> >>>>>>
> >>>>>> Performance:
> >>>>>>
> >>>>>>
> >>>>> Conf Runtime(sec) (Baseline) Events(/sec)
> (Baseline) Results (Baseline)
> >>>>>>
> >>>>>>
> >>>>> 0000 5,8 17283,1
> 100000
> >>>>>>
> >>>>>>
> >>>>> 0001 3,2 31104,2
> 92000
> >>>>>>
> >>>>>>
> >>>>> 0002 1,2 82918,7
> 351
> >>>>>>
> >>>>>>
> >>>>> 0003 2,2 46210,7
> 458
> >>>>>>
> >>>>>>
> >>>>> 0004 1,2 8503,4
> 40
> >>>>>>
> >>>>>>
> >>>>> 0005 4,0 25220,7
> 12
> >>>>>>
> >>>>>>
> >>>>> 0006 0,9 11148,3
> 401
> >>>>>>
> >>>>>>
> >>>>> 0007 13,2 7580,9
> 1
> >>>>>>
> >>>>>>
> >>>>> 0008 1,5 67340,1
> 6000
> >>>>>>
> >>>>>>
> >>>>> 0009 0,7 14025,2
> 298
> >>>>>>
> >>>>>>
> >>>>> 0010 12,8 7793,0
> 1
> >>>>>>
> >>>>>>
> >>>>> 0011 2,4 42319,1
> 1919
> >>>>>>
> >>>>>>
> >>>>> 0012 1,6 61462,8
> 1919
> >>>>>>
> >>>>>
> ==========================================================================================
> >>>>>>
> >>>>>> 2. DR streaming
> >>>>>>
> >>>>>> Performance:
> >>>>>>
> >>>>>>
> >>>>> Conf Runtime(sec) (Baseline) Events(/sec)
> (Baseline) Results (Baseline)
> >>>>>>
> >>>>>>
> >>>>> 0000 6,5 15285,8
> 100000
> >>>>>>
> >>>>>>
> >>>>> 0001 3,7 27397,3
> 92000
> >>>>>>
> >>>>>>
> >>>>> 0002 1,4 69108,5
> 351
> >>>>>>
> >>>>>>
> >>>>> 0003 3,2 31181,8
> 447
> >>>>>>
> >>>>>>
> >>>>> 0004 1,2 8361,2
> 40
> >>>>>>
> >>>>>>
> >>>>> 0005 5,3 18903,6
> 12
> >>>>>>
> >>>>>>
> >>>>> 0006 0,9 11111,1
> 401
> >>>>>>
> >>>>>>
> >>>>> 0007 82,5 1212,2
> 1
> >>>>>>
> >>>>>>
> >>>>> 0008 2,0 51072,5
> 6000
> >>>>>>
> >>>>>>
> >>>>> 0009 0,8 12903,2
> 298
> >>>>>>
> >>>>>>
> >>>>> 0010 49,5 2021,8
> 1
> >>>>>>
> >>>>>>
> >>>>> 0011 3,9 25667,4
> 1919
> >>>>>>
> >>>>>>
> >>>>> 0012 2,4 41067,8
> 1919
> >>>>>>
> >>>>>
> ==========================================================================================
> >>>>>>
> >>>>>> 3. Flink batch
> >>>>>> Performance:
> >>>>>>
> >>>>>>
> >>>>> Conf Runtime(sec) (Baseline) Events(/sec)
> (Baseline) Results (Baseline)
> >>>>>>
> >>>>>>
> >>>>> 0000 1,0 97656,3
> 100000
> >>>>>>
> >>>>>>
> >>>>> 0001 0,7 141643,1
> 92000
> >>>>>>
> >>>>>>
> >>>>> 0002 0,4 228310,5
> 351
> >>>>>>
> >>>>>>
> >>>>> 0003 1,6 64020,5
> 580
> >>>>>>
> >>>>>>
> >>>>> 0004 0,7 13831,3
> 40
> >>>>>>
> >>>>>>
> >>>>> 0005 1,4 72939,5
> 12
> >>>>>>
> >>>>>>
> >>>>> 0006 0,5 20491,8
> 103
> >>>>>>
> >>>>>>
> >>>>> 0007 1,3 74239,0
> 1
> >>>>>>
> >>>>>>
> >>>>> 0008 0,8 121506,7
> 6000
> >>>>>>
> >>>>>>
> >>>>> 0009 0,6 17953,3
> 298
> >>>>>>
> >>>>>>
> >>>>> 0010 1,3 74682,6
> 1
> >>>>>>
> >>>>>>
> >>>>> 0011 1,1 92936,8
> 1919
> >>>>>>
> >>>>>>
> >>>>> 0012 0,8 123001,2
> 1919
> >>>>>>
> >>>>>
> ==========================================================================================
> >>>>>>
> >>>>>> 4. Flink streaming
> >>>>>> Performance:
> >>>>>>
> >>>>>>
> >>>>> Conf Runtime(sec) (Baseline) Events(/sec)
> (Baseline) Results (Baseline)
> >>>>>>
> >>>>>>
> >>>>> 0000 5,4 18677,6
> 100000
> >>>>>>
> >>>>>>
> >>>>> 0001 2,8 35511,4
> 92000
> >>>>>>
> >>>>>>
> >>>>> 0002 1,8 54318,3
> 351
> >>>>>>
> >>>>>>
> >>>>> 0003 2,4 41614,6
> 580
> >>>>>>
> >>>>>>
> >>>>> 0004 1,0 10341,3
> 40
> >>>>>>
> >>>>>>
> >>>>> 0005 3,4 29568,3
> 12
> >>>>>>
> >>>>>>
> >>>>> 0006 0,7 13369,0
> 401
> >>>>>>
> >>>>>>
> >>>>> 0007 2,8 36192,5
> 1
> >>>>>>
> >>>>>>
> >>>>> 0008 1,8 54854,6
> 6000
> >>>>>>
> >>>>>>
> >>>>> 0009 0,7 13369,0
> 298
> >>>>>>
> >>>>>>
> >>>>> 0010 3,4 29841,8
> 2
> >>>>>>
> >>>>>>
> >>>>> 0011 5,0 19932,2
> 1919
> >>>>>>
> >>>>>>
> >>>>> 0012 2,6 38835,0
> 1919
> >>>>>>
> >>>>>
> ==========================================================================================
> >>>>>>
> >>>>>> 5. Spark batch
> >>>>>> Performance:
> >>>>>>
> >>>>>>
> >>>>> Conf Runtime(sec) (Baseline) Events(/sec)
> (Baseline) Results (Baseline)
> >>>>>>
> >>>>>>
> >>>>> 0000 1,5 65445,0
> 100000
> >>>>>>
> >>>>>>
> >>>>> 0001 1,3 79491,3
> 92000
> >>>>>>
> >>>>>>
> >>>>> 0002 0,9 112107,6
> 351
> >>>>>>
> >>>>>>
> >>>>> 0003 2,0 48804,3
> 580
> >>>>>>
> >>>>>>
> >>>>> 0004 1,2 8382,2
> 40
> >>>>>>
> >>>>>>
> >>>>> 0005 2,0 50838,8
> 12
> >>>>>>
> >>>>>>
> >>>>> 0006 1,0 9699,3
> 103
> >>>>>>
> >>>>>>
> >>>>> 0007 2,3 43308,8
> 1
> >>>>>>
> >>>>>>
> >>>>> 0008 2,1 46794,6
> 6000
> >>>>>>
> >>>>>>
> >>>>> 0009 1,1 8976,7
> 298
> >>>>>>
> >>>>>>
> >>>>> 0010 1,6 62111,8
> 1
> >>>>>>
> >>>>>>
> >>>>> 0011 2,1 46598,3
> 1919
> >>>>>>
> >>>>>>
> >>>>> 0012 2,3 43687,2
> 1919
> >>>>>>
> >>>>>
> ==========================================================================================
> >>>>>>
> >>>>>>> Le mercredi 06 juin 2018 à 10:50 +0200, Etienne
> Chauchot a écrit :
> >>>>>>> Thanks JB for all your work ! I believe doing
> the first
> >>>>> gradle release
> >>>>>>> must have been hard.
> >>>>>>> I'll run Nexmark on the release and keep you posted.
> >>>>>>>
> >>>>>>> Best
> >>>>>>> Etienne
> >>>>>>>
> >>>>>>>
> >>>>>>> Le mercredi 06 juin 2018 à 10:44 +0200,
> Jean-Baptiste Onofré
> >>>>> a écrit :
> >>>>>>>> Hi everyone,
> >>>>>>>>
> >>>>>>>> Please review and vote on the release candidate
> #1 for the
> >>>>> version
> >>>>>>>> 2.5.0, as follows:
> >>>>>>>>
> >>>>>>>> [ ] +1, Approve the release
> >>>>>>>> [ ] -1, Do not approve the release (please
> provide specific
> >>>>> comments)
> >>>>>>>>
> >>>>>>>> NB: this is the first release using Gradle, so
> don't be too
> >>>>> harsh ;) A
> >>>>>>>> PR about the release guide will follow thanks
> to this release.
> >>>>>>>>
> >>>>>>>> The complete staging area is available for your
> review,
> >>>>> which includes:
> >>>>>>>> * JIRA release notes [1],
> >>>>>>>> * the official Apache source release to be
> deployed to
> >>>>> dist.apache.org <http://dist.apache.org>
> <http://dist.apache.org/>
> >>>>>>>> [2], which is signed with the key with
> fingerprint C8282E76 [3],
> >>>>>>>> * all artifacts to be deployed to the Maven Central
> >>>>> Repository [4],
> >>>>>>>> * source code tag "v2.5.0-RC1" [5],
> >>>>>>>> * website pull request listing the release and
> publishing
> >>>>> the API
> >>>>>>>> reference manual [6].
> >>>>>>>> * Java artifacts were built with Gradle 4.7
> (wrapper) and
> >>>>> OpenJDK/Oracle
> >>>>>>>> JDK 1.8.0_172 (Oracle Corporation 25.172-b11).
> >>>>>>>> * Python artifacts are deployed along with the
> source
> >>>>> release to the
> >>>>>>>> dist.apache.org <http://dist.apache.org>
> <http://dist.apache.org/> [2].
> >>>>>>>>
> >>>>>>>> The vote will be open for at least 72 hours. It
> is adopted
> >>>>> by majority
> >>>>>>>> approval, with at least 3 PMC affirmative votes.
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> JB
> >>>>>>>>
> >>>>>>>> [1]
> >>>>>>>>
> >>>>>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12342847
> >>>>>>>> [2]
> https://dist.apache.org/repos/dist/dev/beam/2.5.0/
> >>>>>>>> [3]
> https://dist.apache.org/repos/dist/release/beam/KEYS
> >>>>>>>> [4]
> >>>>>
> https://repository.apache.org/content/repositories/orgapachebeam-1041/
> >>>>>>>> [5] https://github.com/apache/beam/tree/v2.5.0-RC1
> >>>>>>>> [6] https://github.com/apache/beam-site/pull/463
> >>>>>>>>
> >>>>>
> >>
> >> --
> >> Got feedback? go/pabloem-feedback
> <https://goto.google.com/pabloem-feedback>
> >
> > --
> > Jean-Baptiste Onofré
> > jbonofre@apache.org <ma...@apache.org>
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
>
> --
> Got feedback? go/pabloem-feedback
> <https://goto.google.com/pabloem-feedback>
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: [VOTE] Apache Beam, version 2.5.0, release candidate #1
Posted by Udi Meiri <eh...@google.com>.
Another bug: reading from PubSub with_attributes=True is broken on Python
with Dataflow.
https://issues.apache.org/jira/browse/BEAM-4536
JB, I'm making a PR that removes this keyword and I'd like to propose it as
a cherrypick to 2.5.0.
(feature should be fixed in the next release)
On Mon, Jun 11, 2018 at 6:19 PM Chamikara Jayalath <ch...@google.com>
wrote:
> FYI: looks like Python tests are failing for Windows. JIRA is
> https://issues.apache.org/jira/browse/BEAM-4535.
>
> I don't think this is a release blocker but this should probably go in
> release notes (for any user that tries to run tests on Python source
> build). And we should try to incorporate a fix if we happen to cut another
> release candidate for some reason.
>
> Thanks,
> Cham
>
> On Mon, Jun 11, 2018 at 5:46 PM Pablo Estrada <pa...@google.com> wrote:
>
>> Thanks everyone who has pitched in to validate the release!
>>
>> Boyuan Zhang and I have also run a few pipelines, and verified that they
>> work properly (see release validation spreadsheet[1]).
>>
>> We have also found that the Game Stats pipeline is failing in Python
>> Streaming Dataflow. I have filed BEAM-4534[2]. This is not a blocker, since
>> Python streaming is not yet fully supported.
>>
>> It seems that the uploaded artifacts look good.
>>
>> We have noticed that the Python artifacts are still missing Python wheel
>> files (compare [3] and [4]). JB, could you please add the wheel files?
>> Boyuan and I can try to help you prepare them / upload them if necessary.
>> Please let us know.
>>
>> Thanks again!
>> -P.
>>
>> [1]
>> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=152451807
>> [2] https://issues.apache.org/jira/browse/BEAM-4534
>> [3] https://dist.apache.org/repos/dist/dev/beam/2.4.0/
>> [4] https://dist.apache.org/repos/dist/dev/beam/2.5.0/
>>
>> On Mon, Jun 11, 2018 at 12:37 PM Alan Myrvold <am...@google.com>
>> wrote:
>>
>>> +1 (non-binding)
>>>
>>> tested some of the quickstarts
>>>
>>> On Sun, Jun 10, 2018 at 1:39 AM Tim <ti...@gmail.com> wrote:
>>>
>>>> Tested by our team:
>>>> - mvn inclusion
>>>> - Avro, ES, Hadoop IF IO
>>>> - Pipelines run on Spark (Cloudera 5.12.0 YARN cluster)
>>>> - Reviewed release notes
>>>>
>>>> +1
>>>>
>>>> Thanks also to everyone who helped get over the gradle hurdle and in
>>>> particular to JB.
>>>>
>>>> Tim
>>>>
>>>> > On 9 Jun 2018, at 05:56, Jean-Baptiste Onofré <jb...@nanthrax.net>
>>>> wrote:
>>>> >
>>>> > No problem Pablo.
>>>> >
>>>> > The vote period is a minimum, it can be extended as requested or if we
>>>> > don't have the minimum of 3 binding votes.
>>>> >
>>>> > Regards
>>>> > JB
>>>> >
>>>> >> On 09/06/2018 01:54, Pablo Estrada wrote:
>>>> >> Hello all,
>>>> >> I'd like to request an extension of the voting period until Monday
>>>> >> evening (US time, so later in other geographical regions). This is
>>>> >> because we were only now able to publish Dataflow Workers, and have
>>>> not
>>>> >> had the chance to run release validation tests on them. The extension
>>>> >> will allow us to validate and vote by Monday.
>>>> >> Is this acceptable to the community?
>>>> >>
>>>> >> Best
>>>> >> -P.
>>>> >>
>>>> >> On Fri, Jun 8, 2018 at 6:20 AM Alexey Romanenko
>>>> >> <aromanenko.dev@gmail.com <ma...@gmail.com>> wrote:
>>>> >>
>>>> >> Thank you JB for your work!
>>>> >>
>>>> >> I tested running simple streaming (/KafkaIO/) and batch (/TextIO /
>>>> >> HDFS/) pipelines with SparkRunner on YARN cluster - it works fine.
>>>> >>
>>>> >> WBR,
>>>> >> Alexey
>>>> >>
>>>> >>
>>>> >>> On 8 Jun 2018, at 10:00, Etienne Chauchot <echauchot@apache.org
>>>> >>> <ma...@apache.org>> wrote:
>>>> >>>
>>>> >>> I forgot to vote:
>>>> >>> +1 (non binding).
>>>> >>> What I tested:
>>>> >>> - no functional or performance regression comparing to v2.4
>>>> >>> - dependencies in the poms are ok
>>>> >>>
>>>> >>> Etienne
>>>> >>>> Le vendredi 08 juin 2018 à 08:27 +0200, Romain Manni-Bucau a
>>>> écrit :
>>>> >>>> +1 (non-binding), mainstream usage is not broken by the pom
>>>> >>>> changes and runtime has no known regression compared to the
>>>> 2.4.0
>>>> >>>>
>>>> >>>> (side note: kudo to JB for this build tool change release, I
>>>> know
>>>> >>>> how it can hurt ;))
>>>> >>>>
>>>> >>>> Romain Manni-Bucau
>>>> >>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>>> >>>> <https://rmannibucau.metawerx.net/> | Old Blog
>>>> >>>> <http://rmannibucau.wordpress.com/> | Github
>>>> >>>> <https://github.com/rmannibucau> | LinkedIn
>>>> >>>> <https://www.linkedin.com/in/rmannibucau> | Book
>>>> >>>> <
>>>> https://www.packtpub.com/application-development/java-ee-8-high-performance
>>>> >
>>>> >>>>
>>>> >>>>
>>>> >>>> Le jeu. 7 juin 2018 à 16:17, Jean-Baptiste Onofré
>>>> >>>> <jb@nanthrax.net <ma...@nanthrax.net>> a écrit :
>>>> >>>>> Thanks for the details Etienne !
>>>> >>>>>
>>>> >>>>> The good news is that the artifacts seem OK and the overall
>>>> Nexmark
>>>> >>>>> results are consistent with the 2.4.0 release ones.
>>>> >>>>>
>>>> >>>>> I'm starting a complete review using the beam-samples as well.
>>>> >>>>>
>>>> >>>>> Regards
>>>> >>>>> JB
>>>> >>>>>
>>>> >>>>>> On 07/06/2018 16:14, Etienne Chauchot wrote:
>>>> >>>>>> Hi,
>>>> >>>>>> I've just run the nexmark queries on v2.5.0-RC1 tag
>>>> >>>>>> What we can notice:
>>>> >>>>>> - query 3 (exercises CoGroupByKey, state and timer) shows
>>>> >>>>> different
>>>> >>>>>> output with DR between batch and streaming and with the other
>>>> >>>>> runners =>
>>>> >>>>>> I compared with v2.4 there were still these differences but with
>>>> >>>>>> different output size numbers
>>>> >>>>>>
>>>> >>>>>> - query 6 (exercises specialized combiner) shows different output
>>>> >>>>>> between the runners => the correct output is 401. strange that
>>>> >>>>> in batch
>>>> >>>>>> mode some runners output les Sellers. I compared with v2.4
>>>> >>>>> same output
>>>> >>>>>>
>>>> >>>>>> - response time of query 7 (exercices Max transform, fanout
>>>> >>>>> and side
>>>> >>>>>> input) is very slow on DR => I compared with v2.4 , comparable
>>>> >>>>> execution
>>>> >>>>>> times
>>>> >>>>>>
>>>> >>>>>> I'm not comparing q10 because it is a write to GCS so it is
>>>> >>>>> very specific.
>>>> >>>>>>
>>>> >>>>>> => Basically no regression comparing to v2.4
>>>> >>>>>>
>>>> >>>>>> For the record here is the output (waiting for ongoing perfkit
>>>> >>>>> integration):
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>>> 1. DR batch
>>>> >>>>>>
>>>> >>>>>> Performance:
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> Conf Runtime(sec) (Baseline) Events(/sec) (Baseline)
>>>> Results (Baseline)
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0000 5,8 17283,1
>>>> 100000
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0001 3,2 31104,2
>>>> 92000
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0002 1,2 82918,7
>>>> 351
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0003 2,2 46210,7
>>>> 458
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0004 1,2 8503,4
>>>> 40
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0005 4,0 25220,7
>>>> 12
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0006 0,9 11148,3
>>>> 401
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0007 13,2 7580,9
>>>> 1
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0008 1,5 67340,1
>>>> 6000
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0009 0,7 14025,2
>>>> 298
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0010 12,8 7793,0
>>>> 1
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0011 2,4 42319,1
>>>> 1919
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0012 1,6 61462,8
>>>> 1919
>>>> >>>>>>
>>>> >>>>>
>>>> ==========================================================================================
>>>> >>>>>>
>>>> >>>>>> 2. DR streaming
>>>> >>>>>>
>>>> >>>>>> Performance:
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> Conf Runtime(sec) (Baseline) Events(/sec) (Baseline)
>>>> Results (Baseline)
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0000 6,5 15285,8
>>>> 100000
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0001 3,7 27397,3
>>>> 92000
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0002 1,4 69108,5
>>>> 351
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0003 3,2 31181,8
>>>> 447
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0004 1,2 8361,2
>>>> 40
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0005 5,3 18903,6
>>>> 12
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0006 0,9 11111,1
>>>> 401
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0007 82,5 1212,2
>>>> 1
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0008 2,0 51072,5
>>>> 6000
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0009 0,8 12903,2
>>>> 298
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0010 49,5 2021,8
>>>> 1
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0011 3,9 25667,4
>>>> 1919
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0012 2,4 41067,8
>>>> 1919
>>>> >>>>>>
>>>> >>>>>
>>>> ==========================================================================================
>>>> >>>>>>
>>>> >>>>>> 3. Flink batch
>>>> >>>>>> Performance:
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> Conf Runtime(sec) (Baseline) Events(/sec) (Baseline)
>>>> Results (Baseline)
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0000 1,0 97656,3
>>>> 100000
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0001 0,7 141643,1
>>>> 92000
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0002 0,4 228310,5
>>>> 351
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0003 1,6 64020,5
>>>> 580
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0004 0,7 13831,3
>>>> 40
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0005 1,4 72939,5
>>>> 12
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0006 0,5 20491,8
>>>> 103
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0007 1,3 74239,0
>>>> 1
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0008 0,8 121506,7
>>>> 6000
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0009 0,6 17953,3
>>>> 298
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0010 1,3 74682,6
>>>> 1
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0011 1,1 92936,8
>>>> 1919
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0012 0,8 123001,2
>>>> 1919
>>>> >>>>>>
>>>> >>>>>
>>>> ==========================================================================================
>>>> >>>>>>
>>>> >>>>>> 4. Flink streaming
>>>> >>>>>> Performance:
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> Conf Runtime(sec) (Baseline) Events(/sec) (Baseline)
>>>> Results (Baseline)
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0000 5,4 18677,6
>>>> 100000
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0001 2,8 35511,4
>>>> 92000
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0002 1,8 54318,3
>>>> 351
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0003 2,4 41614,6
>>>> 580
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0004 1,0 10341,3
>>>> 40
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0005 3,4 29568,3
>>>> 12
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0006 0,7 13369,0
>>>> 401
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0007 2,8 36192,5
>>>> 1
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0008 1,8 54854,6
>>>> 6000
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0009 0,7 13369,0
>>>> 298
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0010 3,4 29841,8
>>>> 2
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0011 5,0 19932,2
>>>> 1919
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0012 2,6 38835,0
>>>> 1919
>>>> >>>>>>
>>>> >>>>>
>>>> ==========================================================================================
>>>> >>>>>>
>>>> >>>>>> 5. Spark batch
>>>> >>>>>> Performance:
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> Conf Runtime(sec) (Baseline) Events(/sec) (Baseline)
>>>> Results (Baseline)
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0000 1,5 65445,0
>>>> 100000
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0001 1,3 79491,3
>>>> 92000
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0002 0,9 112107,6
>>>> 351
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0003 2,0 48804,3
>>>> 580
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0004 1,2 8382,2
>>>> 40
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0005 2,0 50838,8
>>>> 12
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0006 1,0 9699,3
>>>> 103
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0007 2,3 43308,8
>>>> 1
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0008 2,1 46794,6
>>>> 6000
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0009 1,1 8976,7
>>>> 298
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0010 1,6 62111,8
>>>> 1
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0011 2,1 46598,3
>>>> 1919
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>> 0012 2,3 43687,2
>>>> 1919
>>>> >>>>>>
>>>> >>>>>
>>>> ==========================================================================================
>>>> >>>>>>
>>>> >>>>>>> Le mercredi 06 juin 2018 à 10:50 +0200, Etienne Chauchot a
>>>> écrit :
>>>> >>>>>>> Thanks JB for all your work ! I believe doing the first
>>>> >>>>> gradle release
>>>> >>>>>>> must have been hard.
>>>> >>>>>>> I'll run Nexmark on the release and keep you posted.
>>>> >>>>>>>
>>>> >>>>>>> Best
>>>> >>>>>>> Etienne
>>>> >>>>>>>
>>>> >>>>>>>
>>>> >>>>>>> Le mercredi 06 juin 2018 à 10:44 +0200, Jean-Baptiste Onofré
>>>> >>>>> a écrit :
>>>> >>>>>>>> Hi everyone,
>>>> >>>>>>>>
>>>> >>>>>>>> Please review and vote on the release candidate #1 for the
>>>> >>>>> version
>>>> >>>>>>>> 2.5.0, as follows:
>>>> >>>>>>>>
>>>> >>>>>>>> [ ] +1, Approve the release
>>>> >>>>>>>> [ ] -1, Do not approve the release (please provide specific
>>>> >>>>> comments)
>>>> >>>>>>>>
>>>> >>>>>>>> NB: this is the first release using Gradle, so don't be too
>>>> >>>>> harsh ;) A
>>>> >>>>>>>> PR about the release guide will follow thanks to this release.
>>>> >>>>>>>>
>>>> >>>>>>>> The complete staging area is available for your review,
>>>> >>>>> which includes:
>>>> >>>>>>>> * JIRA release notes [1],
>>>> >>>>>>>> * the official Apache source release to be deployed to
>>>> >>>>> dist.apache.org <http://dist.apache.org/>
>>>> >>>>>>>> [2], which is signed with the key with fingerprint C8282E76
>>>> [3],
>>>> >>>>>>>> * all artifacts to be deployed to the Maven Central
>>>> >>>>> Repository [4],
>>>> >>>>>>>> * source code tag "v2.5.0-RC1" [5],
>>>> >>>>>>>> * website pull request listing the release and publishing
>>>> >>>>> the API
>>>> >>>>>>>> reference manual [6].
>>>> >>>>>>>> * Java artifacts were built with Gradle 4.7 (wrapper) and
>>>> >>>>> OpenJDK/Oracle
>>>> >>>>>>>> JDK 1.8.0_172 (Oracle Corporation 25.172-b11).
>>>> >>>>>>>> * Python artifacts are deployed along with the source
>>>> >>>>> release to the
>>>> >>>>>>>> dist.apache.org <http://dist.apache.org/> [2].
>>>> >>>>>>>>
>>>> >>>>>>>> The vote will be open for at least 72 hours. It is adopted
>>>> >>>>> by majority
>>>> >>>>>>>> approval, with at least 3 PMC affirmative votes.
>>>> >>>>>>>>
>>>> >>>>>>>> Thanks,
>>>> >>>>>>>> JB
>>>> >>>>>>>>
>>>> >>>>>>>> [1]
>>>> >>>>>>>>
>>>> >>>>>
>>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12342847
>>>> >>>>>>>> [2] https://dist.apache.org/repos/dist/dev/beam/2.5.0/
>>>> >>>>>>>> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
>>>> >>>>>>>> [4]
>>>> >>>>>
>>>> https://repository.apache.org/content/repositories/orgapachebeam-1041/
>>>> >>>>>>>> [5] https://github.com/apache/beam/tree/v2.5.0-RC1
>>>> >>>>>>>> [6] https://github.com/apache/beam-site/pull/463
>>>> >>>>>>>>
>>>> >>>>>
>>>> >>
>>>> >> --
>>>> >> Got feedback? go/pabloem-feedback
>>>> <https://goto.google.com/pabloem-feedback>
>>>> >
>>>> > --
>>>> > Jean-Baptiste Onofré
>>>> > jbonofre@apache.org
>>>> > http://blog.nanthrax.net
>>>> > Talend - http://www.talend.com
>>>>
>>> --
>> Got feedback? go/pabloem-feedback
>> <https://goto.google.com/pabloem-feedback>
>>
>
Re: [VOTE] Apache Beam, version 2.5.0, release candidate #1
Posted by Chamikara Jayalath <ch...@google.com>.
FYI: looks like Python tests are failing for Windows. JIRA is
https://issues.apache.org/jira/browse/BEAM-4535.
I don't think this is a release blocker but this should probably go in
release notes (for any user that tries to run tests on Python source
build). And we should try to incorporate a fix if we happen to cut another
release candidate for some reason.
Thanks,
Cham
On Mon, Jun 11, 2018 at 5:46 PM Pablo Estrada <pa...@google.com> wrote:
> Thanks everyone who has pitched in to validate the release!
>
> Boyuan Zhang and I have also run a few pipelines, and verified that they
> work properly (see release validation spreadsheet[1]).
>
> We have also found that the Game Stats pipeline is failing in Python
> Streaming Dataflow. I have filed BEAM-4534[2]. This is not a blocker, since
> Python streaming is not yet fully supported.
>
> It seems that the uploaded artifacts look good.
>
> We have noticed that the Python artifacts are still missing Python wheel
> files (compare [3] and [4]). JB, could you please add the wheel files?
> Boyuan and I can try to help you prepare them / upload them if necessary.
> Please let us know.
>
> Thanks again!
> -P.
>
> [1]
> https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=152451807
> [2] https://issues.apache.org/jira/browse/BEAM-4534
> [3] https://dist.apache.org/repos/dist/dev/beam/2.4.0/
> [4] https://dist.apache.org/repos/dist/dev/beam/2.5.0/
>
> On Mon, Jun 11, 2018 at 12:37 PM Alan Myrvold <am...@google.com> wrote:
>
>> +1 (non-binding)
>>
>> tested some of the quickstarts
>>
>> On Sun, Jun 10, 2018 at 1:39 AM Tim <ti...@gmail.com> wrote:
>>
>>> Tested by our team:
>>> - mvn inclusion
>>> - Avro, ES, Hadoop IF IO
>>> - Pipelines run on Spark (Cloudera 5.12.0 YARN cluster)
>>> - Reviewed release notes
>>>
>>> +1
>>>
>>> Thanks also to everyone who helped get over the gradle hurdle and in
>>> particular to JB.
>>>
>>> Tim
>>>
>>> > On 9 Jun 2018, at 05:56, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>>> >
>>> > No problem Pablo.
>>> >
>>> > The vote period is a minimum, it can be extended as requested or if we
>>> > don't have the minimum of 3 binding votes.
>>> >
>>> > Regards
>>> > JB
>>> >
>>> >> On 09/06/2018 01:54, Pablo Estrada wrote:
>>> >> Hello all,
>>> >> I'd like to request an extension of the voting period until Monday
>>> >> evening (US time, so later in other geographical regions). This is
>>> >> because we were only now able to publish Dataflow Workers, and have
>>> not
>>> >> had the chance to run release validation tests on them. The extension
>>> >> will allow us to validate and vote by Monday.
>>> >> Is this acceptable to the community?
>>> >>
>>> >> Best
>>> >> -P.
>>> >>
>>> >> On Fri, Jun 8, 2018 at 6:20 AM Alexey Romanenko
>>> >> <aromanenko.dev@gmail.com <ma...@gmail.com>> wrote:
>>> >>
>>> >> Thank you JB for your work!
>>> >>
>>> >> I tested running simple streaming (/KafkaIO/) and batch (/TextIO /
>>> >> HDFS/) pipelines with SparkRunner on YARN cluster - it works fine.
>>> >>
>>> >> WBR,
>>> >> Alexey
>>> >>
>>> >>
>>> >>> On 8 Jun 2018, at 10:00, Etienne Chauchot <echauchot@apache.org
>>> >>> <ma...@apache.org>> wrote:
>>> >>>
>>> >>> I forgot to vote:
>>> >>> +1 (non binding).
>>> >>> What I tested:
>>> >>> - no functional or performance regression comparing to v2.4
>>> >>> - dependencies in the poms are ok
>>> >>>
>>> >>> Etienne
>>> >>>> Le vendredi 08 juin 2018 à 08:27 +0200, Romain Manni-Bucau a
>>> écrit :
>>> >>>> +1 (non-binding), mainstream usage is not broken by the pom
>>> >>>> changes and runtime has no known regression compared to the 2.4.0
>>> >>>>
>>> >>>> (side note: kudo to JB for this build tool change release, I know
>>> >>>> how it can hurt ;))
>>> >>>>
>>> >>>> Romain Manni-Bucau
>>> >>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>> >>>> <https://rmannibucau.metawerx.net/> | Old Blog
>>> >>>> <http://rmannibucau.wordpress.com/> | Github
>>> >>>> <https://github.com/rmannibucau> | LinkedIn
>>> >>>> <https://www.linkedin.com/in/rmannibucau> | Book
>>> >>>> <
>>> https://www.packtpub.com/application-development/java-ee-8-high-performance
>>> >
>>> >>>>
>>> >>>>
>>> >>>> Le jeu. 7 juin 2018 à 16:17, Jean-Baptiste Onofré
>>> >>>> <jb@nanthrax.net <ma...@nanthrax.net>> a écrit :
>>> >>>>> Thanks for the details Etienne !
>>> >>>>>
>>> >>>>> The good news is that the artifacts seem OK and the overall
>>> Nexmark
>>> >>>>> results are consistent with the 2.4.0 release ones.
>>> >>>>>
>>> >>>>> I'm starting a complete review using the beam-samples as well.
>>> >>>>>
>>> >>>>> Regards
>>> >>>>> JB
>>> >>>>>
>>> >>>>>> On 07/06/2018 16:14, Etienne Chauchot wrote:
>>> >>>>>> Hi,
>>> >>>>>> I've just run the nexmark queries on v2.5.0-RC1 tag
>>> >>>>>> What we can notice:
>>> >>>>>> - query 3 (exercises CoGroupByKey, state and timer) shows
>>> >>>>> different
>>> >>>>>> output with DR between batch and streaming and with the other
>>> >>>>> runners =>
>>> >>>>>> I compared with v2.4 there were still these differences but with
>>> >>>>>> different output size numbers
>>> >>>>>>
>>> >>>>>> - query 6 (exercises specialized combiner) shows different output
>>> >>>>>> between the runners => the correct output is 401. strange that
>>> >>>>> in batch
>>> >>>>>> mode some runners output les Sellers. I compared with v2.4
>>> >>>>> same output
>>> >>>>>>
>>> >>>>>> - response time of query 7 (exercices Max transform, fanout
>>> >>>>> and side
>>> >>>>>> input) is very slow on DR => I compared with v2.4 , comparable
>>> >>>>> execution
>>> >>>>>> times
>>> >>>>>>
>>> >>>>>> I'm not comparing q10 because it is a write to GCS so it is
>>> >>>>> very specific.
>>> >>>>>>
>>> >>>>>> => Basically no regression comparing to v2.4
>>> >>>>>>
>>> >>>>>> For the record here is the output (waiting for ongoing perfkit
>>> >>>>> integration):
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> 1. DR batch
>>> >>>>>>
>>> >>>>>> Performance:
>>> >>>>>>
>>> >>>>>>
>>> >>>>> Conf Runtime(sec) (Baseline) Events(/sec) (Baseline)
>>> Results (Baseline)
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0000 5,8 17283,1
>>> 100000
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0001 3,2 31104,2
>>> 92000
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0002 1,2 82918,7
>>> 351
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0003 2,2 46210,7
>>> 458
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0004 1,2 8503,4
>>> 40
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0005 4,0 25220,7
>>> 12
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0006 0,9 11148,3
>>> 401
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0007 13,2 7580,9
>>> 1
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0008 1,5 67340,1
>>> 6000
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0009 0,7 14025,2
>>> 298
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0010 12,8 7793,0
>>> 1
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0011 2,4 42319,1
>>> 1919
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0012 1,6 61462,8
>>> 1919
>>> >>>>>>
>>> >>>>>
>>> ==========================================================================================
>>> >>>>>>
>>> >>>>>> 2. DR streaming
>>> >>>>>>
>>> >>>>>> Performance:
>>> >>>>>>
>>> >>>>>>
>>> >>>>> Conf Runtime(sec) (Baseline) Events(/sec) (Baseline)
>>> Results (Baseline)
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0000 6,5 15285,8
>>> 100000
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0001 3,7 27397,3
>>> 92000
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0002 1,4 69108,5
>>> 351
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0003 3,2 31181,8
>>> 447
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0004 1,2 8361,2
>>> 40
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0005 5,3 18903,6
>>> 12
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0006 0,9 11111,1
>>> 401
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0007 82,5 1212,2
>>> 1
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0008 2,0 51072,5
>>> 6000
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0009 0,8 12903,2
>>> 298
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0010 49,5 2021,8
>>> 1
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0011 3,9 25667,4
>>> 1919
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0012 2,4 41067,8
>>> 1919
>>> >>>>>>
>>> >>>>>
>>> ==========================================================================================
>>> >>>>>>
>>> >>>>>> 3. Flink batch
>>> >>>>>> Performance:
>>> >>>>>>
>>> >>>>>>
>>> >>>>> Conf Runtime(sec) (Baseline) Events(/sec) (Baseline)
>>> Results (Baseline)
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0000 1,0 97656,3
>>> 100000
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0001 0,7 141643,1
>>> 92000
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0002 0,4 228310,5
>>> 351
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0003 1,6 64020,5
>>> 580
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0004 0,7 13831,3
>>> 40
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0005 1,4 72939,5
>>> 12
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0006 0,5 20491,8
>>> 103
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0007 1,3 74239,0
>>> 1
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0008 0,8 121506,7
>>> 6000
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0009 0,6 17953,3
>>> 298
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0010 1,3 74682,6
>>> 1
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0011 1,1 92936,8
>>> 1919
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0012 0,8 123001,2
>>> 1919
>>> >>>>>>
>>> >>>>>
>>> ==========================================================================================
>>> >>>>>>
>>> >>>>>> 4. Flink streaming
>>> >>>>>> Performance:
>>> >>>>>>
>>> >>>>>>
>>> >>>>> Conf Runtime(sec) (Baseline) Events(/sec) (Baseline)
>>> Results (Baseline)
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0000 5,4 18677,6
>>> 100000
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0001 2,8 35511,4
>>> 92000
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0002 1,8 54318,3
>>> 351
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0003 2,4 41614,6
>>> 580
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0004 1,0 10341,3
>>> 40
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0005 3,4 29568,3
>>> 12
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0006 0,7 13369,0
>>> 401
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0007 2,8 36192,5
>>> 1
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0008 1,8 54854,6
>>> 6000
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0009 0,7 13369,0
>>> 298
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0010 3,4 29841,8
>>> 2
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0011 5,0 19932,2
>>> 1919
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0012 2,6 38835,0
>>> 1919
>>> >>>>>>
>>> >>>>>
>>> ==========================================================================================
>>> >>>>>>
>>> >>>>>> 5. Spark batch
>>> >>>>>> Performance:
>>> >>>>>>
>>> >>>>>>
>>> >>>>> Conf Runtime(sec) (Baseline) Events(/sec) (Baseline)
>>> Results (Baseline)
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0000 1,5 65445,0
>>> 100000
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0001 1,3 79491,3
>>> 92000
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0002 0,9 112107,6
>>> 351
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0003 2,0 48804,3
>>> 580
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0004 1,2 8382,2
>>> 40
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0005 2,0 50838,8
>>> 12
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0006 1,0 9699,3
>>> 103
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0007 2,3 43308,8
>>> 1
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0008 2,1 46794,6
>>> 6000
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0009 1,1 8976,7
>>> 298
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0010 1,6 62111,8
>>> 1
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0011 2,1 46598,3
>>> 1919
>>> >>>>>>
>>> >>>>>>
>>> >>>>> 0012 2,3 43687,2
>>> 1919
>>> >>>>>>
>>> >>>>>
>>> ==========================================================================================
>>> >>>>>>
>>> >>>>>>> Le mercredi 06 juin 2018 à 10:50 +0200, Etienne Chauchot a écrit
>>> :
>>> >>>>>>> Thanks JB for all your work ! I believe doing the first
>>> >>>>> gradle release
>>> >>>>>>> must have been hard.
>>> >>>>>>> I'll run Nexmark on the release and keep you posted.
>>> >>>>>>>
>>> >>>>>>> Best
>>> >>>>>>> Etienne
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>> Le mercredi 06 juin 2018 à 10:44 +0200, Jean-Baptiste Onofré
>>> >>>>> a écrit :
>>> >>>>>>>> Hi everyone,
>>> >>>>>>>>
>>> >>>>>>>> Please review and vote on the release candidate #1 for the
>>> >>>>> version
>>> >>>>>>>> 2.5.0, as follows:
>>> >>>>>>>>
>>> >>>>>>>> [ ] +1, Approve the release
>>> >>>>>>>> [ ] -1, Do not approve the release (please provide specific
>>> >>>>> comments)
>>> >>>>>>>>
>>> >>>>>>>> NB: this is the first release using Gradle, so don't be too
>>> >>>>> harsh ;) A
>>> >>>>>>>> PR about the release guide will follow thanks to this release.
>>> >>>>>>>>
>>> >>>>>>>> The complete staging area is available for your review,
>>> >>>>> which includes:
>>> >>>>>>>> * JIRA release notes [1],
>>> >>>>>>>> * the official Apache source release to be deployed to
>>> >>>>> dist.apache.org <http://dist.apache.org/>
>>> >>>>>>>> [2], which is signed with the key with fingerprint C8282E76 [3],
>>> >>>>>>>> * all artifacts to be deployed to the Maven Central
>>> >>>>> Repository [4],
>>> >>>>>>>> * source code tag "v2.5.0-RC1" [5],
>>> >>>>>>>> * website pull request listing the release and publishing
>>> >>>>> the API
>>> >>>>>>>> reference manual [6].
>>> >>>>>>>> * Java artifacts were built with Gradle 4.7 (wrapper) and
>>> >>>>> OpenJDK/Oracle
>>> >>>>>>>> JDK 1.8.0_172 (Oracle Corporation 25.172-b11).
>>> >>>>>>>> * Python artifacts are deployed along with the source
>>> >>>>> release to the
>>> >>>>>>>> dist.apache.org <http://dist.apache.org/> [2].
>>> >>>>>>>>
>>> >>>>>>>> The vote will be open for at least 72 hours. It is adopted
>>> >>>>> by majority
>>> >>>>>>>> approval, with at least 3 PMC affirmative votes.
>>> >>>>>>>>
>>> >>>>>>>> Thanks,
>>> >>>>>>>> JB
>>> >>>>>>>>
>>> >>>>>>>> [1]
>>> >>>>>>>>
>>> >>>>>
>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12342847
>>> >>>>>>>> [2] https://dist.apache.org/repos/dist/dev/beam/2.5.0/
>>> >>>>>>>> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
>>> >>>>>>>> [4]
>>> >>>>>
>>> https://repository.apache.org/content/repositories/orgapachebeam-1041/
>>> >>>>>>>> [5] https://github.com/apache/beam/tree/v2.5.0-RC1
>>> >>>>>>>> [6] https://github.com/apache/beam-site/pull/463
>>> >>>>>>>>
>>> >>>>>
>>> >>
>>> >> --
>>> >> Got feedback? go/pabloem-feedback
>>> <https://goto.google.com/pabloem-feedback>
>>> >
>>> > --
>>> > Jean-Baptiste Onofré
>>> > jbonofre@apache.org
>>> > http://blog.nanthrax.net
>>> > Talend - http://www.talend.com
>>>
>> --
> Got feedback? go/pabloem-feedback
> <https://goto.google.com/pabloem-feedback>
>
Re: [VOTE] Apache Beam, version 2.5.0, release candidate #1
Posted by Ahmet Altay <al...@google.com>.
Thank you JB.
For the wheel artifacts, Boyuan was trying to get the instructions from
Robert and reproduce the artifacts. She can help you with this if you need.
Ahmet
On Mon, Jun 11, 2018 at 10:29 PM, Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> Hi,
>
> sorry, I missed wheel artifact. Something to add on the release guide ;)
>
> I will add it this morning, I think I know how to generate it ;)
>
> Regards
> JB
>
> On 12/06/2018 02:45, Pablo Estrada wrote:
> > Thanks everyone who has pitched in to validate the release!
> >
> > Boyuan Zhang and I have also run a few pipelines, and verified that they
> > work properly (see release validation spreadsheet[1]).
> >
> > We have also found that the Game Stats pipeline is failing in Python
> > Streaming Dataflow. I have filed BEAM-4534[2]. This is not a blocker,
> > since Python streaming is not yet fully supported.
> >
> > It seems that the uploaded artifacts look good.
> >
> > We have noticed that the Python artifacts are still missing Python wheel
> > files (compare [3] and [4]). JB, could you please add the wheel files?
> > Boyuan and I can try to help you prepare them / upload them if
> > necessary. Please let us know.
> >
> > Thanks again!
> > -P.
> >
> > [1] https://docs.google.com/spreadsheets/d/1qk-
> N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=152451807
> > [2] https://issues.apache.org/jira/browse/BEAM-4534
> > [3] https://dist.apache.org/repos/dist/dev/beam/2.4.0/
> > [4] https://dist.apache.org/repos/dist/dev/beam/2.5.0/
> >
> > On Mon, Jun 11, 2018 at 12:37 PM Alan Myrvold <amyrvold@google.com
> > <ma...@google.com>> wrote:
> >
> > +1 (non-binding)
> >
> > tested some of the quickstarts
> >
> > On Sun, Jun 10, 2018 at 1:39 AM Tim <timrobertson100@gmail.com
> > <ma...@gmail.com>> wrote:
> >
> > Tested by our team:
> > - mvn inclusion
> > - Avro, ES, Hadoop IF IO
> > - Pipelines run on Spark (Cloudera 5.12.0 YARN cluster)
> > - Reviewed release notes
> >
> > +1
> >
> > Thanks also to everyone who helped get over the gradle hurdle
> > and in particular to JB.
> >
> > Tim
> >
> > > On 9 Jun 2018, at 05:56, Jean-Baptiste Onofré <jb@nanthrax.net
> > <ma...@nanthrax.net>> wrote:
> > >
> > > No problem Pablo.
> > >
> > > The vote period is a minimum, it can be extended as requested
> > or if we
> > > don't have the minimum of 3 binding votes.
> > >
> > > Regards
> > > JB
> > >
> > >> On 09/06/2018 01:54, Pablo Estrada wrote:
> > >> Hello all,
> > >> I'd like to request an extension of the voting period until
> > Monday
> > >> evening (US time, so later in other geographical regions).
> > This is
> > >> because we were only now able to publish Dataflow Workers,
> > and have not
> > >> had the chance to run release validation tests on them. The
> > extension
> > >> will allow us to validate and vote by Monday.
> > >> Is this acceptable to the community?
> > >>
> > >> Best
> > >> -P.
> > >>
> > >> On Fri, Jun 8, 2018 at 6:20 AM Alexey Romanenko
> > >> <aromanenko.dev@gmail.com <ma...@gmail.com>
> > <mailto:aromanenko.dev@gmail.com
> > <ma...@gmail.com>>> wrote:
> > >>
> > >> Thank you JB for your work!
> > >>
> > >> I tested running simple streaming (/KafkaIO/) and batch
> > (/TextIO /
> > >> HDFS/) pipelines with SparkRunner on YARN cluster - it
> > works fine.
> > >>
> > >> WBR,
> > >> Alexey
> > >>
> > >>
> > >>> On 8 Jun 2018, at 10:00, Etienne Chauchot
> > <echauchot@apache.org <ma...@apache.org>
> > >>> <mailto:echauchot@apache.org
> > <ma...@apache.org>>> wrote:
> > >>>
> > >>> I forgot to vote:
> > >>> +1 (non binding).
> > >>> What I tested:
> > >>> - no functional or performance regression comparing to
> v2.4
> > >>> - dependencies in the poms are ok
> > >>>
> > >>> Etienne
> > >>>> Le vendredi 08 juin 2018 à 08:27 +0200, Romain
> > Manni-Bucau a écrit :
> > >>>> +1 (non-binding), mainstream usage is not broken by the
> pom
> > >>>> changes and runtime has no known regression compared to
> > the 2.4.0
> > >>>>
> > >>>> (side note: kudo to JB for this build tool change
> > release, I know
> > >>>> how it can hurt ;))
> > >>>>
> > >>>> Romain Manni-Bucau
> > >>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
> > >>>> <https://rmannibucau.metawerx.net/> | Old Blog
> > >>>> <http://rmannibucau.wordpress.com/> | Github
> > >>>> <https://github.com/rmannibucau> | LinkedIn
> > >>>> <https://www.linkedin.com/in/rmannibucau> | Book
> > >>>>
> > <https://www.packtpub.com/application-development/java-
> ee-8-high-performance>
> > >>>>
> > >>>>
> > >>>> Le jeu. 7 juin 2018 à 16:17, Jean-Baptiste Onofré
> > >>>> <jb@nanthrax.net <ma...@nanthrax.net>
> > <mailto:jb@nanthrax.net <ma...@nanthrax.net>>> a écrit :
> > >>>>> Thanks for the details Etienne !
> > >>>>>
> > >>>>> The good news is that the artifacts seem OK and the
> > overall Nexmark
> > >>>>> results are consistent with the 2.4.0 release ones.
> > >>>>>
> > >>>>> I'm starting a complete review using the beam-samples
> > as well.
> > >>>>>
> > >>>>> Regards
> > >>>>> JB
> > >>>>>
> > >>>>>> On 07/06/2018 16:14, Etienne Chauchot wrote:
> > >>>>>> Hi,
> > >>>>>> I've just run the nexmark queries on v2.5.0-RC1 tag
> > >>>>>> What we can notice:
> > >>>>>> - query 3 (exercises CoGroupByKey, state and timer) shows
> > >>>>> different
> > >>>>>> output with DR between batch and streaming and with the
> other
> > >>>>> runners =>
> > >>>>>> I compared with v2.4 there were still these differences
> > but with
> > >>>>>> different output size numbers
> > >>>>>>
> > >>>>>> - query 6 (exercises specialized combiner) shows
> > different output
> > >>>>>> between the runners => the correct output is 401. strange
> > that
> > >>>>> in batch
> > >>>>>> mode some runners output les Sellers. I compared with v2.4
> > >>>>> same output
> > >>>>>>
> > >>>>>> - response time of query 7 (exercices Max transform,
> fanout
> > >>>>> and side
> > >>>>>> input) is very slow on DR => I compared with v2.4 ,
> > comparable
> > >>>>> execution
> > >>>>>> times
> > >>>>>>
> > >>>>>> I'm not comparing q10 because it is a write to GCS so it
> is
> > >>>>> very specific.
> > >>>>>>
> > >>>>>> => Basically no regression comparing to v2.4
> > >>>>>>
> > >>>>>> For the record here is the output (waiting for ongoing
> > perfkit
> > >>>>> integration):
> > >>>>>>
> > >>>>>>
> > >>>>>> 1. DR batch
> > >>>>>>
> > >>>>>> Performance:
> > >>>>>>
> > >>>>>>
> > >>>>> Conf Runtime(sec) (Baseline) Events(/sec)
> > (Baseline) Results (Baseline)
> > >>>>>>
> > >>>>>>
> > >>>>> 0000 5,8 17283,1
> > 100000
> > >>>>>>
> > >>>>>>
> > >>>>> 0001 3,2 31104,2
> > 92000
> > >>>>>>
> > >>>>>>
> > >>>>> 0002 1,2 82918,7
> > 351
> > >>>>>>
> > >>>>>>
> > >>>>> 0003 2,2 46210,7
> > 458
> > >>>>>>
> > >>>>>>
> > >>>>> 0004 1,2 8503,4
> > 40
> > >>>>>>
> > >>>>>>
> > >>>>> 0005 4,0 25220,7
> > 12
> > >>>>>>
> > >>>>>>
> > >>>>> 0006 0,9 11148,3
> > 401
> > >>>>>>
> > >>>>>>
> > >>>>> 0007 13,2 7580,9
> > 1
> > >>>>>>
> > >>>>>>
> > >>>>> 0008 1,5 67340,1
> > 6000
> > >>>>>>
> > >>>>>>
> > >>>>> 0009 0,7 14025,2
> > 298
> > >>>>>>
> > >>>>>>
> > >>>>> 0010 12,8 7793,0
> > 1
> > >>>>>>
> > >>>>>>
> > >>>>> 0011 2,4 42319,1
> > 1919
> > >>>>>>
> > >>>>>>
> > >>>>> 0012 1,6 61462,8
> > 1919
> > >>>>>>
> > >>>>>
> > ============================================================
> ==============================
> > >>>>>>
> > >>>>>> 2. DR streaming
> > >>>>>>
> > >>>>>> Performance:
> > >>>>>>
> > >>>>>>
> > >>>>> Conf Runtime(sec) (Baseline) Events(/sec)
> > (Baseline) Results (Baseline)
> > >>>>>>
> > >>>>>>
> > >>>>> 0000 6,5 15285,8
> > 100000
> > >>>>>>
> > >>>>>>
> > >>>>> 0001 3,7 27397,3
> > 92000
> > >>>>>>
> > >>>>>>
> > >>>>> 0002 1,4 69108,5
> > 351
> > >>>>>>
> > >>>>>>
> > >>>>> 0003 3,2 31181,8
> > 447
> > >>>>>>
> > >>>>>>
> > >>>>> 0004 1,2 8361,2
> > 40
> > >>>>>>
> > >>>>>>
> > >>>>> 0005 5,3 18903,6
> > 12
> > >>>>>>
> > >>>>>>
> > >>>>> 0006 0,9 11111,1
> > 401
> > >>>>>>
> > >>>>>>
> > >>>>> 0007 82,5 1212,2
> > 1
> > >>>>>>
> > >>>>>>
> > >>>>> 0008 2,0 51072,5
> > 6000
> > >>>>>>
> > >>>>>>
> > >>>>> 0009 0,8 12903,2
> > 298
> > >>>>>>
> > >>>>>>
> > >>>>> 0010 49,5 2021,8
> > 1
> > >>>>>>
> > >>>>>>
> > >>>>> 0011 3,9 25667,4
> > 1919
> > >>>>>>
> > >>>>>>
> > >>>>> 0012 2,4 41067,8
> > 1919
> > >>>>>>
> > >>>>>
> > ============================================================
> ==============================
> > >>>>>>
> > >>>>>> 3. Flink batch
> > >>>>>> Performance:
> > >>>>>>
> > >>>>>>
> > >>>>> Conf Runtime(sec) (Baseline) Events(/sec)
> > (Baseline) Results (Baseline)
> > >>>>>>
> > >>>>>>
> > >>>>> 0000 1,0 97656,3
> > 100000
> > >>>>>>
> > >>>>>>
> > >>>>> 0001 0,7 141643,1
> > 92000
> > >>>>>>
> > >>>>>>
> > >>>>> 0002 0,4 228310,5
> > 351
> > >>>>>>
> > >>>>>>
> > >>>>> 0003 1,6 64020,5
> > 580
> > >>>>>>
> > >>>>>>
> > >>>>> 0004 0,7 13831,3
> > 40
> > >>>>>>
> > >>>>>>
> > >>>>> 0005 1,4 72939,5
> > 12
> > >>>>>>
> > >>>>>>
> > >>>>> 0006 0,5 20491,8
> > 103
> > >>>>>>
> > >>>>>>
> > >>>>> 0007 1,3 74239,0
> > 1
> > >>>>>>
> > >>>>>>
> > >>>>> 0008 0,8 121506,7
> > 6000
> > >>>>>>
> > >>>>>>
> > >>>>> 0009 0,6 17953,3
> > 298
> > >>>>>>
> > >>>>>>
> > >>>>> 0010 1,3 74682,6
> > 1
> > >>>>>>
> > >>>>>>
> > >>>>> 0011 1,1 92936,8
> > 1919
> > >>>>>>
> > >>>>>>
> > >>>>> 0012 0,8 123001,2
> > 1919
> > >>>>>>
> > >>>>>
> > ============================================================
> ==============================
> > >>>>>>
> > >>>>>> 4. Flink streaming
> > >>>>>> Performance:
> > >>>>>>
> > >>>>>>
> > >>>>> Conf Runtime(sec) (Baseline) Events(/sec)
> > (Baseline) Results (Baseline)
> > >>>>>>
> > >>>>>>
> > >>>>> 0000 5,4 18677,6
> > 100000
> > >>>>>>
> > >>>>>>
> > >>>>> 0001 2,8 35511,4
> > 92000
> > >>>>>>
> > >>>>>>
> > >>>>> 0002 1,8 54318,3
> > 351
> > >>>>>>
> > >>>>>>
> > >>>>> 0003 2,4 41614,6
> > 580
> > >>>>>>
> > >>>>>>
> > >>>>> 0004 1,0 10341,3
> > 40
> > >>>>>>
> > >>>>>>
> > >>>>> 0005 3,4 29568,3
> > 12
> > >>>>>>
> > >>>>>>
> > >>>>> 0006 0,7 13369,0
> > 401
> > >>>>>>
> > >>>>>>
> > >>>>> 0007 2,8 36192,5
> > 1
> > >>>>>>
> > >>>>>>
> > >>>>> 0008 1,8 54854,6
> > 6000
> > >>>>>>
> > >>>>>>
> > >>>>> 0009 0,7 13369,0
> > 298
> > >>>>>>
> > >>>>>>
> > >>>>> 0010 3,4 29841,8
> > 2
> > >>>>>>
> > >>>>>>
> > >>>>> 0011 5,0 19932,2
> > 1919
> > >>>>>>
> > >>>>>>
> > >>>>> 0012 2,6 38835,0
> > 1919
> > >>>>>>
> > >>>>>
> > ============================================================
> ==============================
> > >>>>>>
> > >>>>>> 5. Spark batch
> > >>>>>> Performance:
> > >>>>>>
> > >>>>>>
> > >>>>> Conf Runtime(sec) (Baseline) Events(/sec)
> > (Baseline) Results (Baseline)
> > >>>>>>
> > >>>>>>
> > >>>>> 0000 1,5 65445,0
> > 100000
> > >>>>>>
> > >>>>>>
> > >>>>> 0001 1,3 79491,3
> > 92000
> > >>>>>>
> > >>>>>>
> > >>>>> 0002 0,9 112107,6
> > 351
> > >>>>>>
> > >>>>>>
> > >>>>> 0003 2,0 48804,3
> > 580
> > >>>>>>
> > >>>>>>
> > >>>>> 0004 1,2 8382,2
> > 40
> > >>>>>>
> > >>>>>>
> > >>>>> 0005 2,0 50838,8
> > 12
> > >>>>>>
> > >>>>>>
> > >>>>> 0006 1,0 9699,3
> > 103
> > >>>>>>
> > >>>>>>
> > >>>>> 0007 2,3 43308,8
> > 1
> > >>>>>>
> > >>>>>>
> > >>>>> 0008 2,1 46794,6
> > 6000
> > >>>>>>
> > >>>>>>
> > >>>>> 0009 1,1 8976,7
> > 298
> > >>>>>>
> > >>>>>>
> > >>>>> 0010 1,6 62111,8
> > 1
> > >>>>>>
> > >>>>>>
> > >>>>> 0011 2,1 46598,3
> > 1919
> > >>>>>>
> > >>>>>>
> > >>>>> 0012 2,3 43687,2
> > 1919
> > >>>>>>
> > >>>>>
> > ============================================================
> ==============================
> > >>>>>>
> > >>>>>>> Le mercredi 06 juin 2018 à 10:50 +0200, Etienne Chauchot
> > a écrit :
> > >>>>>>> Thanks JB for all your work ! I believe doing the first
> > >>>>> gradle release
> > >>>>>>> must have been hard.
> > >>>>>>> I'll run Nexmark on the release and keep you posted.
> > >>>>>>>
> > >>>>>>> Best
> > >>>>>>> Etienne
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> Le mercredi 06 juin 2018 à 10:44 +0200, Jean-Baptiste
> Onofré
> > >>>>> a écrit :
> > >>>>>>>> Hi everyone,
> > >>>>>>>>
> > >>>>>>>> Please review and vote on the release candidate #1 for
> the
> > >>>>> version
> > >>>>>>>> 2.5.0, as follows:
> > >>>>>>>>
> > >>>>>>>> [ ] +1, Approve the release
> > >>>>>>>> [ ] -1, Do not approve the release (please provide
> specific
> > >>>>> comments)
> > >>>>>>>>
> > >>>>>>>> NB: this is the first release using Gradle, so don't be
> too
> > >>>>> harsh ;) A
> > >>>>>>>> PR about the release guide will follow thanks to this
> > release.
> > >>>>>>>>
> > >>>>>>>> The complete staging area is available for your review,
> > >>>>> which includes:
> > >>>>>>>> * JIRA release notes [1],
> > >>>>>>>> * the official Apache source release to be deployed to
> > >>>>> dist.apache.org <http://dist.apache.org>
> > <http://dist.apache.org/>
> > >>>>>>>> [2], which is signed with the key with fingerprint
> > C8282E76 [3],
> > >>>>>>>> * all artifacts to be deployed to the Maven Central
> > >>>>> Repository [4],
> > >>>>>>>> * source code tag "v2.5.0-RC1" [5],
> > >>>>>>>> * website pull request listing the release and
> publishing
> > >>>>> the API
> > >>>>>>>> reference manual [6].
> > >>>>>>>> * Java artifacts were built with Gradle 4.7 (wrapper)
> and
> > >>>>> OpenJDK/Oracle
> > >>>>>>>> JDK 1.8.0_172 (Oracle Corporation 25.172-b11).
> > >>>>>>>> * Python artifacts are deployed along with the source
> > >>>>> release to the
> > >>>>>>>> dist.apache.org <http://dist.apache.org>
> > <http://dist.apache.org/> [2].
> > >>>>>>>>
> > >>>>>>>> The vote will be open for at least 72 hours. It is
> adopted
> > >>>>> by majority
> > >>>>>>>> approval, with at least 3 PMC affirmative votes.
> > >>>>>>>>
> > >>>>>>>> Thanks,
> > >>>>>>>> JB
> > >>>>>>>>
> > >>>>>>>> [1]
> > >>>>>>>>
> > >>>>>
> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> projectId=12319527&version=12342847
> > >>>>>>>> [2] https://dist.apache.org/repos/dist/dev/beam/2.5.0/
> > >>>>>>>> [3] https://dist.apache.org/repos/
> dist/release/beam/KEYS
> > >>>>>>>> [4]
> > >>>>>
> > https://repository.apache.org/content/repositories/
> orgapachebeam-1041/
> > >>>>>>>> [5] https://github.com/apache/beam/tree/v2.5.0-RC1
> > >>>>>>>> [6] https://github.com/apache/beam-site/pull/463
> > >>>>>>>>
> > >>>>>
> > >>
> > >> --
> > >> Got feedback? go/pabloem-feedback
> > <https://goto.google.com/pabloem-feedback>
> > >
> > > --
> > > Jean-Baptiste Onofré
> > > jbonofre@apache.org <ma...@apache.org>
> > > http://blog.nanthrax.net
> > > Talend - http://www.talend.com
> >
> > --
> > Got feedback? go/pabloem-feedback
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
Re: [VOTE] Apache Beam, version 2.5.0, release candidate #1
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi,
sorry, I missed wheel artifact. Something to add on the release guide ;)
I will add it this morning, I think I know how to generate it ;)
Regards
JB
On 12/06/2018 02:45, Pablo Estrada wrote:
> Thanks everyone who has pitched in to validate the release!
>
> Boyuan Zhang and I have also run a few pipelines, and verified that they
> work properly (see release validation spreadsheet[1]).
>
> We have also found that the Game Stats pipeline is failing in Python
> Streaming Dataflow. I have filed BEAM-4534[2]. This is not a blocker,
> since Python streaming is not yet fully supported.
>
> It seems that the uploaded artifacts look good.
>
> We have noticed that the Python artifacts are still missing Python wheel
> files (compare [3] and [4]). JB, could you please add the wheel files?
> Boyuan and I can try to help you prepare them / upload them if
> necessary. Please let us know.
>
> Thanks again!
> -P.
>
> [1] https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=152451807
> [2] https://issues.apache.org/jira/browse/BEAM-4534
> [3] https://dist.apache.org/repos/dist/dev/beam/2.4.0/
> [4] https://dist.apache.org/repos/dist/dev/beam/2.5.0/
>
> On Mon, Jun 11, 2018 at 12:37 PM Alan Myrvold <amyrvold@google.com
> <ma...@google.com>> wrote:
>
> +1 (non-binding)
>
> tested some of the quickstarts
>
> On Sun, Jun 10, 2018 at 1:39 AM Tim <timrobertson100@gmail.com
> <ma...@gmail.com>> wrote:
>
> Tested by our team:
> - mvn inclusion
> - Avro, ES, Hadoop IF IO
> - Pipelines run on Spark (Cloudera 5.12.0 YARN cluster)
> - Reviewed release notes
>
> +1
>
> Thanks also to everyone who helped get over the gradle hurdle
> and in particular to JB.
>
> Tim
>
> > On 9 Jun 2018, at 05:56, Jean-Baptiste Onofré <jb@nanthrax.net
> <ma...@nanthrax.net>> wrote:
> >
> > No problem Pablo.
> >
> > The vote period is a minimum, it can be extended as requested
> or if we
> > don't have the minimum of 3 binding votes.
> >
> > Regards
> > JB
> >
> >> On 09/06/2018 01:54, Pablo Estrada wrote:
> >> Hello all,
> >> I'd like to request an extension of the voting period until
> Monday
> >> evening (US time, so later in other geographical regions).
> This is
> >> because we were only now able to publish Dataflow Workers,
> and have not
> >> had the chance to run release validation tests on them. The
> extension
> >> will allow us to validate and vote by Monday.
> >> Is this acceptable to the community?
> >>
> >> Best
> >> -P.
> >>
> >> On Fri, Jun 8, 2018 at 6:20 AM Alexey Romanenko
> >> <aromanenko.dev@gmail.com <ma...@gmail.com>
> <mailto:aromanenko.dev@gmail.com
> <ma...@gmail.com>>> wrote:
> >>
> >> Thank you JB for your work!
> >>
> >> I tested running simple streaming (/KafkaIO/) and batch
> (/TextIO /
> >> HDFS/) pipelines with SparkRunner on YARN cluster - it
> works fine.
> >>
> >> WBR,
> >> Alexey
> >>
> >>
> >>> On 8 Jun 2018, at 10:00, Etienne Chauchot
> <echauchot@apache.org <ma...@apache.org>
> >>> <mailto:echauchot@apache.org
> <ma...@apache.org>>> wrote:
> >>>
> >>> I forgot to vote:
> >>> +1 (non binding).
> >>> What I tested:
> >>> - no functional or performance regression comparing to v2.4
> >>> - dependencies in the poms are ok
> >>>
> >>> Etienne
> >>>> Le vendredi 08 juin 2018 à 08:27 +0200, Romain
> Manni-Bucau a écrit :
> >>>> +1 (non-binding), mainstream usage is not broken by the pom
> >>>> changes and runtime has no known regression compared to
> the 2.4.0
> >>>>
> >>>> (side note: kudo to JB for this build tool change
> release, I know
> >>>> how it can hurt ;))
> >>>>
> >>>> Romain Manni-Bucau
> >>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
> >>>> <https://rmannibucau.metawerx.net/> | Old Blog
> >>>> <http://rmannibucau.wordpress.com/> | Github
> >>>> <https://github.com/rmannibucau> | LinkedIn
> >>>> <https://www.linkedin.com/in/rmannibucau> | Book
> >>>>
> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
> >>>>
> >>>>
> >>>> Le jeu. 7 juin 2018 à 16:17, Jean-Baptiste Onofré
> >>>> <jb@nanthrax.net <ma...@nanthrax.net>
> <mailto:jb@nanthrax.net <ma...@nanthrax.net>>> a écrit :
> >>>>> Thanks for the details Etienne !
> >>>>>
> >>>>> The good news is that the artifacts seem OK and the
> overall Nexmark
> >>>>> results are consistent with the 2.4.0 release ones.
> >>>>>
> >>>>> I'm starting a complete review using the beam-samples
> as well.
> >>>>>
> >>>>> Regards
> >>>>> JB
> >>>>>
> >>>>>> On 07/06/2018 16:14, Etienne Chauchot wrote:
> >>>>>> Hi,
> >>>>>> I've just run the nexmark queries on v2.5.0-RC1 tag
> >>>>>> What we can notice:
> >>>>>> - query 3 (exercises CoGroupByKey, state and timer) shows
> >>>>> different
> >>>>>> output with DR between batch and streaming and with the other
> >>>>> runners =>
> >>>>>> I compared with v2.4 there were still these differences
> but with
> >>>>>> different output size numbers
> >>>>>>
> >>>>>> - query 6 (exercises specialized combiner) shows
> different output
> >>>>>> between the runners => the correct output is 401. strange
> that
> >>>>> in batch
> >>>>>> mode some runners output les Sellers. I compared with v2.4
> >>>>> same output
> >>>>>>
> >>>>>> - response time of query 7 (exercices Max transform, fanout
> >>>>> and side
> >>>>>> input) is very slow on DR => I compared with v2.4 ,
> comparable
> >>>>> execution
> >>>>>> times
> >>>>>>
> >>>>>> I'm not comparing q10 because it is a write to GCS so it is
> >>>>> very specific.
> >>>>>>
> >>>>>> => Basically no regression comparing to v2.4
> >>>>>>
> >>>>>> For the record here is the output (waiting for ongoing
> perfkit
> >>>>> integration):
> >>>>>>
> >>>>>>
> >>>>>> 1. DR batch
> >>>>>>
> >>>>>> Performance:
> >>>>>>
> >>>>>>
> >>>>> Conf Runtime(sec) (Baseline) Events(/sec)
> (Baseline) Results (Baseline)
> >>>>>>
> >>>>>>
> >>>>> 0000 5,8 17283,1
> 100000
> >>>>>>
> >>>>>>
> >>>>> 0001 3,2 31104,2
> 92000
> >>>>>>
> >>>>>>
> >>>>> 0002 1,2 82918,7
> 351
> >>>>>>
> >>>>>>
> >>>>> 0003 2,2 46210,7
> 458
> >>>>>>
> >>>>>>
> >>>>> 0004 1,2 8503,4
> 40
> >>>>>>
> >>>>>>
> >>>>> 0005 4,0 25220,7
> 12
> >>>>>>
> >>>>>>
> >>>>> 0006 0,9 11148,3
> 401
> >>>>>>
> >>>>>>
> >>>>> 0007 13,2 7580,9
> 1
> >>>>>>
> >>>>>>
> >>>>> 0008 1,5 67340,1
> 6000
> >>>>>>
> >>>>>>
> >>>>> 0009 0,7 14025,2
> 298
> >>>>>>
> >>>>>>
> >>>>> 0010 12,8 7793,0
> 1
> >>>>>>
> >>>>>>
> >>>>> 0011 2,4 42319,1
> 1919
> >>>>>>
> >>>>>>
> >>>>> 0012 1,6 61462,8
> 1919
> >>>>>>
> >>>>>
> ==========================================================================================
> >>>>>>
> >>>>>> 2. DR streaming
> >>>>>>
> >>>>>> Performance:
> >>>>>>
> >>>>>>
> >>>>> Conf Runtime(sec) (Baseline) Events(/sec)
> (Baseline) Results (Baseline)
> >>>>>>
> >>>>>>
> >>>>> 0000 6,5 15285,8
> 100000
> >>>>>>
> >>>>>>
> >>>>> 0001 3,7 27397,3
> 92000
> >>>>>>
> >>>>>>
> >>>>> 0002 1,4 69108,5
> 351
> >>>>>>
> >>>>>>
> >>>>> 0003 3,2 31181,8
> 447
> >>>>>>
> >>>>>>
> >>>>> 0004 1,2 8361,2
> 40
> >>>>>>
> >>>>>>
> >>>>> 0005 5,3 18903,6
> 12
> >>>>>>
> >>>>>>
> >>>>> 0006 0,9 11111,1
> 401
> >>>>>>
> >>>>>>
> >>>>> 0007 82,5 1212,2
> 1
> >>>>>>
> >>>>>>
> >>>>> 0008 2,0 51072,5
> 6000
> >>>>>>
> >>>>>>
> >>>>> 0009 0,8 12903,2
> 298
> >>>>>>
> >>>>>>
> >>>>> 0010 49,5 2021,8
> 1
> >>>>>>
> >>>>>>
> >>>>> 0011 3,9 25667,4
> 1919
> >>>>>>
> >>>>>>
> >>>>> 0012 2,4 41067,8
> 1919
> >>>>>>
> >>>>>
> ==========================================================================================
> >>>>>>
> >>>>>> 3. Flink batch
> >>>>>> Performance:
> >>>>>>
> >>>>>>
> >>>>> Conf Runtime(sec) (Baseline) Events(/sec)
> (Baseline) Results (Baseline)
> >>>>>>
> >>>>>>
> >>>>> 0000 1,0 97656,3
> 100000
> >>>>>>
> >>>>>>
> >>>>> 0001 0,7 141643,1
> 92000
> >>>>>>
> >>>>>>
> >>>>> 0002 0,4 228310,5
> 351
> >>>>>>
> >>>>>>
> >>>>> 0003 1,6 64020,5
> 580
> >>>>>>
> >>>>>>
> >>>>> 0004 0,7 13831,3
> 40
> >>>>>>
> >>>>>>
> >>>>> 0005 1,4 72939,5
> 12
> >>>>>>
> >>>>>>
> >>>>> 0006 0,5 20491,8
> 103
> >>>>>>
> >>>>>>
> >>>>> 0007 1,3 74239,0
> 1
> >>>>>>
> >>>>>>
> >>>>> 0008 0,8 121506,7
> 6000
> >>>>>>
> >>>>>>
> >>>>> 0009 0,6 17953,3
> 298
> >>>>>>
> >>>>>>
> >>>>> 0010 1,3 74682,6
> 1
> >>>>>>
> >>>>>>
> >>>>> 0011 1,1 92936,8
> 1919
> >>>>>>
> >>>>>>
> >>>>> 0012 0,8 123001,2
> 1919
> >>>>>>
> >>>>>
> ==========================================================================================
> >>>>>>
> >>>>>> 4. Flink streaming
> >>>>>> Performance:
> >>>>>>
> >>>>>>
> >>>>> Conf Runtime(sec) (Baseline) Events(/sec)
> (Baseline) Results (Baseline)
> >>>>>>
> >>>>>>
> >>>>> 0000 5,4 18677,6
> 100000
> >>>>>>
> >>>>>>
> >>>>> 0001 2,8 35511,4
> 92000
> >>>>>>
> >>>>>>
> >>>>> 0002 1,8 54318,3
> 351
> >>>>>>
> >>>>>>
> >>>>> 0003 2,4 41614,6
> 580
> >>>>>>
> >>>>>>
> >>>>> 0004 1,0 10341,3
> 40
> >>>>>>
> >>>>>>
> >>>>> 0005 3,4 29568,3
> 12
> >>>>>>
> >>>>>>
> >>>>> 0006 0,7 13369,0
> 401
> >>>>>>
> >>>>>>
> >>>>> 0007 2,8 36192,5
> 1
> >>>>>>
> >>>>>>
> >>>>> 0008 1,8 54854,6
> 6000
> >>>>>>
> >>>>>>
> >>>>> 0009 0,7 13369,0
> 298
> >>>>>>
> >>>>>>
> >>>>> 0010 3,4 29841,8
> 2
> >>>>>>
> >>>>>>
> >>>>> 0011 5,0 19932,2
> 1919
> >>>>>>
> >>>>>>
> >>>>> 0012 2,6 38835,0
> 1919
> >>>>>>
> >>>>>
> ==========================================================================================
> >>>>>>
> >>>>>> 5. Spark batch
> >>>>>> Performance:
> >>>>>>
> >>>>>>
> >>>>> Conf Runtime(sec) (Baseline) Events(/sec)
> (Baseline) Results (Baseline)
> >>>>>>
> >>>>>>
> >>>>> 0000 1,5 65445,0
> 100000
> >>>>>>
> >>>>>>
> >>>>> 0001 1,3 79491,3
> 92000
> >>>>>>
> >>>>>>
> >>>>> 0002 0,9 112107,6
> 351
> >>>>>>
> >>>>>>
> >>>>> 0003 2,0 48804,3
> 580
> >>>>>>
> >>>>>>
> >>>>> 0004 1,2 8382,2
> 40
> >>>>>>
> >>>>>>
> >>>>> 0005 2,0 50838,8
> 12
> >>>>>>
> >>>>>>
> >>>>> 0006 1,0 9699,3
> 103
> >>>>>>
> >>>>>>
> >>>>> 0007 2,3 43308,8
> 1
> >>>>>>
> >>>>>>
> >>>>> 0008 2,1 46794,6
> 6000
> >>>>>>
> >>>>>>
> >>>>> 0009 1,1 8976,7
> 298
> >>>>>>
> >>>>>>
> >>>>> 0010 1,6 62111,8
> 1
> >>>>>>
> >>>>>>
> >>>>> 0011 2,1 46598,3
> 1919
> >>>>>>
> >>>>>>
> >>>>> 0012 2,3 43687,2
> 1919
> >>>>>>
> >>>>>
> ==========================================================================================
> >>>>>>
> >>>>>>> Le mercredi 06 juin 2018 à 10:50 +0200, Etienne Chauchot
> a écrit :
> >>>>>>> Thanks JB for all your work ! I believe doing the first
> >>>>> gradle release
> >>>>>>> must have been hard.
> >>>>>>> I'll run Nexmark on the release and keep you posted.
> >>>>>>>
> >>>>>>> Best
> >>>>>>> Etienne
> >>>>>>>
> >>>>>>>
> >>>>>>> Le mercredi 06 juin 2018 à 10:44 +0200, Jean-Baptiste Onofré
> >>>>> a écrit :
> >>>>>>>> Hi everyone,
> >>>>>>>>
> >>>>>>>> Please review and vote on the release candidate #1 for the
> >>>>> version
> >>>>>>>> 2.5.0, as follows:
> >>>>>>>>
> >>>>>>>> [ ] +1, Approve the release
> >>>>>>>> [ ] -1, Do not approve the release (please provide specific
> >>>>> comments)
> >>>>>>>>
> >>>>>>>> NB: this is the first release using Gradle, so don't be too
> >>>>> harsh ;) A
> >>>>>>>> PR about the release guide will follow thanks to this
> release.
> >>>>>>>>
> >>>>>>>> The complete staging area is available for your review,
> >>>>> which includes:
> >>>>>>>> * JIRA release notes [1],
> >>>>>>>> * the official Apache source release to be deployed to
> >>>>> dist.apache.org <http://dist.apache.org>
> <http://dist.apache.org/>
> >>>>>>>> [2], which is signed with the key with fingerprint
> C8282E76 [3],
> >>>>>>>> * all artifacts to be deployed to the Maven Central
> >>>>> Repository [4],
> >>>>>>>> * source code tag "v2.5.0-RC1" [5],
> >>>>>>>> * website pull request listing the release and publishing
> >>>>> the API
> >>>>>>>> reference manual [6].
> >>>>>>>> * Java artifacts were built with Gradle 4.7 (wrapper) and
> >>>>> OpenJDK/Oracle
> >>>>>>>> JDK 1.8.0_172 (Oracle Corporation 25.172-b11).
> >>>>>>>> * Python artifacts are deployed along with the source
> >>>>> release to the
> >>>>>>>> dist.apache.org <http://dist.apache.org>
> <http://dist.apache.org/> [2].
> >>>>>>>>
> >>>>>>>> The vote will be open for at least 72 hours. It is adopted
> >>>>> by majority
> >>>>>>>> approval, with at least 3 PMC affirmative votes.
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> JB
> >>>>>>>>
> >>>>>>>> [1]
> >>>>>>>>
> >>>>>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12342847
> >>>>>>>> [2] https://dist.apache.org/repos/dist/dev/beam/2.5.0/
> >>>>>>>> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
> >>>>>>>> [4]
> >>>>>
> https://repository.apache.org/content/repositories/orgapachebeam-1041/
> >>>>>>>> [5] https://github.com/apache/beam/tree/v2.5.0-RC1
> >>>>>>>> [6] https://github.com/apache/beam-site/pull/463
> >>>>>>>>
> >>>>>
> >>
> >> --
> >> Got feedback? go/pabloem-feedback
> <https://goto.google.com/pabloem-feedback>
> >
> > --
> > Jean-Baptiste Onofré
> > jbonofre@apache.org <ma...@apache.org>
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
>
> --
> Got feedback? go/pabloem-feedback
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: [VOTE] Apache Beam, version 2.5.0, release candidate #1
Posted by Pablo Estrada <pa...@google.com>.
Thanks everyone who has pitched in to validate the release!
Boyuan Zhang and I have also run a few pipelines, and verified that they
work properly (see release validation spreadsheet[1]).
We have also found that the Game Stats pipeline is failing in Python
Streaming Dataflow. I have filed BEAM-4534[2]. This is not a blocker, since
Python streaming is not yet fully supported.
It seems that the uploaded artifacts look good.
We have noticed that the Python artifacts are still missing Python wheel
files (compare [3] and [4]). JB, could you please add the wheel files?
Boyuan and I can try to help you prepare them / upload them if necessary.
Please let us know.
Thanks again!
-P.
[1]
https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=152451807
[2] https://issues.apache.org/jira/browse/BEAM-4534
[3] https://dist.apache.org/repos/dist/dev/beam/2.4.0/
[4] https://dist.apache.org/repos/dist/dev/beam/2.5.0/
On Mon, Jun 11, 2018 at 12:37 PM Alan Myrvold <am...@google.com> wrote:
> +1 (non-binding)
>
> tested some of the quickstarts
>
> On Sun, Jun 10, 2018 at 1:39 AM Tim <ti...@gmail.com> wrote:
>
>> Tested by our team:
>> - mvn inclusion
>> - Avro, ES, Hadoop IF IO
>> - Pipelines run on Spark (Cloudera 5.12.0 YARN cluster)
>> - Reviewed release notes
>>
>> +1
>>
>> Thanks also to everyone who helped get over the gradle hurdle and in
>> particular to JB.
>>
>> Tim
>>
>> > On 9 Jun 2018, at 05:56, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>> >
>> > No problem Pablo.
>> >
>> > The vote period is a minimum, it can be extended as requested or if we
>> > don't have the minimum of 3 binding votes.
>> >
>> > Regards
>> > JB
>> >
>> >> On 09/06/2018 01:54, Pablo Estrada wrote:
>> >> Hello all,
>> >> I'd like to request an extension of the voting period until Monday
>> >> evening (US time, so later in other geographical regions). This is
>> >> because we were only now able to publish Dataflow Workers, and have not
>> >> had the chance to run release validation tests on them. The extension
>> >> will allow us to validate and vote by Monday.
>> >> Is this acceptable to the community?
>> >>
>> >> Best
>> >> -P.
>> >>
>> >> On Fri, Jun 8, 2018 at 6:20 AM Alexey Romanenko
>> >> <aromanenko.dev@gmail.com <ma...@gmail.com>> wrote:
>> >>
>> >> Thank you JB for your work!
>> >>
>> >> I tested running simple streaming (/KafkaIO/) and batch (/TextIO /
>> >> HDFS/) pipelines with SparkRunner on YARN cluster - it works fine.
>> >>
>> >> WBR,
>> >> Alexey
>> >>
>> >>
>> >>> On 8 Jun 2018, at 10:00, Etienne Chauchot <echauchot@apache.org
>> >>> <ma...@apache.org>> wrote:
>> >>>
>> >>> I forgot to vote:
>> >>> +1 (non binding).
>> >>> What I tested:
>> >>> - no functional or performance regression comparing to v2.4
>> >>> - dependencies in the poms are ok
>> >>>
>> >>> Etienne
>> >>>> Le vendredi 08 juin 2018 à 08:27 +0200, Romain Manni-Bucau a
>> écrit :
>> >>>> +1 (non-binding), mainstream usage is not broken by the pom
>> >>>> changes and runtime has no known regression compared to the 2.4.0
>> >>>>
>> >>>> (side note: kudo to JB for this build tool change release, I know
>> >>>> how it can hurt ;))
>> >>>>
>> >>>> Romain Manni-Bucau
>> >>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>> >>>> <https://rmannibucau.metawerx.net/> | Old Blog
>> >>>> <http://rmannibucau.wordpress.com/> | Github
>> >>>> <https://github.com/rmannibucau> | LinkedIn
>> >>>> <https://www.linkedin.com/in/rmannibucau> | Book
>> >>>> <
>> https://www.packtpub.com/application-development/java-ee-8-high-performance
>> >
>> >>>>
>> >>>>
>> >>>> Le jeu. 7 juin 2018 à 16:17, Jean-Baptiste Onofré
>> >>>> <jb@nanthrax.net <ma...@nanthrax.net>> a écrit :
>> >>>>> Thanks for the details Etienne !
>> >>>>>
>> >>>>> The good news is that the artifacts seem OK and the overall
>> Nexmark
>> >>>>> results are consistent with the 2.4.0 release ones.
>> >>>>>
>> >>>>> I'm starting a complete review using the beam-samples as well.
>> >>>>>
>> >>>>> Regards
>> >>>>> JB
>> >>>>>
>> >>>>>> On 07/06/2018 16:14, Etienne Chauchot wrote:
>> >>>>>> Hi,
>> >>>>>> I've just run the nexmark queries on v2.5.0-RC1 tag
>> >>>>>> What we can notice:
>> >>>>>> - query 3 (exercises CoGroupByKey, state and timer) shows
>> >>>>> different
>> >>>>>> output with DR between batch and streaming and with the other
>> >>>>> runners =>
>> >>>>>> I compared with v2.4 there were still these differences but with
>> >>>>>> different output size numbers
>> >>>>>>
>> >>>>>> - query 6 (exercises specialized combiner) shows different output
>> >>>>>> between the runners => the correct output is 401. strange that
>> >>>>> in batch
>> >>>>>> mode some runners output les Sellers. I compared with v2.4
>> >>>>> same output
>> >>>>>>
>> >>>>>> - response time of query 7 (exercices Max transform, fanout
>> >>>>> and side
>> >>>>>> input) is very slow on DR => I compared with v2.4 , comparable
>> >>>>> execution
>> >>>>>> times
>> >>>>>>
>> >>>>>> I'm not comparing q10 because it is a write to GCS so it is
>> >>>>> very specific.
>> >>>>>>
>> >>>>>> => Basically no regression comparing to v2.4
>> >>>>>>
>> >>>>>> For the record here is the output (waiting for ongoing perfkit
>> >>>>> integration):
>> >>>>>>
>> >>>>>>
>> >>>>>> 1. DR batch
>> >>>>>>
>> >>>>>> Performance:
>> >>>>>>
>> >>>>>>
>> >>>>> Conf Runtime(sec) (Baseline) Events(/sec) (Baseline)
>> Results (Baseline)
>> >>>>>>
>> >>>>>>
>> >>>>> 0000 5,8 17283,1
>> 100000
>> >>>>>>
>> >>>>>>
>> >>>>> 0001 3,2 31104,2
>> 92000
>> >>>>>>
>> >>>>>>
>> >>>>> 0002 1,2 82918,7
>> 351
>> >>>>>>
>> >>>>>>
>> >>>>> 0003 2,2 46210,7
>> 458
>> >>>>>>
>> >>>>>>
>> >>>>> 0004 1,2 8503,4
>> 40
>> >>>>>>
>> >>>>>>
>> >>>>> 0005 4,0 25220,7
>> 12
>> >>>>>>
>> >>>>>>
>> >>>>> 0006 0,9 11148,3
>> 401
>> >>>>>>
>> >>>>>>
>> >>>>> 0007 13,2 7580,9
>> 1
>> >>>>>>
>> >>>>>>
>> >>>>> 0008 1,5 67340,1
>> 6000
>> >>>>>>
>> >>>>>>
>> >>>>> 0009 0,7 14025,2
>> 298
>> >>>>>>
>> >>>>>>
>> >>>>> 0010 12,8 7793,0
>> 1
>> >>>>>>
>> >>>>>>
>> >>>>> 0011 2,4 42319,1
>> 1919
>> >>>>>>
>> >>>>>>
>> >>>>> 0012 1,6 61462,8
>> 1919
>> >>>>>>
>> >>>>>
>> ==========================================================================================
>> >>>>>>
>> >>>>>> 2. DR streaming
>> >>>>>>
>> >>>>>> Performance:
>> >>>>>>
>> >>>>>>
>> >>>>> Conf Runtime(sec) (Baseline) Events(/sec) (Baseline)
>> Results (Baseline)
>> >>>>>>
>> >>>>>>
>> >>>>> 0000 6,5 15285,8
>> 100000
>> >>>>>>
>> >>>>>>
>> >>>>> 0001 3,7 27397,3
>> 92000
>> >>>>>>
>> >>>>>>
>> >>>>> 0002 1,4 69108,5
>> 351
>> >>>>>>
>> >>>>>>
>> >>>>> 0003 3,2 31181,8
>> 447
>> >>>>>>
>> >>>>>>
>> >>>>> 0004 1,2 8361,2
>> 40
>> >>>>>>
>> >>>>>>
>> >>>>> 0005 5,3 18903,6
>> 12
>> >>>>>>
>> >>>>>>
>> >>>>> 0006 0,9 11111,1
>> 401
>> >>>>>>
>> >>>>>>
>> >>>>> 0007 82,5 1212,2
>> 1
>> >>>>>>
>> >>>>>>
>> >>>>> 0008 2,0 51072,5
>> 6000
>> >>>>>>
>> >>>>>>
>> >>>>> 0009 0,8 12903,2
>> 298
>> >>>>>>
>> >>>>>>
>> >>>>> 0010 49,5 2021,8
>> 1
>> >>>>>>
>> >>>>>>
>> >>>>> 0011 3,9 25667,4
>> 1919
>> >>>>>>
>> >>>>>>
>> >>>>> 0012 2,4 41067,8
>> 1919
>> >>>>>>
>> >>>>>
>> ==========================================================================================
>> >>>>>>
>> >>>>>> 3. Flink batch
>> >>>>>> Performance:
>> >>>>>>
>> >>>>>>
>> >>>>> Conf Runtime(sec) (Baseline) Events(/sec) (Baseline)
>> Results (Baseline)
>> >>>>>>
>> >>>>>>
>> >>>>> 0000 1,0 97656,3
>> 100000
>> >>>>>>
>> >>>>>>
>> >>>>> 0001 0,7 141643,1
>> 92000
>> >>>>>>
>> >>>>>>
>> >>>>> 0002 0,4 228310,5
>> 351
>> >>>>>>
>> >>>>>>
>> >>>>> 0003 1,6 64020,5
>> 580
>> >>>>>>
>> >>>>>>
>> >>>>> 0004 0,7 13831,3
>> 40
>> >>>>>>
>> >>>>>>
>> >>>>> 0005 1,4 72939,5
>> 12
>> >>>>>>
>> >>>>>>
>> >>>>> 0006 0,5 20491,8
>> 103
>> >>>>>>
>> >>>>>>
>> >>>>> 0007 1,3 74239,0
>> 1
>> >>>>>>
>> >>>>>>
>> >>>>> 0008 0,8 121506,7
>> 6000
>> >>>>>>
>> >>>>>>
>> >>>>> 0009 0,6 17953,3
>> 298
>> >>>>>>
>> >>>>>>
>> >>>>> 0010 1,3 74682,6
>> 1
>> >>>>>>
>> >>>>>>
>> >>>>> 0011 1,1 92936,8
>> 1919
>> >>>>>>
>> >>>>>>
>> >>>>> 0012 0,8 123001,2
>> 1919
>> >>>>>>
>> >>>>>
>> ==========================================================================================
>> >>>>>>
>> >>>>>> 4. Flink streaming
>> >>>>>> Performance:
>> >>>>>>
>> >>>>>>
>> >>>>> Conf Runtime(sec) (Baseline) Events(/sec) (Baseline)
>> Results (Baseline)
>> >>>>>>
>> >>>>>>
>> >>>>> 0000 5,4 18677,6
>> 100000
>> >>>>>>
>> >>>>>>
>> >>>>> 0001 2,8 35511,4
>> 92000
>> >>>>>>
>> >>>>>>
>> >>>>> 0002 1,8 54318,3
>> 351
>> >>>>>>
>> >>>>>>
>> >>>>> 0003 2,4 41614,6
>> 580
>> >>>>>>
>> >>>>>>
>> >>>>> 0004 1,0 10341,3
>> 40
>> >>>>>>
>> >>>>>>
>> >>>>> 0005 3,4 29568,3
>> 12
>> >>>>>>
>> >>>>>>
>> >>>>> 0006 0,7 13369,0
>> 401
>> >>>>>>
>> >>>>>>
>> >>>>> 0007 2,8 36192,5
>> 1
>> >>>>>>
>> >>>>>>
>> >>>>> 0008 1,8 54854,6
>> 6000
>> >>>>>>
>> >>>>>>
>> >>>>> 0009 0,7 13369,0
>> 298
>> >>>>>>
>> >>>>>>
>> >>>>> 0010 3,4 29841,8
>> 2
>> >>>>>>
>> >>>>>>
>> >>>>> 0011 5,0 19932,2
>> 1919
>> >>>>>>
>> >>>>>>
>> >>>>> 0012 2,6 38835,0
>> 1919
>> >>>>>>
>> >>>>>
>> ==========================================================================================
>> >>>>>>
>> >>>>>> 5. Spark batch
>> >>>>>> Performance:
>> >>>>>>
>> >>>>>>
>> >>>>> Conf Runtime(sec) (Baseline) Events(/sec) (Baseline)
>> Results (Baseline)
>> >>>>>>
>> >>>>>>
>> >>>>> 0000 1,5 65445,0
>> 100000
>> >>>>>>
>> >>>>>>
>> >>>>> 0001 1,3 79491,3
>> 92000
>> >>>>>>
>> >>>>>>
>> >>>>> 0002 0,9 112107,6
>> 351
>> >>>>>>
>> >>>>>>
>> >>>>> 0003 2,0 48804,3
>> 580
>> >>>>>>
>> >>>>>>
>> >>>>> 0004 1,2 8382,2
>> 40
>> >>>>>>
>> >>>>>>
>> >>>>> 0005 2,0 50838,8
>> 12
>> >>>>>>
>> >>>>>>
>> >>>>> 0006 1,0 9699,3
>> 103
>> >>>>>>
>> >>>>>>
>> >>>>> 0007 2,3 43308,8
>> 1
>> >>>>>>
>> >>>>>>
>> >>>>> 0008 2,1 46794,6
>> 6000
>> >>>>>>
>> >>>>>>
>> >>>>> 0009 1,1 8976,7
>> 298
>> >>>>>>
>> >>>>>>
>> >>>>> 0010 1,6 62111,8
>> 1
>> >>>>>>
>> >>>>>>
>> >>>>> 0011 2,1 46598,3
>> 1919
>> >>>>>>
>> >>>>>>
>> >>>>> 0012 2,3 43687,2
>> 1919
>> >>>>>>
>> >>>>>
>> ==========================================================================================
>> >>>>>>
>> >>>>>>> Le mercredi 06 juin 2018 à 10:50 +0200, Etienne Chauchot a écrit :
>> >>>>>>> Thanks JB for all your work ! I believe doing the first
>> >>>>> gradle release
>> >>>>>>> must have been hard.
>> >>>>>>> I'll run Nexmark on the release and keep you posted.
>> >>>>>>>
>> >>>>>>> Best
>> >>>>>>> Etienne
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> Le mercredi 06 juin 2018 à 10:44 +0200, Jean-Baptiste Onofré
>> >>>>> a écrit :
>> >>>>>>>> Hi everyone,
>> >>>>>>>>
>> >>>>>>>> Please review and vote on the release candidate #1 for the
>> >>>>> version
>> >>>>>>>> 2.5.0, as follows:
>> >>>>>>>>
>> >>>>>>>> [ ] +1, Approve the release
>> >>>>>>>> [ ] -1, Do not approve the release (please provide specific
>> >>>>> comments)
>> >>>>>>>>
>> >>>>>>>> NB: this is the first release using Gradle, so don't be too
>> >>>>> harsh ;) A
>> >>>>>>>> PR about the release guide will follow thanks to this release.
>> >>>>>>>>
>> >>>>>>>> The complete staging area is available for your review,
>> >>>>> which includes:
>> >>>>>>>> * JIRA release notes [1],
>> >>>>>>>> * the official Apache source release to be deployed to
>> >>>>> dist.apache.org <http://dist.apache.org/>
>> >>>>>>>> [2], which is signed with the key with fingerprint C8282E76 [3],
>> >>>>>>>> * all artifacts to be deployed to the Maven Central
>> >>>>> Repository [4],
>> >>>>>>>> * source code tag "v2.5.0-RC1" [5],
>> >>>>>>>> * website pull request listing the release and publishing
>> >>>>> the API
>> >>>>>>>> reference manual [6].
>> >>>>>>>> * Java artifacts were built with Gradle 4.7 (wrapper) and
>> >>>>> OpenJDK/Oracle
>> >>>>>>>> JDK 1.8.0_172 (Oracle Corporation 25.172-b11).
>> >>>>>>>> * Python artifacts are deployed along with the source
>> >>>>> release to the
>> >>>>>>>> dist.apache.org <http://dist.apache.org/> [2].
>> >>>>>>>>
>> >>>>>>>> The vote will be open for at least 72 hours. It is adopted
>> >>>>> by majority
>> >>>>>>>> approval, with at least 3 PMC affirmative votes.
>> >>>>>>>>
>> >>>>>>>> Thanks,
>> >>>>>>>> JB
>> >>>>>>>>
>> >>>>>>>> [1]
>> >>>>>>>>
>> >>>>>
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12342847
>> >>>>>>>> [2] https://dist.apache.org/repos/dist/dev/beam/2.5.0/
>> >>>>>>>> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
>> >>>>>>>> [4]
>> >>>>>
>> https://repository.apache.org/content/repositories/orgapachebeam-1041/
>> >>>>>>>> [5] https://github.com/apache/beam/tree/v2.5.0-RC1
>> >>>>>>>> [6] https://github.com/apache/beam-site/pull/463
>> >>>>>>>>
>> >>>>>
>> >>
>> >> --
>> >> Got feedback? go/pabloem-feedback
>> <https://goto.google.com/pabloem-feedback>
>> >
>> > --
>> > Jean-Baptiste Onofré
>> > jbonofre@apache.org
>> > http://blog.nanthrax.net
>> > Talend - http://www.talend.com
>>
> --
Got feedback? go/pabloem-feedback
Re: [VOTE] Apache Beam, version 2.5.0, release candidate #1
Posted by Alan Myrvold <am...@google.com>.
+1 (non-binding)
tested some of the quickstarts
On Sun, Jun 10, 2018 at 1:39 AM Tim <ti...@gmail.com> wrote:
> Tested by our team:
> - mvn inclusion
> - Avro, ES, Hadoop IF IO
> - Pipelines run on Spark (Cloudera 5.12.0 YARN cluster)
> - Reviewed release notes
>
> +1
>
> Thanks also to everyone who helped get over the gradle hurdle and in
> particular to JB.
>
> Tim
>
> > On 9 Jun 2018, at 05:56, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
> >
> > No problem Pablo.
> >
> > The vote period is a minimum, it can be extended as requested or if we
> > don't have the minimum of 3 binding votes.
> >
> > Regards
> > JB
> >
> >> On 09/06/2018 01:54, Pablo Estrada wrote:
> >> Hello all,
> >> I'd like to request an extension of the voting period until Monday
> >> evening (US time, so later in other geographical regions). This is
> >> because we were only now able to publish Dataflow Workers, and have not
> >> had the chance to run release validation tests on them. The extension
> >> will allow us to validate and vote by Monday.
> >> Is this acceptable to the community?
> >>
> >> Best
> >> -P.
> >>
> >> On Fri, Jun 8, 2018 at 6:20 AM Alexey Romanenko
> >> <aromanenko.dev@gmail.com <ma...@gmail.com>> wrote:
> >>
> >> Thank you JB for your work!
> >>
> >> I tested running simple streaming (/KafkaIO/) and batch (/TextIO /
> >> HDFS/) pipelines with SparkRunner on YARN cluster - it works fine.
> >>
> >> WBR,
> >> Alexey
> >>
> >>
> >>> On 8 Jun 2018, at 10:00, Etienne Chauchot <echauchot@apache.org
> >>> <ma...@apache.org>> wrote:
> >>>
> >>> I forgot to vote:
> >>> +1 (non binding).
> >>> What I tested:
> >>> - no functional or performance regression comparing to v2.4
> >>> - dependencies in the poms are ok
> >>>
> >>> Etienne
> >>>> Le vendredi 08 juin 2018 à 08:27 +0200, Romain Manni-Bucau a écrit
> :
> >>>> +1 (non-binding), mainstream usage is not broken by the pom
> >>>> changes and runtime has no known regression compared to the 2.4.0
> >>>>
> >>>> (side note: kudo to JB for this build tool change release, I know
> >>>> how it can hurt ;))
> >>>>
> >>>> Romain Manni-Bucau
> >>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
> >>>> <https://rmannibucau.metawerx.net/> | Old Blog
> >>>> <http://rmannibucau.wordpress.com/> | Github
> >>>> <https://github.com/rmannibucau> | LinkedIn
> >>>> <https://www.linkedin.com/in/rmannibucau> | Book
> >>>> <
> https://www.packtpub.com/application-development/java-ee-8-high-performance
> >
> >>>>
> >>>>
> >>>> Le jeu. 7 juin 2018 à 16:17, Jean-Baptiste Onofré
> >>>> <jb@nanthrax.net <ma...@nanthrax.net>> a écrit :
> >>>>> Thanks for the details Etienne !
> >>>>>
> >>>>> The good news is that the artifacts seem OK and the overall
> Nexmark
> >>>>> results are consistent with the 2.4.0 release ones.
> >>>>>
> >>>>> I'm starting a complete review using the beam-samples as well.
> >>>>>
> >>>>> Regards
> >>>>> JB
> >>>>>
> >>>>>> On 07/06/2018 16:14, Etienne Chauchot wrote:
> >>>>>> Hi,
> >>>>>> I've just run the nexmark queries on v2.5.0-RC1 tag
> >>>>>> What we can notice:
> >>>>>> - query 3 (exercises CoGroupByKey, state and timer) shows
> >>>>> different
> >>>>>> output with DR between batch and streaming and with the other
> >>>>> runners =>
> >>>>>> I compared with v2.4 there were still these differences but with
> >>>>>> different output size numbers
> >>>>>>
> >>>>>> - query 6 (exercises specialized combiner) shows different output
> >>>>>> between the runners => the correct output is 401. strange that
> >>>>> in batch
> >>>>>> mode some runners output les Sellers. I compared with v2.4
> >>>>> same output
> >>>>>>
> >>>>>> - response time of query 7 (exercices Max transform, fanout
> >>>>> and side
> >>>>>> input) is very slow on DR => I compared with v2.4 , comparable
> >>>>> execution
> >>>>>> times
> >>>>>>
> >>>>>> I'm not comparing q10 because it is a write to GCS so it is
> >>>>> very specific.
> >>>>>>
> >>>>>> => Basically no regression comparing to v2.4
> >>>>>>
> >>>>>> For the record here is the output (waiting for ongoing perfkit
> >>>>> integration):
> >>>>>>
> >>>>>>
> >>>>>> 1. DR batch
> >>>>>>
> >>>>>> Performance:
> >>>>>>
> >>>>>>
> >>>>> Conf Runtime(sec) (Baseline) Events(/sec) (Baseline)
> Results (Baseline)
> >>>>>>
> >>>>>>
> >>>>> 0000 5,8 17283,1
> 100000
> >>>>>>
> >>>>>>
> >>>>> 0001 3,2 31104,2
> 92000
> >>>>>>
> >>>>>>
> >>>>> 0002 1,2 82918,7
> 351
> >>>>>>
> >>>>>>
> >>>>> 0003 2,2 46210,7
> 458
> >>>>>>
> >>>>>>
> >>>>> 0004 1,2 8503,4
> 40
> >>>>>>
> >>>>>>
> >>>>> 0005 4,0 25220,7
> 12
> >>>>>>
> >>>>>>
> >>>>> 0006 0,9 11148,3
> 401
> >>>>>>
> >>>>>>
> >>>>> 0007 13,2 7580,9
> 1
> >>>>>>
> >>>>>>
> >>>>> 0008 1,5 67340,1
> 6000
> >>>>>>
> >>>>>>
> >>>>> 0009 0,7 14025,2
> 298
> >>>>>>
> >>>>>>
> >>>>> 0010 12,8 7793,0
> 1
> >>>>>>
> >>>>>>
> >>>>> 0011 2,4 42319,1
> 1919
> >>>>>>
> >>>>>>
> >>>>> 0012 1,6 61462,8
> 1919
> >>>>>>
> >>>>>
> ==========================================================================================
> >>>>>>
> >>>>>> 2. DR streaming
> >>>>>>
> >>>>>> Performance:
> >>>>>>
> >>>>>>
> >>>>> Conf Runtime(sec) (Baseline) Events(/sec) (Baseline)
> Results (Baseline)
> >>>>>>
> >>>>>>
> >>>>> 0000 6,5 15285,8
> 100000
> >>>>>>
> >>>>>>
> >>>>> 0001 3,7 27397,3
> 92000
> >>>>>>
> >>>>>>
> >>>>> 0002 1,4 69108,5
> 351
> >>>>>>
> >>>>>>
> >>>>> 0003 3,2 31181,8
> 447
> >>>>>>
> >>>>>>
> >>>>> 0004 1,2 8361,2
> 40
> >>>>>>
> >>>>>>
> >>>>> 0005 5,3 18903,6
> 12
> >>>>>>
> >>>>>>
> >>>>> 0006 0,9 11111,1
> 401
> >>>>>>
> >>>>>>
> >>>>> 0007 82,5 1212,2
> 1
> >>>>>>
> >>>>>>
> >>>>> 0008 2,0 51072,5
> 6000
> >>>>>>
> >>>>>>
> >>>>> 0009 0,8 12903,2
> 298
> >>>>>>
> >>>>>>
> >>>>> 0010 49,5 2021,8
> 1
> >>>>>>
> >>>>>>
> >>>>> 0011 3,9 25667,4
> 1919
> >>>>>>
> >>>>>>
> >>>>> 0012 2,4 41067,8
> 1919
> >>>>>>
> >>>>>
> ==========================================================================================
> >>>>>>
> >>>>>> 3. Flink batch
> >>>>>> Performance:
> >>>>>>
> >>>>>>
> >>>>> Conf Runtime(sec) (Baseline) Events(/sec) (Baseline)
> Results (Baseline)
> >>>>>>
> >>>>>>
> >>>>> 0000 1,0 97656,3
> 100000
> >>>>>>
> >>>>>>
> >>>>> 0001 0,7 141643,1
> 92000
> >>>>>>
> >>>>>>
> >>>>> 0002 0,4 228310,5
> 351
> >>>>>>
> >>>>>>
> >>>>> 0003 1,6 64020,5
> 580
> >>>>>>
> >>>>>>
> >>>>> 0004 0,7 13831,3
> 40
> >>>>>>
> >>>>>>
> >>>>> 0005 1,4 72939,5
> 12
> >>>>>>
> >>>>>>
> >>>>> 0006 0,5 20491,8
> 103
> >>>>>>
> >>>>>>
> >>>>> 0007 1,3 74239,0
> 1
> >>>>>>
> >>>>>>
> >>>>> 0008 0,8 121506,7
> 6000
> >>>>>>
> >>>>>>
> >>>>> 0009 0,6 17953,3
> 298
> >>>>>>
> >>>>>>
> >>>>> 0010 1,3 74682,6
> 1
> >>>>>>
> >>>>>>
> >>>>> 0011 1,1 92936,8
> 1919
> >>>>>>
> >>>>>>
> >>>>> 0012 0,8 123001,2
> 1919
> >>>>>>
> >>>>>
> ==========================================================================================
> >>>>>>
> >>>>>> 4. Flink streaming
> >>>>>> Performance:
> >>>>>>
> >>>>>>
> >>>>> Conf Runtime(sec) (Baseline) Events(/sec) (Baseline)
> Results (Baseline)
> >>>>>>
> >>>>>>
> >>>>> 0000 5,4 18677,6
> 100000
> >>>>>>
> >>>>>>
> >>>>> 0001 2,8 35511,4
> 92000
> >>>>>>
> >>>>>>
> >>>>> 0002 1,8 54318,3
> 351
> >>>>>>
> >>>>>>
> >>>>> 0003 2,4 41614,6
> 580
> >>>>>>
> >>>>>>
> >>>>> 0004 1,0 10341,3
> 40
> >>>>>>
> >>>>>>
> >>>>> 0005 3,4 29568,3
> 12
> >>>>>>
> >>>>>>
> >>>>> 0006 0,7 13369,0
> 401
> >>>>>>
> >>>>>>
> >>>>> 0007 2,8 36192,5
> 1
> >>>>>>
> >>>>>>
> >>>>> 0008 1,8 54854,6
> 6000
> >>>>>>
> >>>>>>
> >>>>> 0009 0,7 13369,0
> 298
> >>>>>>
> >>>>>>
> >>>>> 0010 3,4 29841,8
> 2
> >>>>>>
> >>>>>>
> >>>>> 0011 5,0 19932,2
> 1919
> >>>>>>
> >>>>>>
> >>>>> 0012 2,6 38835,0
> 1919
> >>>>>>
> >>>>>
> ==========================================================================================
> >>>>>>
> >>>>>> 5. Spark batch
> >>>>>> Performance:
> >>>>>>
> >>>>>>
> >>>>> Conf Runtime(sec) (Baseline) Events(/sec) (Baseline)
> Results (Baseline)
> >>>>>>
> >>>>>>
> >>>>> 0000 1,5 65445,0
> 100000
> >>>>>>
> >>>>>>
> >>>>> 0001 1,3 79491,3
> 92000
> >>>>>>
> >>>>>>
> >>>>> 0002 0,9 112107,6
> 351
> >>>>>>
> >>>>>>
> >>>>> 0003 2,0 48804,3
> 580
> >>>>>>
> >>>>>>
> >>>>> 0004 1,2 8382,2
> 40
> >>>>>>
> >>>>>>
> >>>>> 0005 2,0 50838,8
> 12
> >>>>>>
> >>>>>>
> >>>>> 0006 1,0 9699,3
> 103
> >>>>>>
> >>>>>>
> >>>>> 0007 2,3 43308,8
> 1
> >>>>>>
> >>>>>>
> >>>>> 0008 2,1 46794,6
> 6000
> >>>>>>
> >>>>>>
> >>>>> 0009 1,1 8976,7
> 298
> >>>>>>
> >>>>>>
> >>>>> 0010 1,6 62111,8
> 1
> >>>>>>
> >>>>>>
> >>>>> 0011 2,1 46598,3
> 1919
> >>>>>>
> >>>>>>
> >>>>> 0012 2,3 43687,2
> 1919
> >>>>>>
> >>>>>
> ==========================================================================================
> >>>>>>
> >>>>>>> Le mercredi 06 juin 2018 à 10:50 +0200, Etienne Chauchot a écrit :
> >>>>>>> Thanks JB for all your work ! I believe doing the first
> >>>>> gradle release
> >>>>>>> must have been hard.
> >>>>>>> I'll run Nexmark on the release and keep you posted.
> >>>>>>>
> >>>>>>> Best
> >>>>>>> Etienne
> >>>>>>>
> >>>>>>>
> >>>>>>> Le mercredi 06 juin 2018 à 10:44 +0200, Jean-Baptiste Onofré
> >>>>> a écrit :
> >>>>>>>> Hi everyone,
> >>>>>>>>
> >>>>>>>> Please review and vote on the release candidate #1 for the
> >>>>> version
> >>>>>>>> 2.5.0, as follows:
> >>>>>>>>
> >>>>>>>> [ ] +1, Approve the release
> >>>>>>>> [ ] -1, Do not approve the release (please provide specific
> >>>>> comments)
> >>>>>>>>
> >>>>>>>> NB: this is the first release using Gradle, so don't be too
> >>>>> harsh ;) A
> >>>>>>>> PR about the release guide will follow thanks to this release.
> >>>>>>>>
> >>>>>>>> The complete staging area is available for your review,
> >>>>> which includes:
> >>>>>>>> * JIRA release notes [1],
> >>>>>>>> * the official Apache source release to be deployed to
> >>>>> dist.apache.org <http://dist.apache.org/>
> >>>>>>>> [2], which is signed with the key with fingerprint C8282E76 [3],
> >>>>>>>> * all artifacts to be deployed to the Maven Central
> >>>>> Repository [4],
> >>>>>>>> * source code tag "v2.5.0-RC1" [5],
> >>>>>>>> * website pull request listing the release and publishing
> >>>>> the API
> >>>>>>>> reference manual [6].
> >>>>>>>> * Java artifacts were built with Gradle 4.7 (wrapper) and
> >>>>> OpenJDK/Oracle
> >>>>>>>> JDK 1.8.0_172 (Oracle Corporation 25.172-b11).
> >>>>>>>> * Python artifacts are deployed along with the source
> >>>>> release to the
> >>>>>>>> dist.apache.org <http://dist.apache.org/> [2].
> >>>>>>>>
> >>>>>>>> The vote will be open for at least 72 hours. It is adopted
> >>>>> by majority
> >>>>>>>> approval, with at least 3 PMC affirmative votes.
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> JB
> >>>>>>>>
> >>>>>>>> [1]
> >>>>>>>>
> >>>>>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12342847
> >>>>>>>> [2] https://dist.apache.org/repos/dist/dev/beam/2.5.0/
> >>>>>>>> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
> >>>>>>>> [4]
> >>>>>
> https://repository.apache.org/content/repositories/orgapachebeam-1041/
> >>>>>>>> [5] https://github.com/apache/beam/tree/v2.5.0-RC1
> >>>>>>>> [6] https://github.com/apache/beam-site/pull/463
> >>>>>>>>
> >>>>>
> >>
> >> --
> >> Got feedback? go/pabloem-feedback
> <https://goto.google.com/pabloem-feedback>
> >
> > --
> > Jean-Baptiste Onofré
> > jbonofre@apache.org
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
>
Re: [VOTE] Apache Beam, version 2.5.0, release candidate #1
Posted by Tim <ti...@gmail.com>.
Tested by our team:
- mvn inclusion
- Avro, ES, Hadoop IF IO
- Pipelines run on Spark (Cloudera 5.12.0 YARN cluster)
- Reviewed release notes
+1
Thanks also to everyone who helped get over the gradle hurdle and in particular to JB.
Tim
> On 9 Jun 2018, at 05:56, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>
> No problem Pablo.
>
> The vote period is a minimum, it can be extended as requested or if we
> don't have the minimum of 3 binding votes.
>
> Regards
> JB
>
>> On 09/06/2018 01:54, Pablo Estrada wrote:
>> Hello all,
>> I'd like to request an extension of the voting period until Monday
>> evening (US time, so later in other geographical regions). This is
>> because we were only now able to publish Dataflow Workers, and have not
>> had the chance to run release validation tests on them. The extension
>> will allow us to validate and vote by Monday.
>> Is this acceptable to the community?
>>
>> Best
>> -P.
>>
>> On Fri, Jun 8, 2018 at 6:20 AM Alexey Romanenko
>> <aromanenko.dev@gmail.com <ma...@gmail.com>> wrote:
>>
>> Thank you JB for your work!
>>
>> I tested running simple streaming (/KafkaIO/) and batch (/TextIO /
>> HDFS/) pipelines with SparkRunner on YARN cluster - it works fine.
>>
>> WBR,
>> Alexey
>>
>>
>>> On 8 Jun 2018, at 10:00, Etienne Chauchot <echauchot@apache.org
>>> <ma...@apache.org>> wrote:
>>>
>>> I forgot to vote:
>>> +1 (non binding).
>>> What I tested:
>>> - no functional or performance regression comparing to v2.4
>>> - dependencies in the poms are ok
>>>
>>> Etienne
>>>> Le vendredi 08 juin 2018 à 08:27 +0200, Romain Manni-Bucau a écrit :
>>>> +1 (non-binding), mainstream usage is not broken by the pom
>>>> changes and runtime has no known regression compared to the 2.4.0
>>>>
>>>> (side note: kudo to JB for this build tool change release, I know
>>>> how it can hurt ;))
>>>>
>>>> Romain Manni-Bucau
>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>>> <https://rmannibucau.metawerx.net/> | Old Blog
>>>> <http://rmannibucau.wordpress.com/> | Github
>>>> <https://github.com/rmannibucau> | LinkedIn
>>>> <https://www.linkedin.com/in/rmannibucau> | Book
>>>> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>>>
>>>>
>>>> Le jeu. 7 juin 2018 à 16:17, Jean-Baptiste Onofré
>>>> <jb@nanthrax.net <ma...@nanthrax.net>> a écrit :
>>>>> Thanks for the details Etienne !
>>>>>
>>>>> The good news is that the artifacts seem OK and the overall Nexmark
>>>>> results are consistent with the 2.4.0 release ones.
>>>>>
>>>>> I'm starting a complete review using the beam-samples as well.
>>>>>
>>>>> Regards
>>>>> JB
>>>>>
>>>>>> On 07/06/2018 16:14, Etienne Chauchot wrote:
>>>>>> Hi,
>>>>>> I've just run the nexmark queries on v2.5.0-RC1 tag
>>>>>> What we can notice:
>>>>>> - query 3 (exercises CoGroupByKey, state and timer) shows
>>>>> different
>>>>>> output with DR between batch and streaming and with the other
>>>>> runners =>
>>>>>> I compared with v2.4 there were still these differences but with
>>>>>> different output size numbers
>>>>>>
>>>>>> - query 6 (exercises specialized combiner) shows different output
>>>>>> between the runners => the correct output is 401. strange that
>>>>> in batch
>>>>>> mode some runners output les Sellers. I compared with v2.4
>>>>> same output
>>>>>>
>>>>>> - response time of query 7 (exercices Max transform, fanout
>>>>> and side
>>>>>> input) is very slow on DR => I compared with v2.4 , comparable
>>>>> execution
>>>>>> times
>>>>>>
>>>>>> I'm not comparing q10 because it is a write to GCS so it is
>>>>> very specific.
>>>>>>
>>>>>> => Basically no regression comparing to v2.4
>>>>>>
>>>>>> For the record here is the output (waiting for ongoing perfkit
>>>>> integration):
>>>>>>
>>>>>>
>>>>>> 1. DR batch
>>>>>>
>>>>>> Performance:
>>>>>>
>>>>>>
>>>>> Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results (Baseline)
>>>>>>
>>>>>>
>>>>> 0000 5,8 17283,1 100000
>>>>>>
>>>>>>
>>>>> 0001 3,2 31104,2 92000
>>>>>>
>>>>>>
>>>>> 0002 1,2 82918,7 351
>>>>>>
>>>>>>
>>>>> 0003 2,2 46210,7 458
>>>>>>
>>>>>>
>>>>> 0004 1,2 8503,4 40
>>>>>>
>>>>>>
>>>>> 0005 4,0 25220,7 12
>>>>>>
>>>>>>
>>>>> 0006 0,9 11148,3 401
>>>>>>
>>>>>>
>>>>> 0007 13,2 7580,9 1
>>>>>>
>>>>>>
>>>>> 0008 1,5 67340,1 6000
>>>>>>
>>>>>>
>>>>> 0009 0,7 14025,2 298
>>>>>>
>>>>>>
>>>>> 0010 12,8 7793,0 1
>>>>>>
>>>>>>
>>>>> 0011 2,4 42319,1 1919
>>>>>>
>>>>>>
>>>>> 0012 1,6 61462,8 1919
>>>>>>
>>>>> ==========================================================================================
>>>>>>
>>>>>> 2. DR streaming
>>>>>>
>>>>>> Performance:
>>>>>>
>>>>>>
>>>>> Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results (Baseline)
>>>>>>
>>>>>>
>>>>> 0000 6,5 15285,8 100000
>>>>>>
>>>>>>
>>>>> 0001 3,7 27397,3 92000
>>>>>>
>>>>>>
>>>>> 0002 1,4 69108,5 351
>>>>>>
>>>>>>
>>>>> 0003 3,2 31181,8 447
>>>>>>
>>>>>>
>>>>> 0004 1,2 8361,2 40
>>>>>>
>>>>>>
>>>>> 0005 5,3 18903,6 12
>>>>>>
>>>>>>
>>>>> 0006 0,9 11111,1 401
>>>>>>
>>>>>>
>>>>> 0007 82,5 1212,2 1
>>>>>>
>>>>>>
>>>>> 0008 2,0 51072,5 6000
>>>>>>
>>>>>>
>>>>> 0009 0,8 12903,2 298
>>>>>>
>>>>>>
>>>>> 0010 49,5 2021,8 1
>>>>>>
>>>>>>
>>>>> 0011 3,9 25667,4 1919
>>>>>>
>>>>>>
>>>>> 0012 2,4 41067,8 1919
>>>>>>
>>>>> ==========================================================================================
>>>>>>
>>>>>> 3. Flink batch
>>>>>> Performance:
>>>>>>
>>>>>>
>>>>> Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results (Baseline)
>>>>>>
>>>>>>
>>>>> 0000 1,0 97656,3 100000
>>>>>>
>>>>>>
>>>>> 0001 0,7 141643,1 92000
>>>>>>
>>>>>>
>>>>> 0002 0,4 228310,5 351
>>>>>>
>>>>>>
>>>>> 0003 1,6 64020,5 580
>>>>>>
>>>>>>
>>>>> 0004 0,7 13831,3 40
>>>>>>
>>>>>>
>>>>> 0005 1,4 72939,5 12
>>>>>>
>>>>>>
>>>>> 0006 0,5 20491,8 103
>>>>>>
>>>>>>
>>>>> 0007 1,3 74239,0 1
>>>>>>
>>>>>>
>>>>> 0008 0,8 121506,7 6000
>>>>>>
>>>>>>
>>>>> 0009 0,6 17953,3 298
>>>>>>
>>>>>>
>>>>> 0010 1,3 74682,6 1
>>>>>>
>>>>>>
>>>>> 0011 1,1 92936,8 1919
>>>>>>
>>>>>>
>>>>> 0012 0,8 123001,2 1919
>>>>>>
>>>>> ==========================================================================================
>>>>>>
>>>>>> 4. Flink streaming
>>>>>> Performance:
>>>>>>
>>>>>>
>>>>> Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results (Baseline)
>>>>>>
>>>>>>
>>>>> 0000 5,4 18677,6 100000
>>>>>>
>>>>>>
>>>>> 0001 2,8 35511,4 92000
>>>>>>
>>>>>>
>>>>> 0002 1,8 54318,3 351
>>>>>>
>>>>>>
>>>>> 0003 2,4 41614,6 580
>>>>>>
>>>>>>
>>>>> 0004 1,0 10341,3 40
>>>>>>
>>>>>>
>>>>> 0005 3,4 29568,3 12
>>>>>>
>>>>>>
>>>>> 0006 0,7 13369,0 401
>>>>>>
>>>>>>
>>>>> 0007 2,8 36192,5 1
>>>>>>
>>>>>>
>>>>> 0008 1,8 54854,6 6000
>>>>>>
>>>>>>
>>>>> 0009 0,7 13369,0 298
>>>>>>
>>>>>>
>>>>> 0010 3,4 29841,8 2
>>>>>>
>>>>>>
>>>>> 0011 5,0 19932,2 1919
>>>>>>
>>>>>>
>>>>> 0012 2,6 38835,0 1919
>>>>>>
>>>>> ==========================================================================================
>>>>>>
>>>>>> 5. Spark batch
>>>>>> Performance:
>>>>>>
>>>>>>
>>>>> Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results (Baseline)
>>>>>>
>>>>>>
>>>>> 0000 1,5 65445,0 100000
>>>>>>
>>>>>>
>>>>> 0001 1,3 79491,3 92000
>>>>>>
>>>>>>
>>>>> 0002 0,9 112107,6 351
>>>>>>
>>>>>>
>>>>> 0003 2,0 48804,3 580
>>>>>>
>>>>>>
>>>>> 0004 1,2 8382,2 40
>>>>>>
>>>>>>
>>>>> 0005 2,0 50838,8 12
>>>>>>
>>>>>>
>>>>> 0006 1,0 9699,3 103
>>>>>>
>>>>>>
>>>>> 0007 2,3 43308,8 1
>>>>>>
>>>>>>
>>>>> 0008 2,1 46794,6 6000
>>>>>>
>>>>>>
>>>>> 0009 1,1 8976,7 298
>>>>>>
>>>>>>
>>>>> 0010 1,6 62111,8 1
>>>>>>
>>>>>>
>>>>> 0011 2,1 46598,3 1919
>>>>>>
>>>>>>
>>>>> 0012 2,3 43687,2 1919
>>>>>>
>>>>> ==========================================================================================
>>>>>>
>>>>>>> Le mercredi 06 juin 2018 à 10:50 +0200, Etienne Chauchot a écrit :
>>>>>>> Thanks JB for all your work ! I believe doing the first
>>>>> gradle release
>>>>>>> must have been hard.
>>>>>>> I'll run Nexmark on the release and keep you posted.
>>>>>>>
>>>>>>> Best
>>>>>>> Etienne
>>>>>>>
>>>>>>>
>>>>>>> Le mercredi 06 juin 2018 à 10:44 +0200, Jean-Baptiste Onofré
>>>>> a écrit :
>>>>>>>> Hi everyone,
>>>>>>>>
>>>>>>>> Please review and vote on the release candidate #1 for the
>>>>> version
>>>>>>>> 2.5.0, as follows:
>>>>>>>>
>>>>>>>> [ ] +1, Approve the release
>>>>>>>> [ ] -1, Do not approve the release (please provide specific
>>>>> comments)
>>>>>>>>
>>>>>>>> NB: this is the first release using Gradle, so don't be too
>>>>> harsh ;) A
>>>>>>>> PR about the release guide will follow thanks to this release.
>>>>>>>>
>>>>>>>> The complete staging area is available for your review,
>>>>> which includes:
>>>>>>>> * JIRA release notes [1],
>>>>>>>> * the official Apache source release to be deployed to
>>>>> dist.apache.org <http://dist.apache.org/>
>>>>>>>> [2], which is signed with the key with fingerprint C8282E76 [3],
>>>>>>>> * all artifacts to be deployed to the Maven Central
>>>>> Repository [4],
>>>>>>>> * source code tag "v2.5.0-RC1" [5],
>>>>>>>> * website pull request listing the release and publishing
>>>>> the API
>>>>>>>> reference manual [6].
>>>>>>>> * Java artifacts were built with Gradle 4.7 (wrapper) and
>>>>> OpenJDK/Oracle
>>>>>>>> JDK 1.8.0_172 (Oracle Corporation 25.172-b11).
>>>>>>>> * Python artifacts are deployed along with the source
>>>>> release to the
>>>>>>>> dist.apache.org <http://dist.apache.org/> [2].
>>>>>>>>
>>>>>>>> The vote will be open for at least 72 hours. It is adopted
>>>>> by majority
>>>>>>>> approval, with at least 3 PMC affirmative votes.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> JB
>>>>>>>>
>>>>>>>> [1]
>>>>>>>>
>>>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12342847
>>>>>>>> [2] https://dist.apache.org/repos/dist/dev/beam/2.5.0/
>>>>>>>> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
>>>>>>>> [4]
>>>>> https://repository.apache.org/content/repositories/orgapachebeam-1041/
>>>>>>>> [5] https://github.com/apache/beam/tree/v2.5.0-RC1
>>>>>>>> [6] https://github.com/apache/beam-site/pull/463
>>>>>>>>
>>>>>
>>
>> --
>> Got feedback? go/pabloem-feedback
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
Re: [VOTE] Apache Beam, version 2.5.0, release candidate #1
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
No problem Pablo.
The vote period is a minimum, it can be extended as requested or if we
don't have the minimum of 3 binding votes.
Regards
JB
On 09/06/2018 01:54, Pablo Estrada wrote:
> Hello all,
> I'd like to request an extension of the voting period until Monday
> evening (US time, so later in other geographical regions). This is
> because we were only now able to publish Dataflow Workers, and have not
> had the chance to run release validation tests on them. The extension
> will allow us to validate and vote by Monday.
> Is this acceptable to the community?
>
> Best
> -P.
>
> On Fri, Jun 8, 2018 at 6:20 AM Alexey Romanenko
> <aromanenko.dev@gmail.com <ma...@gmail.com>> wrote:
>
> Thank you JB for your work!
>
> I tested running simple streaming (/KafkaIO/) and batch (/TextIO /
> HDFS/) pipelines with SparkRunner on YARN cluster - it works fine.
>
> WBR,
> Alexey
>
>
>> On 8 Jun 2018, at 10:00, Etienne Chauchot <echauchot@apache.org
>> <ma...@apache.org>> wrote:
>>
>> I forgot to vote:
>> +1 (non binding).
>> What I tested:
>> - no functional or performance regression comparing to v2.4
>> - dependencies in the poms are ok
>>
>> Etienne
>> Le vendredi 08 juin 2018 à 08:27 +0200, Romain Manni-Bucau a écrit :
>>> +1 (non-binding), mainstream usage is not broken by the pom
>>> changes and runtime has no known regression compared to the 2.4.0
>>>
>>> (side note: kudo to JB for this build tool change release, I know
>>> how it can hurt ;))
>>>
>>> Romain Manni-Bucau
>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>> <https://rmannibucau.metawerx.net/> | Old Blog
>>> <http://rmannibucau.wordpress.com/> | Github
>>> <https://github.com/rmannibucau> | LinkedIn
>>> <https://www.linkedin.com/in/rmannibucau> | Book
>>> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>>
>>>
>>> Le jeu. 7 juin 2018 à 16:17, Jean-Baptiste Onofré
>>> <jb@nanthrax.net <ma...@nanthrax.net>> a écrit :
>>>> Thanks for the details Etienne !
>>>>
>>>> The good news is that the artifacts seem OK and the overall Nexmark
>>>> results are consistent with the 2.4.0 release ones.
>>>>
>>>> I'm starting a complete review using the beam-samples as well.
>>>>
>>>> Regards
>>>> JB
>>>>
>>>> On 07/06/2018 16:14, Etienne Chauchot wrote:
>>>> > Hi,
>>>> > I've just run the nexmark queries on v2.5.0-RC1 tag
>>>> > What we can notice:
>>>> > - query 3 (exercises CoGroupByKey, state and timer) shows
>>>> different
>>>> > output with DR between batch and streaming and with the other
>>>> runners =>
>>>> > I compared with v2.4 there were still these differences but with
>>>> > different output size numbers
>>>> >
>>>> > - query 6 (exercises specialized combiner) shows different output
>>>> > between the runners => the correct output is 401. strange that
>>>> in batch
>>>> > mode some runners output les Sellers. I compared with v2.4
>>>> same output
>>>> >
>>>> > - response time of query 7 (exercices Max transform, fanout
>>>> and side
>>>> > input) is very slow on DR => I compared with v2.4 , comparable
>>>> execution
>>>> > times
>>>> >
>>>> > I'm not comparing q10 because it is a write to GCS so it is
>>>> very specific.
>>>> >
>>>> > => Basically no regression comparing to v2.4
>>>> >
>>>> > For the record here is the output (waiting for ongoing perfkit
>>>> integration):
>>>> >
>>>> >
>>>> > 1. DR batch
>>>> >
>>>> > Performance:
>>>> >
>>>> >
>>>> Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results (Baseline)
>>>> >
>>>> >
>>>> 0000 5,8 17283,1 100000
>>>> >
>>>> >
>>>> 0001 3,2 31104,2 92000
>>>> >
>>>> >
>>>> 0002 1,2 82918,7 351
>>>> >
>>>> >
>>>> 0003 2,2 46210,7 458
>>>> >
>>>> >
>>>> 0004 1,2 8503,4 40
>>>> >
>>>> >
>>>> 0005 4,0 25220,7 12
>>>> >
>>>> >
>>>> 0006 0,9 11148,3 401
>>>> >
>>>> >
>>>> 0007 13,2 7580,9 1
>>>> >
>>>> >
>>>> 0008 1,5 67340,1 6000
>>>> >
>>>> >
>>>> 0009 0,7 14025,2 298
>>>> >
>>>> >
>>>> 0010 12,8 7793,0 1
>>>> >
>>>> >
>>>> 0011 2,4 42319,1 1919
>>>> >
>>>> >
>>>> 0012 1,6 61462,8 1919
>>>> >
>>>> ==========================================================================================
>>>> >
>>>> > 2. DR streaming
>>>> >
>>>> > Performance:
>>>> >
>>>> >
>>>> Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results (Baseline)
>>>> >
>>>> >
>>>> 0000 6,5 15285,8 100000
>>>> >
>>>> >
>>>> 0001 3,7 27397,3 92000
>>>> >
>>>> >
>>>> 0002 1,4 69108,5 351
>>>> >
>>>> >
>>>> 0003 3,2 31181,8 447
>>>> >
>>>> >
>>>> 0004 1,2 8361,2 40
>>>> >
>>>> >
>>>> 0005 5,3 18903,6 12
>>>> >
>>>> >
>>>> 0006 0,9 11111,1 401
>>>> >
>>>> >
>>>> 0007 82,5 1212,2 1
>>>> >
>>>> >
>>>> 0008 2,0 51072,5 6000
>>>> >
>>>> >
>>>> 0009 0,8 12903,2 298
>>>> >
>>>> >
>>>> 0010 49,5 2021,8 1
>>>> >
>>>> >
>>>> 0011 3,9 25667,4 1919
>>>> >
>>>> >
>>>> 0012 2,4 41067,8 1919
>>>> >
>>>> ==========================================================================================
>>>> >
>>>> > 3. Flink batch
>>>> > Performance:
>>>> >
>>>> >
>>>> Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results (Baseline)
>>>> >
>>>> >
>>>> 0000 1,0 97656,3 100000
>>>> >
>>>> >
>>>> 0001 0,7 141643,1 92000
>>>> >
>>>> >
>>>> 0002 0,4 228310,5 351
>>>> >
>>>> >
>>>> 0003 1,6 64020,5 580
>>>> >
>>>> >
>>>> 0004 0,7 13831,3 40
>>>> >
>>>> >
>>>> 0005 1,4 72939,5 12
>>>> >
>>>> >
>>>> 0006 0,5 20491,8 103
>>>> >
>>>> >
>>>> 0007 1,3 74239,0 1
>>>> >
>>>> >
>>>> 0008 0,8 121506,7 6000
>>>> >
>>>> >
>>>> 0009 0,6 17953,3 298
>>>> >
>>>> >
>>>> 0010 1,3 74682,6 1
>>>> >
>>>> >
>>>> 0011 1,1 92936,8 1919
>>>> >
>>>> >
>>>> 0012 0,8 123001,2 1919
>>>> >
>>>> ==========================================================================================
>>>> >
>>>> > 4. Flink streaming
>>>> > Performance:
>>>> >
>>>> >
>>>> Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results (Baseline)
>>>> >
>>>> >
>>>> 0000 5,4 18677,6 100000
>>>> >
>>>> >
>>>> 0001 2,8 35511,4 92000
>>>> >
>>>> >
>>>> 0002 1,8 54318,3 351
>>>> >
>>>> >
>>>> 0003 2,4 41614,6 580
>>>> >
>>>> >
>>>> 0004 1,0 10341,3 40
>>>> >
>>>> >
>>>> 0005 3,4 29568,3 12
>>>> >
>>>> >
>>>> 0006 0,7 13369,0 401
>>>> >
>>>> >
>>>> 0007 2,8 36192,5 1
>>>> >
>>>> >
>>>> 0008 1,8 54854,6 6000
>>>> >
>>>> >
>>>> 0009 0,7 13369,0 298
>>>> >
>>>> >
>>>> 0010 3,4 29841,8 2
>>>> >
>>>> >
>>>> 0011 5,0 19932,2 1919
>>>> >
>>>> >
>>>> 0012 2,6 38835,0 1919
>>>> >
>>>> ==========================================================================================
>>>> >
>>>> > 5. Spark batch
>>>> > Performance:
>>>> >
>>>> >
>>>> Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results (Baseline)
>>>> >
>>>> >
>>>> 0000 1,5 65445,0 100000
>>>> >
>>>> >
>>>> 0001 1,3 79491,3 92000
>>>> >
>>>> >
>>>> 0002 0,9 112107,6 351
>>>> >
>>>> >
>>>> 0003 2,0 48804,3 580
>>>> >
>>>> >
>>>> 0004 1,2 8382,2 40
>>>> >
>>>> >
>>>> 0005 2,0 50838,8 12
>>>> >
>>>> >
>>>> 0006 1,0 9699,3 103
>>>> >
>>>> >
>>>> 0007 2,3 43308,8 1
>>>> >
>>>> >
>>>> 0008 2,1 46794,6 6000
>>>> >
>>>> >
>>>> 0009 1,1 8976,7 298
>>>> >
>>>> >
>>>> 0010 1,6 62111,8 1
>>>> >
>>>> >
>>>> 0011 2,1 46598,3 1919
>>>> >
>>>> >
>>>> 0012 2,3 43687,2 1919
>>>> >
>>>> ==========================================================================================
>>>> >
>>>> > Le mercredi 06 juin 2018 à 10:50 +0200, Etienne Chauchot a écrit :
>>>> >> Thanks JB for all your work ! I believe doing the first
>>>> gradle release
>>>> >> must have been hard.
>>>> >> I'll run Nexmark on the release and keep you posted.
>>>> >>
>>>> >> Best
>>>> >> Etienne
>>>> >>
>>>> >>
>>>> >> Le mercredi 06 juin 2018 à 10:44 +0200, Jean-Baptiste Onofré
>>>> a écrit :
>>>> >>> Hi everyone,
>>>> >>>
>>>> >>> Please review and vote on the release candidate #1 for the
>>>> version
>>>> >>> 2.5.0, as follows:
>>>> >>>
>>>> >>> [ ] +1, Approve the release
>>>> >>> [ ] -1, Do not approve the release (please provide specific
>>>> comments)
>>>> >>>
>>>> >>> NB: this is the first release using Gradle, so don't be too
>>>> harsh ;) A
>>>> >>> PR about the release guide will follow thanks to this release.
>>>> >>>
>>>> >>> The complete staging area is available for your review,
>>>> which includes:
>>>> >>> * JIRA release notes [1],
>>>> >>> * the official Apache source release to be deployed to
>>>> dist.apache.org <http://dist.apache.org/>
>>>> >>> [2], which is signed with the key with fingerprint C8282E76 [3],
>>>> >>> * all artifacts to be deployed to the Maven Central
>>>> Repository [4],
>>>> >>> * source code tag "v2.5.0-RC1" [5],
>>>> >>> * website pull request listing the release and publishing
>>>> the API
>>>> >>> reference manual [6].
>>>> >>> * Java artifacts were built with Gradle 4.7 (wrapper) and
>>>> OpenJDK/Oracle
>>>> >>> JDK 1.8.0_172 (Oracle Corporation 25.172-b11).
>>>> >>> * Python artifacts are deployed along with the source
>>>> release to the
>>>> >>> dist.apache.org <http://dist.apache.org/> [2].
>>>> >>>
>>>> >>> The vote will be open for at least 72 hours. It is adopted
>>>> by majority
>>>> >>> approval, with at least 3 PMC affirmative votes.
>>>> >>>
>>>> >>> Thanks,
>>>> >>> JB
>>>> >>>
>>>> >>> [1]
>>>> >>>
>>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12342847
>>>> >>> [2] https://dist.apache.org/repos/dist/dev/beam/2.5.0/
>>>> >>> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
>>>> >>> [4]
>>>> https://repository.apache.org/content/repositories/orgapachebeam-1041/
>>>> >>> [5] https://github.com/apache/beam/tree/v2.5.0-RC1
>>>> >>> [6] https://github.com/apache/beam-site/pull/463
>>>> >>>
>>>>
>
> --
> Got feedback? go/pabloem-feedback
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: [VOTE] Apache Beam, version 2.5.0, release candidate #1
Posted by Pablo Estrada <pa...@google.com>.
Hello all,
I'd like to request an extension of the voting period until Monday evening
(US time, so later in other geographical regions). This is because we were
only now able to publish Dataflow Workers, and have not had the chance to
run release validation tests on them. The extension will allow us to
validate and vote by Monday.
Is this acceptable to the community?
Best
-P.
On Fri, Jun 8, 2018 at 6:20 AM Alexey Romanenko <ar...@gmail.com>
wrote:
> Thank you JB for your work!
>
> I tested running simple streaming (*KafkaIO*) and batch (*TextIO / HDFS*)
> pipelines with SparkRunner on YARN cluster - it works fine.
>
> WBR,
> Alexey
>
>
> On 8 Jun 2018, at 10:00, Etienne Chauchot <ec...@apache.org> wrote:
>
> I forgot to vote:
> +1 (non binding).
> What I tested:
> - no functional or performance regression comparing to v2.4
> - dependencies in the poms are ok
>
> Etienne
> Le vendredi 08 juin 2018 à 08:27 +0200, Romain Manni-Bucau a écrit :
>
> +1 (non-binding), mainstream usage is not broken by the pom changes and
> runtime has no known regression compared to the 2.4.0
>
> (side note: kudo to JB for this build tool change release, I know how it
> can hurt ;))
>
> Romain Manni-Bucau
> @rmannibucau <https://twitter.com/rmannibucau> | Blog
> <https://rmannibucau.metawerx.net/> | Old Blog
> <http://rmannibucau.wordpress.com/> | Github
> <https://github.com/rmannibucau> | LinkedIn
> <https://www.linkedin.com/in/rmannibucau> | Book
> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>
>
> Le jeu. 7 juin 2018 à 16:17, Jean-Baptiste Onofré <jb...@nanthrax.net> a
> écrit :
>
> Thanks for the details Etienne !
>
> The good news is that the artifacts seem OK and the overall Nexmark
> results are consistent with the 2.4.0 release ones.
>
> I'm starting a complete review using the beam-samples as well.
>
> Regards
> JB
>
> On 07/06/2018 16:14, Etienne Chauchot wrote:
> > Hi,
> > I've just run the nexmark queries on v2.5.0-RC1 tag
> > What we can notice:
> > - query 3 (exercises CoGroupByKey, state and timer) shows different
> > output with DR between batch and streaming and with the other runners =>
> > I compared with v2.4 there were still these differences but with
> > different output size numbers
> >
> > - query 6 (exercises specialized combiner) shows different output
> > between the runners => the correct output is 401. strange that in batch
> > mode some runners output les Sellers. I compared with v2.4 same output
> >
> > - response time of query 7 (exercices Max transform, fanout and side
> > input) is very slow on DR => I compared with v2.4 , comparable execution
> > times
> >
> > I'm not comparing q10 because it is a write to GCS so it is very
> specific.
> >
> > => Basically no regression comparing to v2.4
> >
> > For the record here is the output (waiting for ongoing perfkit
> integration):
> >
> >
> > 1. DR batch
> >
> > Performance:
> >
> >
> Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results (Baseline)
> >
> >
> 0000 5,8 17283,1 100000
> >
> >
> 0001 3,2 31104,2 92000
> >
> >
> 0002 1,2 82918,7 351
> >
> >
> 0003 2,2 46210,7 458
> >
> >
> 0004 1,2 8503,4 40
> >
> >
> 0005 4,0 25220,7 12
> >
> >
> 0006 0,9 11148,3 401
> >
> >
> 0007 13,2 7580,9 1
> >
> >
> 0008 1,5 67340,1 6000
> >
> >
> 0009 0,7 14025,2 298
> >
> >
> 0010 12,8 7793,0 1
> >
> >
> 0011 2,4 42319,1 1919
> >
> >
> 0012 1,6 61462,8 1919
> >
> ==========================================================================================
> >
> > 2. DR streaming
> >
> > Performance:
> >
> >
> Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results (Baseline)
> >
> >
> 0000 6,5 15285,8 100000
> >
> >
> 0001 3,7 27397,3 92000
> >
> >
> 0002 1,4 69108,5 351
> >
> >
> 0003 3,2 31181,8 447
> >
> >
> 0004 1,2 8361,2 40
> >
> >
> 0005 5,3 18903,6 12
> >
> >
> 0006 0,9 11111,1 401
> >
> >
> 0007 82,5 1212,2 1
> >
> >
> 0008 2,0 51072,5 6000
> >
> >
> 0009 0,8 12903,2 298
> >
> >
> 0010 49,5 2021,8 1
> >
> >
> 0011 3,9 25667,4 1919
> >
> >
> 0012 2,4 41067,8 1919
> >
> ==========================================================================================
> >
> > 3. Flink batch
> > Performance:
> >
> >
> Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results (Baseline)
> >
> >
> 0000 1,0 97656,3 100000
> >
> >
> 0001 0,7 141643,1 92000
> >
> >
> 0002 0,4 228310,5 351
> >
> >
> 0003 1,6 64020,5 580
> >
> >
> 0004 0,7 13831,3 40
> >
> >
> 0005 1,4 72939,5 12
> >
> >
> 0006 0,5 20491,8 103
> >
> >
> 0007 1,3 74239,0 1
> >
> >
> 0008 0,8 121506,7 6000
> >
> >
> 0009 0,6 17953,3 298
> >
> >
> 0010 1,3 74682,6 1
> >
> >
> 0011 1,1 92936,8 1919
> >
> >
> 0012 0,8 123001,2 1919
> >
> ==========================================================================================
> >
> > 4. Flink streaming
> > Performance:
> >
> >
> Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results (Baseline)
> >
> >
> 0000 5,4 18677,6 100000
> >
> >
> 0001 2,8 35511,4 92000
> >
> >
> 0002 1,8 54318,3 351
> >
> >
> 0003 2,4 41614,6 580
> >
> >
> 0004 1,0 10341,3 40
> >
> >
> 0005 3,4 29568,3 12
> >
> >
> 0006 0,7 13369,0 401
> >
> >
> 0007 2,8 36192,5 1
> >
> >
> 0008 1,8 54854,6 6000
> >
> >
> 0009 0,7 13369,0 298
> >
> >
> 0010 3,4 29841,8 2
> >
> >
> 0011 5,0 19932,2 1919
> >
> >
> 0012 2,6 38835,0 1919
> >
> ==========================================================================================
> >
> > 5. Spark batch
> > Performance:
> >
> >
> Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results (Baseline)
> >
> >
> 0000 1,5 65445,0 100000
> >
> >
> 0001 1,3 79491,3 92000
> >
> >
> 0002 0,9 112107,6 351
> >
> >
> 0003 2,0 48804,3 580
> >
> >
> 0004 1,2 8382,2 40
> >
> >
> 0005 2,0 50838,8 12
> >
> >
> 0006 1,0 9699,3 103
> >
> >
> 0007 2,3 43308,8 1
> >
> >
> 0008 2,1 46794,6 6000
> >
> >
> 0009 1,1 8976,7 298
> >
> >
> 0010 1,6 62111,8 1
> >
> >
> 0011 2,1 46598,3 1919
> >
> >
> 0012 2,3 43687,2 1919
> >
> ==========================================================================================
> >
> > Le mercredi 06 juin 2018 à 10:50 +0200, Etienne Chauchot a écrit :
> >> Thanks JB for all your work ! I believe doing the first gradle release
> >> must have been hard.
> >> I'll run Nexmark on the release and keep you posted.
> >>
> >> Best
> >> Etienne
> >>
> >>
> >> Le mercredi 06 juin 2018 à 10:44 +0200, Jean-Baptiste Onofré a écrit :
> >>> Hi everyone,
> >>>
> >>> Please review and vote on the release candidate #1 for the version
> >>> 2.5.0, as follows:
> >>>
> >>> [ ] +1, Approve the release
> >>> [ ] -1, Do not approve the release (please provide specific comments)
> >>>
> >>> NB: this is the first release using Gradle, so don't be too harsh ;) A
> >>> PR about the release guide will follow thanks to this release.
> >>>
> >>> The complete staging area is available for your review, which includes:
> >>> * JIRA release notes [1],
> >>> * the official Apache source release to be deployed to dist.apache.org
> >>> [2], which is signed with the key with fingerprint C8282E76 [3],
> >>> * all artifacts to be deployed to the Maven Central Repository [4],
> >>> * source code tag "v2.5.0-RC1" [5],
> >>> * website pull request listing the release and publishing the API
> >>> reference manual [6].
> >>> * Java artifacts were built with Gradle 4.7 (wrapper) and
> OpenJDK/Oracle
> >>> JDK 1.8.0_172 (Oracle Corporation 25.172-b11).
> >>> * Python artifacts are deployed along with the source release to the
> >>> dist.apache.org [2].
> >>>
> >>> The vote will be open for at least 72 hours. It is adopted by majority
> >>> approval, with at least 3 PMC affirmative votes.
> >>>
> >>> Thanks,
> >>> JB
> >>>
> >>> [1]
> >>>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12342847
> >>> [2] https://dist.apache.org/repos/dist/dev/beam/2.5.0/
> >>> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
> >>> [4]
> https://repository.apache.org/content/repositories/orgapachebeam-1041/
> >>> [5] https://github.com/apache/beam/tree/v2.5.0-RC1
> >>> [6] https://github.com/apache/beam-site/pull/463
> >>>
>
>
> --
Got feedback? go/pabloem-feedback
Re: [VOTE] Apache Beam, version 2.5.0, release candidate #1
Posted by Alexey Romanenko <ar...@gmail.com>.
Thank you JB for your work!
I tested running simple streaming (KafkaIO) and batch (TextIO / HDFS) pipelines with SparkRunner on YARN cluster - it works fine.
WBR,
Alexey
> On 8 Jun 2018, at 10:00, Etienne Chauchot <ec...@apache.org> wrote:
>
> I forgot to vote:
> +1 (non binding).
> What I tested:
> - no functional or performance regression comparing to v2.4
> - dependencies in the poms are ok
>
> Etienne
> Le vendredi 08 juin 2018 à 08:27 +0200, Romain Manni-Bucau a écrit :
>> +1 (non-binding), mainstream usage is not broken by the pom changes and runtime has no known regression compared to the 2.4.0
>>
>> (side note: kudo to JB for this build tool change release, I know how it can hurt ;))
>>
>> Romain Manni-Bucau
>> @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Old Blog <http://rmannibucau.wordpress.com/> | Github <https://github.com/rmannibucau> | LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>
>> Le jeu. 7 juin 2018 à 16:17, Jean-Baptiste Onofré <jb@nanthrax.net <ma...@nanthrax.net>> a écrit :
>>> Thanks for the details Etienne !
>>>
>>> The good news is that the artifacts seem OK and the overall Nexmark
>>> results are consistent with the 2.4.0 release ones.
>>>
>>> I'm starting a complete review using the beam-samples as well.
>>>
>>> Regards
>>> JB
>>>
>>> On 07/06/2018 16:14, Etienne Chauchot wrote:
>>> > Hi,
>>> > I've just run the nexmark queries on v2.5.0-RC1 tag
>>> > What we can notice:
>>> > - query 3 (exercises CoGroupByKey, state and timer) shows different
>>> > output with DR between batch and streaming and with the other runners =>
>>> > I compared with v2.4 there were still these differences but with
>>> > different output size numbers
>>> >
>>> > - query 6 (exercises specialized combiner) shows different output
>>> > between the runners => the correct output is 401. strange that in batch
>>> > mode some runners output les Sellers. I compared with v2.4 same output
>>> >
>>> > - response time of query 7 (exercices Max transform, fanout and side
>>> > input) is very slow on DR => I compared with v2.4 , comparable execution
>>> > times
>>> >
>>> > I'm not comparing q10 because it is a write to GCS so it is very specific.
>>> >
>>> > => Basically no regression comparing to v2.4
>>> >
>>> > For the record here is the output (waiting for ongoing perfkit integration):
>>> >
>>> >
>>> > 1. DR batch
>>> >
>>> > Performance:
>>> >
>>> > Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results (Baseline)
>>> >
>>> > 0000 5,8 17283,1 100000
>>> >
>>> > 0001 3,2 31104,2 92000
>>> >
>>> > 0002 1,2 82918,7 351
>>> >
>>> > 0003 2,2 46210,7 458
>>> >
>>> > 0004 1,2 8503,4 40
>>> >
>>> > 0005 4,0 25220,7 12
>>> >
>>> > 0006 0,9 11148,3 401
>>> >
>>> > 0007 13,2 7580,9 1
>>> >
>>> > 0008 1,5 67340,1 6000
>>> >
>>> > 0009 0,7 14025,2 298
>>> >
>>> > 0010 12,8 7793,0 1
>>> >
>>> > 0011 2,4 42319,1 1919
>>> >
>>> > 0012 1,6 61462,8 1919
>>> > ==========================================================================================
>>> >
>>> > 2. DR streaming
>>> >
>>> > Performance:
>>> >
>>> > Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results (Baseline)
>>> >
>>> > 0000 6,5 15285,8 100000
>>> >
>>> > 0001 3,7 27397,3 92000
>>> >
>>> > 0002 1,4 69108,5 351
>>> >
>>> > 0003 3,2 31181,8 447
>>> >
>>> > 0004 1,2 8361,2 40
>>> >
>>> > 0005 5,3 18903,6 12
>>> >
>>> > 0006 0,9 11111,1 401
>>> >
>>> > 0007 82,5 1212,2 1
>>> >
>>> > 0008 2,0 51072,5 6000
>>> >
>>> > 0009 0,8 12903,2 298
>>> >
>>> > 0010 49,5 2021,8 1
>>> >
>>> > 0011 3,9 25667,4 1919
>>> >
>>> > 0012 2,4 41067,8 1919
>>> > ==========================================================================================
>>> >
>>> > 3. Flink batch
>>> > Performance:
>>> >
>>> > Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results (Baseline)
>>> >
>>> > 0000 1,0 97656,3 100000
>>> >
>>> > 0001 0,7 141643,1 92000
>>> >
>>> > 0002 0,4 228310,5 351
>>> >
>>> > 0003 1,6 64020,5 580
>>> >
>>> > 0004 0,7 13831,3 40
>>> >
>>> > 0005 1,4 72939,5 12
>>> >
>>> > 0006 0,5 20491,8 103
>>> >
>>> > 0007 1,3 74239,0 1
>>> >
>>> > 0008 0,8 121506,7 6000
>>> >
>>> > 0009 0,6 17953,3 298
>>> >
>>> > 0010 1,3 74682,6 1
>>> >
>>> > 0011 1,1 92936,8 1919
>>> >
>>> > 0012 0,8 123001,2 1919
>>> > ==========================================================================================
>>> >
>>> > 4. Flink streaming
>>> > Performance:
>>> >
>>> > Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results (Baseline)
>>> >
>>> > 0000 5,4 18677,6 100000
>>> >
>>> > 0001 2,8 35511,4 92000
>>> >
>>> > 0002 1,8 54318,3 351
>>> >
>>> > 0003 2,4 41614,6 580
>>> >
>>> > 0004 1,0 10341,3 40
>>> >
>>> > 0005 3,4 29568,3 12
>>> >
>>> > 0006 0,7 13369,0 401
>>> >
>>> > 0007 2,8 36192,5 1
>>> >
>>> > 0008 1,8 54854,6 6000
>>> >
>>> > 0009 0,7 13369,0 298
>>> >
>>> > 0010 3,4 29841,8 2
>>> >
>>> > 0011 5,0 19932,2 1919
>>> >
>>> > 0012 2,6 38835,0 1919
>>> > ==========================================================================================
>>> >
>>> > 5. Spark batch
>>> > Performance:
>>> >
>>> > Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results (Baseline)
>>> >
>>> > 0000 1,5 65445,0 100000
>>> >
>>> > 0001 1,3 79491,3 92000
>>> >
>>> > 0002 0,9 112107,6 351
>>> >
>>> > 0003 2,0 48804,3 580
>>> >
>>> > 0004 1,2 8382,2 40
>>> >
>>> > 0005 2,0 50838,8 12
>>> >
>>> > 0006 1,0 9699,3 103
>>> >
>>> > 0007 2,3 43308,8 1
>>> >
>>> > 0008 2,1 46794,6 6000
>>> >
>>> > 0009 1,1 8976,7 298
>>> >
>>> > 0010 1,6 62111,8 1
>>> >
>>> > 0011 2,1 46598,3 1919
>>> >
>>> > 0012 2,3 43687,2 1919
>>> > ==========================================================================================
>>> >
>>> > Le mercredi 06 juin 2018 à 10:50 +0200, Etienne Chauchot a écrit :
>>> >> Thanks JB for all your work ! I believe doing the first gradle release
>>> >> must have been hard.
>>> >> I'll run Nexmark on the release and keep you posted.
>>> >>
>>> >> Best
>>> >> Etienne
>>> >>
>>> >>
>>> >> Le mercredi 06 juin 2018 à 10:44 +0200, Jean-Baptiste Onofré a écrit :
>>> >>> Hi everyone,
>>> >>>
>>> >>> Please review and vote on the release candidate #1 for the version
>>> >>> 2.5.0, as follows:
>>> >>>
>>> >>> [ ] +1, Approve the release
>>> >>> [ ] -1, Do not approve the release (please provide specific comments)
>>> >>>
>>> >>> NB: this is the first release using Gradle, so don't be too harsh ;) A
>>> >>> PR about the release guide will follow thanks to this release.
>>> >>>
>>> >>> The complete staging area is available for your review, which includes:
>>> >>> * JIRA release notes [1],
>>> >>> * the official Apache source release to be deployed to dist.apache.org <http://dist.apache.org/>
>>> >>> [2], which is signed with the key with fingerprint C8282E76 [3],
>>> >>> * all artifacts to be deployed to the Maven Central Repository [4],
>>> >>> * source code tag "v2.5.0-RC1" [5],
>>> >>> * website pull request listing the release and publishing the API
>>> >>> reference manual [6].
>>> >>> * Java artifacts were built with Gradle 4.7 (wrapper) and OpenJDK/Oracle
>>> >>> JDK 1.8.0_172 (Oracle Corporation 25.172-b11).
>>> >>> * Python artifacts are deployed along with the source release to the
>>> >>> dist.apache.org <http://dist.apache.org/> [2].
>>> >>>
>>> >>> The vote will be open for at least 72 hours. It is adopted by majority
>>> >>> approval, with at least 3 PMC affirmative votes.
>>> >>>
>>> >>> Thanks,
>>> >>> JB
>>> >>>
>>> >>> [1]
>>> >>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12342847 <https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12342847>
>>> >>> [2] https://dist.apache.org/repos/dist/dev/beam/2.5.0/ <https://dist.apache.org/repos/dist/dev/beam/2.5.0/>
>>> >>> [3] https://dist.apache.org/repos/dist/release/beam/KEYS <https://dist.apache.org/repos/dist/release/beam/KEYS>
>>> >>> [4] https://repository.apache.org/content/repositories/orgapachebeam-1041/ <https://repository.apache.org/content/repositories/orgapachebeam-1041/>
>>> >>> [5] https://github.com/apache/beam/tree/v2.5.0-RC1 <https://github.com/apache/beam/tree/v2.5.0-RC1>
>>> >>> [6] https://github.com/apache/beam-site/pull/463 <https://github.com/apache/beam-site/pull/463>
>>> >>>
>>>
Re: [VOTE] Apache Beam, version 2.5.0, release candidate #1
Posted by Etienne Chauchot <ec...@apache.org>.
I forgot to vote:+1 (non binding). What I tested: - no functional or performance regression comparing to v2.4-
dependencies in the poms are ok
EtienneLe vendredi 08 juin 2018 à 08:27 +0200, Romain Manni-Bucau a écrit :
> +1 (non-binding), mainstream usage is not broken by the pom changes and runtime has no known regression compared to
> the 2.4.0
> (side note: kudo to JB for this build tool change release, I know how it can hurt ;))
>
> Romain Manni-Bucau
> @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book
>
>
> Le jeu. 7 juin 2018 à 16:17, Jean-Baptiste Onofré <jb...@nanthrax.net> a écrit :
> > Thanks for the details Etienne !
> >
> >
> >
> > The good news is that the artifacts seem OK and the overall Nexmark
> >
> > results are consistent with the 2.4.0 release ones.
> >
> >
> >
> > I'm starting a complete review using the beam-samples as well.
> >
> >
> >
> > Regards
> >
> > JB
> >
> >
> >
> > On 07/06/2018 16:14, Etienne Chauchot wrote:
> >
> > > Hi,
> >
> > > I've just run the nexmark queries on v2.5.0-RC1 tag
> >
> > > What we can notice:
> >
> > > - query 3 (exercises CoGroupByKey, state and timer) shows different
> >
> > > output with DR between batch and streaming and with the other runners =>
> >
> > > I compared with v2.4 there were still these differences but with
> >
> > > different output size numbers
> >
> > >
> >
> > > - query 6 (exercises specialized combiner) shows different output
> >
> > > between the runners => the correct output is 401. strange that in batch
> >
> > > mode some runners output les Sellers. I compared with v2.4 same output
> >
> > >
> >
> > > - response time of query 7 (exercices Max transform, fanout and side
> >
> > > input) is very slow on DR => I compared with v2.4 , comparable execution
> >
> > > times
> >
> > >
> >
> > > I'm not comparing q10 because it is a write to GCS so it is very specific.
> >
> > >
> >
> > > => Basically no regression comparing to v2.4
> >
> > >
> >
> > > For the record here is the output (waiting for ongoing perfkit integration):
> >
> > >
> >
> > >
> >
> > > 1. DR batch
> >
> > >
> >
> > > Performance:
> >
> > >
> >
> > > Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results (Baseline)
> >
> > >
> >
> > > 0000 5,8 17283,1 100000
> >
> > >
> >
> > > 0001 3,2 31104,2 92000
> >
> > >
> >
> > > 0002 1,2 82918,7 351
> >
> > >
> >
> > > 0003 2,2 46210,7 458
> >
> > >
> >
> > > 0004 1,2 8503,4 40
> >
> > >
> >
> > > 0005 4,0 25220,7 12
> >
> > >
> >
> > > 0006 0,9 11148,3 401
> >
> > >
> >
> > > 0007 13,2 7580,9 1
> >
> > >
> >
> > > 0008 1,5 67340,1 6000
> >
> > >
> >
> > > 0009 0,7 14025,2 298
> >
> > >
> >
> > > 0010 12,8 7793,0 1
> >
> > >
> >
> > > 0011 2,4 42319,1 1919
> >
> > >
> >
> > > 0012 1,6 61462,8 1919
> >
> > > ==========================================================================================
> >
> > >
> >
> > > 2. DR streaming
> >
> > >
> >
> > > Performance:
> >
> > >
> >
> > > Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results (Baseline)
> >
> > >
> >
> > > 0000 6,5 15285,8 100000
> >
> > >
> >
> > > 0001 3,7 27397,3 92000
> >
> > >
> >
> > > 0002 1,4 69108,5 351
> >
> > >
> >
> > > 0003 3,2 31181,8 447
> >
> > >
> >
> > > 0004 1,2 8361,2 40
> >
> > >
> >
> > > 0005 5,3 18903,6 12
> >
> > >
> >
> > > 0006 0,9 11111,1 401
> >
> > >
> >
> > > 0007 82,5 1212,2 1
> >
> > >
> >
> > > 0008 2,0 51072,5 6000
> >
> > >
> >
> > > 0009 0,8 12903,2 298
> >
> > >
> >
> > > 0010 49,5 2021,8 1
> >
> > >
> >
> > > 0011 3,9 25667,4 1919
> >
> > >
> >
> > > 0012 2,4 41067,8 1919
> >
> > > ==========================================================================================
> >
> > >
> >
> > > 3. Flink batch
> >
> > > Performance:
> >
> > >
> >
> > > Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results (Baseline)
> >
> > >
> >
> > > 0000 1,0 97656,3 100000
> >
> > >
> >
> > > 0001 0,7 141643,1 92000
> >
> > >
> >
> > > 0002 0,4 228310,5 351
> >
> > >
> >
> > > 0003 1,6 64020,5 580
> >
> > >
> >
> > > 0004 0,7 13831,3 40
> >
> > >
> >
> > > 0005 1,4 72939,5 12
> >
> > >
> >
> > > 0006 0,5 20491,8 103
> >
> > >
> >
> > > 0007 1,3 74239,0 1
> >
> > >
> >
> > > 0008 0,8 121506,7 6000
> >
> > >
> >
> > > 0009 0,6 17953,3 298
> >
> > >
> >
> > > 0010 1,3 74682,6 1
> >
> > >
> >
> > > 0011 1,1 92936,8 1919
> >
> > >
> >
> > > 0012 0,8 123001,2 1919
> >
> > > ==========================================================================================
> >
> > >
> >
> > > 4. Flink streaming
> >
> > > Performance:
> >
> > >
> >
> > > Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results (Baseline)
> >
> > >
> >
> > > 0000 5,4 18677,6 100000
> >
> > >
> >
> > > 0001 2,8 35511,4 92000
> >
> > >
> >
> > > 0002 1,8 54318,3 351
> >
> > >
> >
> > > 0003 2,4 41614,6 580
> >
> > >
> >
> > > 0004 1,0 10341,3 40
> >
> > >
> >
> > > 0005 3,4 29568,3 12
> >
> > >
> >
> > > 0006 0,7 13369,0 401
> >
> > >
> >
> > > 0007 2,8 36192,5 1
> >
> > >
> >
> > > 0008 1,8 54854,6 6000
> >
> > >
> >
> > > 0009 0,7 13369,0 298
> >
> > >
> >
> > > 0010 3,4 29841,8 2
> >
> > >
> >
> > > 0011 5,0 19932,2 1919
> >
> > >
> >
> > > 0012 2,6 38835,0 1919
> >
> > > ==========================================================================================
> >
> > >
> >
> > > 5. Spark batch
> >
> > > Performance:
> >
> > >
> >
> > > Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results (Baseline)
> >
> > >
> >
> > > 0000 1,5 65445,0 100000
> >
> > >
> >
> > > 0001 1,3 79491,3 92000
> >
> > >
> >
> > > 0002 0,9 112107,6 351
> >
> > >
> >
> > > 0003 2,0 48804,3 580
> >
> > >
> >
> > > 0004 1,2 8382,2 40
> >
> > >
> >
> > > 0005 2,0 50838,8 12
> >
> > >
> >
> > > 0006 1,0 9699,3 103
> >
> > >
> >
> > > 0007 2,3 43308,8 1
> >
> > >
> >
> > > 0008 2,1 46794,6 6000
> >
> > >
> >
> > > 0009 1,1 8976,7 298
> >
> > >
> >
> > > 0010 1,6 62111,8 1
> >
> > >
> >
> > > 0011 2,1 46598,3 1919
> >
> > >
> >
> > > 0012 2,3 43687,2 1919
> >
> > > ==========================================================================================
> >
> > >
> >
> > > Le mercredi 06 juin 2018 à 10:50 +0200, Etienne Chauchot a écrit :
> >
> > >> Thanks JB for all your work ! I believe doing the first gradle release
> >
> > >> must have been hard.
> >
> > >> I'll run Nexmark on the release and keep you posted.
> >
> > >>
> >
> > >> Best
> >
> > >> Etienne
> >
> > >>
> >
> > >>
> >
> > >> Le mercredi 06 juin 2018 à 10:44 +0200, Jean-Baptiste Onofré a écrit :
> >
> > >>> Hi everyone,
> >
> > >>>
> >
> > >>> Please review and vote on the release candidate #1 for the version
> >
> > >>> 2.5.0, as follows:
> >
> > >>>
> >
> > >>> [ ] +1, Approve the release
> >
> > >>> [ ] -1, Do not approve the release (please provide specific comments)
> >
> > >>>
> >
> > >>> NB: this is the first release using Gradle, so don't be too harsh ;) A
> >
> > >>> PR about the release guide will follow thanks to this release.
> >
> > >>>
> >
> > >>> The complete staging area is available for your review, which includes:
> >
> > >>> * JIRA release notes [1],
> >
> > >>> * the official Apache source release to be deployed to dist.apache.org
> >
> > >>> [2], which is signed with the key with fingerprint C8282E76 [3],
> >
> > >>> * all artifacts to be deployed to the Maven Central Repository [4],
> >
> > >>> * source code tag "v2.5.0-RC1" [5],
> >
> > >>> * website pull request listing the release and publishing the API
> >
> > >>> reference manual [6].
> >
> > >>> * Java artifacts were built with Gradle 4.7 (wrapper) and OpenJDK/Oracle
> >
> > >>> JDK 1.8.0_172 (Oracle Corporation 25.172-b11).
> >
> > >>> * Python artifacts are deployed along with the source release to the
> >
> > >>> dist.apache.org [2].
> >
> > >>>
> >
> > >>> The vote will be open for at least 72 hours. It is adopted by majority
> >
> > >>> approval, with at least 3 PMC affirmative votes.
> >
> > >>>
> >
> > >>> Thanks,
> >
> > >>> JB
> >
> > >>>
> >
> > >>> [1]
> >
> > >>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12342847
> >
> > >>> [2] https://dist.apache.org/repos/dist/dev/beam/2.5.0/
> >
> > >>> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
> >
> > >>> [4] https://repository.apache.org/content/repositories/orgapachebeam-1041/
> >
> > >>> [5] https://github.com/apache/beam/tree/v2.5.0-RC1
> >
> > >>> [6] https://github.com/apache/beam-site/pull/463
> >
> > >>>
> >
> >
> >
Re: [VOTE] Apache Beam, version 2.5.0, release candidate #1
Posted by Romain Manni-Bucau <rm...@gmail.com>.
+1 (non-binding), mainstream usage is not broken by the pom changes and
runtime has no known regression compared to the 2.4.0
(side note: kudo to JB for this build tool change release, I know how it
can hurt ;))
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<https://rmannibucau.metawerx.net/> | Old Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
<https://www.packtpub.com/application-development/java-ee-8-high-performance>
Le jeu. 7 juin 2018 à 16:17, Jean-Baptiste Onofré <jb...@nanthrax.net> a
écrit :
> Thanks for the details Etienne !
>
> The good news is that the artifacts seem OK and the overall Nexmark
> results are consistent with the 2.4.0 release ones.
>
> I'm starting a complete review using the beam-samples as well.
>
> Regards
> JB
>
> On 07/06/2018 16:14, Etienne Chauchot wrote:
> > Hi,
> > I've just run the nexmark queries on v2.5.0-RC1 tag
> > What we can notice:
> > - query 3 (exercises CoGroupByKey, state and timer) shows different
> > output with DR between batch and streaming and with the other runners =>
> > I compared with v2.4 there were still these differences but with
> > different output size numbers
> >
> > - query 6 (exercises specialized combiner) shows different output
> > between the runners => the correct output is 401. strange that in batch
> > mode some runners output les Sellers. I compared with v2.4 same output
> >
> > - response time of query 7 (exercices Max transform, fanout and side
> > input) is very slow on DR => I compared with v2.4 , comparable execution
> > times
> >
> > I'm not comparing q10 because it is a write to GCS so it is very
> specific.
> >
> > => Basically no regression comparing to v2.4
> >
> > For the record here is the output (waiting for ongoing perfkit
> integration):
> >
> >
> > 1. DR batch
> >
> > Performance:
> >
> >
> Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results (Baseline)
> >
> >
> 0000 5,8 17283,1 100000
> >
> >
> 0001 3,2 31104,2 92000
> >
> >
> 0002 1,2 82918,7 351
> >
> >
> 0003 2,2 46210,7 458
> >
> >
> 0004 1,2 8503,4 40
> >
> >
> 0005 4,0 25220,7 12
> >
> >
> 0006 0,9 11148,3 401
> >
> >
> 0007 13,2 7580,9 1
> >
> >
> 0008 1,5 67340,1 6000
> >
> >
> 0009 0,7 14025,2 298
> >
> >
> 0010 12,8 7793,0 1
> >
> >
> 0011 2,4 42319,1 1919
> >
> >
> 0012 1,6 61462,8 1919
> >
> ==========================================================================================
> >
> > 2. DR streaming
> >
> > Performance:
> >
> >
> Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results (Baseline)
> >
> >
> 0000 6,5 15285,8 100000
> >
> >
> 0001 3,7 27397,3 92000
> >
> >
> 0002 1,4 69108,5 351
> >
> >
> 0003 3,2 31181,8 447
> >
> >
> 0004 1,2 8361,2 40
> >
> >
> 0005 5,3 18903,6 12
> >
> >
> 0006 0,9 11111,1 401
> >
> >
> 0007 82,5 1212,2 1
> >
> >
> 0008 2,0 51072,5 6000
> >
> >
> 0009 0,8 12903,2 298
> >
> >
> 0010 49,5 2021,8 1
> >
> >
> 0011 3,9 25667,4 1919
> >
> >
> 0012 2,4 41067,8 1919
> >
> ==========================================================================================
> >
> > 3. Flink batch
> > Performance:
> >
> >
> Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results (Baseline)
> >
> >
> 0000 1,0 97656,3 100000
> >
> >
> 0001 0,7 141643,1 92000
> >
> >
> 0002 0,4 228310,5 351
> >
> >
> 0003 1,6 64020,5 580
> >
> >
> 0004 0,7 13831,3 40
> >
> >
> 0005 1,4 72939,5 12
> >
> >
> 0006 0,5 20491,8 103
> >
> >
> 0007 1,3 74239,0 1
> >
> >
> 0008 0,8 121506,7 6000
> >
> >
> 0009 0,6 17953,3 298
> >
> >
> 0010 1,3 74682,6 1
> >
> >
> 0011 1,1 92936,8 1919
> >
> >
> 0012 0,8 123001,2 1919
> >
> ==========================================================================================
> >
> > 4. Flink streaming
> > Performance:
> >
> >
> Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results (Baseline)
> >
> >
> 0000 5,4 18677,6 100000
> >
> >
> 0001 2,8 35511,4 92000
> >
> >
> 0002 1,8 54318,3 351
> >
> >
> 0003 2,4 41614,6 580
> >
> >
> 0004 1,0 10341,3 40
> >
> >
> 0005 3,4 29568,3 12
> >
> >
> 0006 0,7 13369,0 401
> >
> >
> 0007 2,8 36192,5 1
> >
> >
> 0008 1,8 54854,6 6000
> >
> >
> 0009 0,7 13369,0 298
> >
> >
> 0010 3,4 29841,8 2
> >
> >
> 0011 5,0 19932,2 1919
> >
> >
> 0012 2,6 38835,0 1919
> >
> ==========================================================================================
> >
> > 5. Spark batch
> > Performance:
> >
> >
> Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results (Baseline)
> >
> >
> 0000 1,5 65445,0 100000
> >
> >
> 0001 1,3 79491,3 92000
> >
> >
> 0002 0,9 112107,6 351
> >
> >
> 0003 2,0 48804,3 580
> >
> >
> 0004 1,2 8382,2 40
> >
> >
> 0005 2,0 50838,8 12
> >
> >
> 0006 1,0 9699,3 103
> >
> >
> 0007 2,3 43308,8 1
> >
> >
> 0008 2,1 46794,6 6000
> >
> >
> 0009 1,1 8976,7 298
> >
> >
> 0010 1,6 62111,8 1
> >
> >
> 0011 2,1 46598,3 1919
> >
> >
> 0012 2,3 43687,2 1919
> >
> ==========================================================================================
> >
> > Le mercredi 06 juin 2018 à 10:50 +0200, Etienne Chauchot a écrit :
> >> Thanks JB for all your work ! I believe doing the first gradle release
> >> must have been hard.
> >> I'll run Nexmark on the release and keep you posted.
> >>
> >> Best
> >> Etienne
> >>
> >>
> >> Le mercredi 06 juin 2018 à 10:44 +0200, Jean-Baptiste Onofré a écrit :
> >>> Hi everyone,
> >>>
> >>> Please review and vote on the release candidate #1 for the version
> >>> 2.5.0, as follows:
> >>>
> >>> [ ] +1, Approve the release
> >>> [ ] -1, Do not approve the release (please provide specific comments)
> >>>
> >>> NB: this is the first release using Gradle, so don't be too harsh ;) A
> >>> PR about the release guide will follow thanks to this release.
> >>>
> >>> The complete staging area is available for your review, which includes:
> >>> * JIRA release notes [1],
> >>> * the official Apache source release to be deployed to dist.apache.org
> >>> [2], which is signed with the key with fingerprint C8282E76 [3],
> >>> * all artifacts to be deployed to the Maven Central Repository [4],
> >>> * source code tag "v2.5.0-RC1" [5],
> >>> * website pull request listing the release and publishing the API
> >>> reference manual [6].
> >>> * Java artifacts were built with Gradle 4.7 (wrapper) and
> OpenJDK/Oracle
> >>> JDK 1.8.0_172 (Oracle Corporation 25.172-b11).
> >>> * Python artifacts are deployed along with the source release to the
> >>> dist.apache.org [2].
> >>>
> >>> The vote will be open for at least 72 hours. It is adopted by majority
> >>> approval, with at least 3 PMC affirmative votes.
> >>>
> >>> Thanks,
> >>> JB
> >>>
> >>> [1]
> >>>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12342847
> >>> [2] https://dist.apache.org/repos/dist/dev/beam/2.5.0/
> >>> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
> >>> [4]
> https://repository.apache.org/content/repositories/orgapachebeam-1041/
> >>> [5] https://github.com/apache/beam/tree/v2.5.0-RC1
> >>> [6] https://github.com/apache/beam-site/pull/463
> >>>
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
Re: [VOTE] Apache Beam, version 2.5.0, release candidate #1
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Thanks for the details Etienne !
The good news is that the artifacts seem OK and the overall Nexmark
results are consistent with the 2.4.0 release ones.
I'm starting a complete review using the beam-samples as well.
Regards
JB
On 07/06/2018 16:14, Etienne Chauchot wrote:
> Hi,
> I've just run the nexmark queries on v2.5.0-RC1 tag
> What we can notice:
> - query 3 (exercises CoGroupByKey, state and timer) shows different
> output with DR between batch and streaming and with the other runners =>
> I compared with v2.4 there were still these differences but with
> different output size numbers
>
> - query 6 (exercises specialized combiner) shows different output
> between the runners => the correct output is 401. strange that in batch
> mode some runners output les Sellers. I compared with v2.4 same output
>
> - response time of query 7 (exercices Max transform, fanout and side
> input) is very slow on DR => I compared with v2.4 , comparable execution
> times
>
> I'm not comparing q10 because it is a write to GCS so it is very specific.
>
> => Basically no regression comparing to v2.4
>
> For the record here is the output (waiting for ongoing perfkit integration):
>
>
> 1. DR batch
>
> Performance:
>
> Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results (Baseline)
>
> 0000 5,8 17283,1 100000
>
> 0001 3,2 31104,2 92000
>
> 0002 1,2 82918,7 351
>
> 0003 2,2 46210,7 458
>
> 0004 1,2 8503,4 40
>
> 0005 4,0 25220,7 12
>
> 0006 0,9 11148,3 401
>
> 0007 13,2 7580,9 1
>
> 0008 1,5 67340,1 6000
>
> 0009 0,7 14025,2 298
>
> 0010 12,8 7793,0 1
>
> 0011 2,4 42319,1 1919
>
> 0012 1,6 61462,8 1919
> ==========================================================================================
>
> 2. DR streaming
>
> Performance:
>
> Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results (Baseline)
>
> 0000 6,5 15285,8 100000
>
> 0001 3,7 27397,3 92000
>
> 0002 1,4 69108,5 351
>
> 0003 3,2 31181,8 447
>
> 0004 1,2 8361,2 40
>
> 0005 5,3 18903,6 12
>
> 0006 0,9 11111,1 401
>
> 0007 82,5 1212,2 1
>
> 0008 2,0 51072,5 6000
>
> 0009 0,8 12903,2 298
>
> 0010 49,5 2021,8 1
>
> 0011 3,9 25667,4 1919
>
> 0012 2,4 41067,8 1919
> ==========================================================================================
>
> 3. Flink batch
> Performance:
>
> Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results (Baseline)
>
> 0000 1,0 97656,3 100000
>
> 0001 0,7 141643,1 92000
>
> 0002 0,4 228310,5 351
>
> 0003 1,6 64020,5 580
>
> 0004 0,7 13831,3 40
>
> 0005 1,4 72939,5 12
>
> 0006 0,5 20491,8 103
>
> 0007 1,3 74239,0 1
>
> 0008 0,8 121506,7 6000
>
> 0009 0,6 17953,3 298
>
> 0010 1,3 74682,6 1
>
> 0011 1,1 92936,8 1919
>
> 0012 0,8 123001,2 1919
> ==========================================================================================
>
> 4. Flink streaming
> Performance:
>
> Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results (Baseline)
>
> 0000 5,4 18677,6 100000
>
> 0001 2,8 35511,4 92000
>
> 0002 1,8 54318,3 351
>
> 0003 2,4 41614,6 580
>
> 0004 1,0 10341,3 40
>
> 0005 3,4 29568,3 12
>
> 0006 0,7 13369,0 401
>
> 0007 2,8 36192,5 1
>
> 0008 1,8 54854,6 6000
>
> 0009 0,7 13369,0 298
>
> 0010 3,4 29841,8 2
>
> 0011 5,0 19932,2 1919
>
> 0012 2,6 38835,0 1919
> ==========================================================================================
>
> 5. Spark batch
> Performance:
>
> Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results (Baseline)
>
> 0000 1,5 65445,0 100000
>
> 0001 1,3 79491,3 92000
>
> 0002 0,9 112107,6 351
>
> 0003 2,0 48804,3 580
>
> 0004 1,2 8382,2 40
>
> 0005 2,0 50838,8 12
>
> 0006 1,0 9699,3 103
>
> 0007 2,3 43308,8 1
>
> 0008 2,1 46794,6 6000
>
> 0009 1,1 8976,7 298
>
> 0010 1,6 62111,8 1
>
> 0011 2,1 46598,3 1919
>
> 0012 2,3 43687,2 1919
> ==========================================================================================
>
> Le mercredi 06 juin 2018 à 10:50 +0200, Etienne Chauchot a écrit :
>> Thanks JB for all your work ! I believe doing the first gradle release
>> must have been hard.
>> I'll run Nexmark on the release and keep you posted.
>>
>> Best
>> Etienne
>>
>>
>> Le mercredi 06 juin 2018 à 10:44 +0200, Jean-Baptiste Onofré a écrit :
>>> Hi everyone,
>>>
>>> Please review and vote on the release candidate #1 for the version
>>> 2.5.0, as follows:
>>>
>>> [ ] +1, Approve the release
>>> [ ] -1, Do not approve the release (please provide specific comments)
>>>
>>> NB: this is the first release using Gradle, so don't be too harsh ;) A
>>> PR about the release guide will follow thanks to this release.
>>>
>>> The complete staging area is available for your review, which includes:
>>> * JIRA release notes [1],
>>> * the official Apache source release to be deployed to dist.apache.org
>>> [2], which is signed with the key with fingerprint C8282E76 [3],
>>> * all artifacts to be deployed to the Maven Central Repository [4],
>>> * source code tag "v2.5.0-RC1" [5],
>>> * website pull request listing the release and publishing the API
>>> reference manual [6].
>>> * Java artifacts were built with Gradle 4.7 (wrapper) and OpenJDK/Oracle
>>> JDK 1.8.0_172 (Oracle Corporation 25.172-b11).
>>> * Python artifacts are deployed along with the source release to the
>>> dist.apache.org [2].
>>>
>>> The vote will be open for at least 72 hours. It is adopted by majority
>>> approval, with at least 3 PMC affirmative votes.
>>>
>>> Thanks,
>>> JB
>>>
>>> [1]
>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12342847
>>> [2] https://dist.apache.org/repos/dist/dev/beam/2.5.0/
>>> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
>>> [4] https://repository.apache.org/content/repositories/orgapachebeam-1041/
>>> [5] https://github.com/apache/beam/tree/v2.5.0-RC1
>>> [6] https://github.com/apache/beam-site/pull/463
>>>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: [VOTE] Apache Beam, version 2.5.0, release candidate #1
Posted by Etienne Chauchot <ec...@apache.org>.
Hi,I've just run the nexmark queries on v2.5.0-RC1 tagWhat we can notice: - query 3 (exercises CoGroupByKey, state and
timer) shows different output with DR between batch and streaming and with the other runners => I compared with v2.4
there were still these differences but with different output size numbers
- query 6 (exercises specialized combiner) shows different output between the runners => the correct output is 401.
strange that in batch mode some runners output les Sellers. I compared with v2.4 same output
- response time of query 7 (exercices Max transform, fanout and side input) is very slow on DR => I compared with v2.4
, comparable execution times
I'm not comparing q10 because it is a write to GCS so it is very specific.
=> Basically no regression comparing to v2.4
For the record here is the output (waiting for ongoing perfkit integration):
1. DR batch
Performance: Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results (Baseline)
0000 5,8 17283,1 100000
0001 3,2 31104,2 92000
0002 1,2 82918,7 351
0003 2,2 46210,7 458
0004 1,2 8503,4 40
0005 4,0 25220,7 12
0006 0,9 11148,3 401
0007 13,2 7580,9 1
0008 1,5 67340,1 6000
0009 0,7 14025,2 298
0010 12,8 7793,0 1
0011 2,4 42319,1 1919
0012 1,6 61462,8 1919 ================================
==========================================================
2. DR streaming
Performance: Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results (Baseline)
0000 6,5 15285,8 100000
0001 3,7 27397,3 92000
0002 1,4 69108,5 351
0003 3,2 31181,8 447
0004 1,2 8361,2 40
0005 5,3 18903,6 12
0006 0,9 11111,1 401
0007 82,5 1212,2 1
0008 2,0 51072,5 6000
0009 0,8 12903,2 298
0010 49,5 2021,8 1
0011 3,9 25667,4 1919
0012 2,4 41067,8 1919 ================================
==========================================================
3. Flink batchPerformance: Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results (Baseline)
0000 1,0 97656,3 100000
0001 0,7 141643,1 92000
0002 0,4 228310,5 351
0003 1,6 64020,5 580
0004 0,7 13831,3 40
0005 1,4 72939,5 12
0006 0,5 20491,8 103
0007 1,3 74239,0 1
0008 0,8 121506,7 6000
0009 0,6 17953,3 298
0010 1,3 74682,6 1
0011 1,1 92936,8 1919
0012 0,8 123001,2 1919 ================================
==========================================================
4. Flink streamingPerformance:
Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results (Baseline)
0000 5,4 18677,6 100000
0001 2,8 35511,4 92000
0002 1,8 54318,3 351
0003 2,4 41614,6 580
0004 1,0 10341,3 40
0005 3,4 29568,3 12
0006 0,7 13369,0 401
0007 2,8 36192,5 1
0008 1,8 54854,6 6000
0009 0,7 13369,0 298
0010 3,4 29841,8 2
0011 5,0 19932,2 1919
0012 2,6 38835,0 1919 ================================
==========================================================
5. Spark batch Performance: Conf Runtime(sec) (Baseline) Events(/sec) (Baseline) Results (Baseline)
0000 1,5 65445,0 100000
0001 1,3 79491,3 92000
0002 0,9 112107,6 351
0003 2,0 48804,3 580
0004 1,2 8382,2 40
0005 2,0 50838,8 12
0006 1,0 9699,3 103
0007 2,3 43308,8 1
0008 2,1 46794,6 6000
0009 1,1 8976,7 298
0010 1,6 62111,8 1
0011 2,1 46598,3 1919
0012 2,3 43687,2 1919 ================================
==========================================================
Le mercredi 06 juin 2018 à 10:50 +0200, Etienne Chauchot a écrit :
> Thanks JB for all your work ! I believe doing the first gradle release must have been hard.I'll run Nexmark on the
> release and keep you posted.
> Best Etienne
>
> Le mercredi 06 juin 2018 à 10:44 +0200, Jean-Baptiste Onofré a écrit :
> > Hi everyone,
> > Please review and vote on the release candidate #1 for the version2.5.0, as follows:
> > [ ] +1, Approve the release[ ] -1, Do not approve the release (please provide specific comments)
> > NB: this is the first release using Gradle, so don't be too harsh ;) APR about the release guide will follow thanks
> > to this release.
> > The complete staging area is available for your review, which includes:* JIRA release notes [1],* the official
> > Apache source release to be deployed to dist.apache.org[2], which is signed with the key with fingerprint C8282E76
> > [3],* all artifacts to be deployed to the Maven Central Repository [4],* source code tag "v2.5.0-RC1" [5],* website
> > pull request listing the release and publishing the APIreference manual [6].* Java artifacts were built with Gradle
> > 4.7 (wrapper) and OpenJDK/OracleJDK 1.8.0_172 (Oracle Corporation 25.172-b11).* Python artifacts are deployed along
> > with the source release to thedist.apache.org [2].
> > The vote will be open for at least 72 hours. It is adopted by majorityapproval, with at least 3 PMC affirmative
> > votes.
> > Thanks,JB
> > [1]https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12342847[2] https://dist.apache
> > .org/repos/dist/dev/beam/2.5.0/[3] https://dist.apache.org/repos/dist/release/beam/KEYS[4] https://repository.apache
> > .org/content/repositories/orgapachebeam-1041/[5] https://github.com/apache/beam/tree/v2.5.0-RC1[6] https://github.co
> > m/apache/beam-site/pull/463
Re: [VOTE] Apache Beam, version 2.5.0, release candidate #1
Posted by Etienne Chauchot <ec...@apache.org>.
Thanks JB for all your work ! I believe doing the first gradle release must have been hard.
I'll run Nexmark on the release and keep you posted.
Best
Etienne
Le mercredi 06 juin 2018 à 10:44 +0200, Jean-Baptiste Onofré a écrit :
> Hi everyone,
>
> Please review and vote on the release candidate #1 for the version
> 2.5.0, as follows:
>
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> NB: this is the first release using Gradle, so don't be too harsh ;) A
> PR about the release guide will follow thanks to this release.
>
> The complete staging area is available for your review, which includes:
> * JIRA release notes [1],
> * the official Apache source release to be deployed to dist.apache.org
> [2], which is signed with the key with fingerprint C8282E76 [3],
> * all artifacts to be deployed to the Maven Central Repository [4],
> * source code tag "v2.5.0-RC1" [5],
> * website pull request listing the release and publishing the API
> reference manual [6].
> * Java artifacts were built with Gradle 4.7 (wrapper) and OpenJDK/Oracle
> JDK 1.8.0_172 (Oracle Corporation 25.172-b11).
> * Python artifacts are deployed along with the source release to the
> dist.apache.org [2].
>
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
>
> Thanks,
> JB
>
> [1]
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12342847
> [2] https://dist.apache.org/repos/dist/dev/beam/2.5.0/
> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
> [4] https://repository.apache.org/content/repositories/orgapachebeam-1041/
> [5] https://github.com/apache/beam/tree/v2.5.0-RC1
> [6] https://github.com/apache/beam-site/pull/463
>
Re: [VOTE] Apache Beam, version 2.5.0, release candidate #1
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Thanks, I'm testing it as well.
Regards
JB
On 15/06/2018 10:25, Charles Chen wrote:
> Thank you and sorry for the delay. Been testing the fix the past few
> hours. This CP PR fixes the
> issue: https://github.com/apache/beam/pull/5658.
>
> On Thu, Jun 14, 2018 at 10:25 PM Jean-Baptiste Onofré <jb@nanthrax.net
> <ma...@nanthrax.net>> wrote:
>
> OK, I started the RC2, but I'm stopping the process to cut a new one.
>
> Is it ok from your side ?
>
> Regards
> JB
>
> On 15/06/2018 01:54, Charles Chen wrote:
> > Looks like there is something wrong with PR 5636
> > <https://github.com/apache/beam/pull/5636> which we cherry-picked
> > above. It breaks leaderboard examples which previously passed. I've
> > reopened the issue and will update this thread shortly.
> >
> > On Thu, Jun 14, 2018 at 12:55 PM Jean-Baptiste Onofré
> <jb@nanthrax.net <ma...@nanthrax.net>
> > <mailto:jb@nanthrax.net <ma...@nanthrax.net>>> wrote:
> >
> > Sure, just in time ;)
> >
> > Regards
> > JB
> >
> > On 14/06/2018 20:58, Charles Chen wrote:
> > > Can you also merge the
> CP https://github.com/apache/beam/pull/5636 for
> > > https://issues.apache.org/jira/browse/BEAM-4549?
> > >
> > > On Thu, Jun 14, 2018 at 6:52 AM Jean-Baptiste Onofré
> > <jb@nanthrax.net <ma...@nanthrax.net>
> <mailto:jb@nanthrax.net <ma...@nanthrax.net>>
> > > <mailto:jb@nanthrax.net <ma...@nanthrax.net>
> <mailto:jb@nanthrax.net <ma...@nanthrax.net>>>> wrote:
> > >
> > > FYI, I'm starting RC2 right now.
> > >
> > > Stay tuned !
> > >
> > > Regards
> > > JB
> > >
> > > On 06/06/2018 10:44, Jean-Baptiste Onofré wrote:
> > > > Hi everyone,
> > > >
> > > > Please review and vote on the release candidate #1 for the
> > version
> > > > 2.5.0, as follows:
> > > >
> > > > [ ] +1, Approve the release
> > > > [ ] -1, Do not approve the release (please provide
> specific
> > comments)
> > > >
> > > > NB: this is the first release using Gradle, so don't
> be too
> > harsh ;) A
> > > > PR about the release guide will follow thanks to this
> release.
> > > >
> > > > The complete staging area is available for your
> review, which
> > > includes:
> > > > * JIRA release notes [1],
> > > > * the official Apache source release to be deployed to
> > > dist.apache.org <http://dist.apache.org>
> <http://dist.apache.org> <http://dist.apache.org>
> > > > [2], which is signed with the key with fingerprint
> C8282E76 [3],
> > > > * all artifacts to be deployed to the Maven Central
> > Repository [4],
> > > > * source code tag "v2.5.0-RC1" [5],
> > > > * website pull request listing the release and publishing
> > the API
> > > > reference manual [6].
> > > > * Java artifacts were built with Gradle 4.7 (wrapper) and
> > > OpenJDK/Oracle
> > > > JDK 1.8.0_172 (Oracle Corporation 25.172-b11).
> > > > * Python artifacts are deployed along with the source
> > release to the
> > > > dist.apache.org <http://dist.apache.org>
> <http://dist.apache.org>
> > <http://dist.apache.org> [2].
> > > >
> > > > The vote will be open for at least 72 hours. It is adopted
> > by majority
> > > > approval, with at least 3 PMC affirmative votes.
> > > >
> > > > Thanks,
> > > > JB
> > > >
> > > > [1]
> > > >
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12342847
> > > > [2] https://dist.apache.org/repos/dist/dev/beam/2.5.0/
> > > > [3] https://dist.apache.org/repos/dist/release/beam/KEYS
> > > > [4]
> > >
> >
> https://repository.apache.org/content/repositories/orgapachebeam-1041/
> > > > [5] https://github.com/apache/beam/tree/v2.5.0-RC1
> > > > [6] https://github.com/apache/beam-site/pull/463
> > > >
> > >
> > > --
> > > Jean-Baptiste Onofré
> > > jbonofre@apache.org <ma...@apache.org>
> <mailto:jbonofre@apache.org <ma...@apache.org>>
> > <mailto:jbonofre@apache.org <ma...@apache.org>
> <mailto:jbonofre@apache.org <ma...@apache.org>>>
> > > http://blog.nanthrax.net
> > > Talend - http://www.talend.com
> > >
> >
> > --
> > Jean-Baptiste Onofré
> > jbonofre@apache.org <ma...@apache.org>
> <mailto:jbonofre@apache.org <ma...@apache.org>>
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
> >
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org <ma...@apache.org>
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: [VOTE] Apache Beam, version 2.5.0, release candidate #1
Posted by Charles Chen <cc...@google.com>.
Thank you and sorry for the delay. Been testing the fix the past few
hours. This CP PR fixes the issue: https://github.com/apache/beam/pull/5658
.
On Thu, Jun 14, 2018 at 10:25 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> OK, I started the RC2, but I'm stopping the process to cut a new one.
>
> Is it ok from your side ?
>
> Regards
> JB
>
> On 15/06/2018 01:54, Charles Chen wrote:
> > Looks like there is something wrong with PR 5636
> > <https://github.com/apache/beam/pull/5636> which we cherry-picked
> > above. It breaks leaderboard examples which previously passed. I've
> > reopened the issue and will update this thread shortly.
> >
> > On Thu, Jun 14, 2018 at 12:55 PM Jean-Baptiste Onofré <jb@nanthrax.net
> > <ma...@nanthrax.net>> wrote:
> >
> > Sure, just in time ;)
> >
> > Regards
> > JB
> >
> > On 14/06/2018 20:58, Charles Chen wrote:
> > > Can you also merge the CP https://github.com/apache/beam/pull/5636
> for
> > > https://issues.apache.org/jira/browse/BEAM-4549?
> > >
> > > On Thu, Jun 14, 2018 at 6:52 AM Jean-Baptiste Onofré
> > <jb@nanthrax.net <ma...@nanthrax.net>
> > > <mailto:jb@nanthrax.net <ma...@nanthrax.net>>> wrote:
> > >
> > > FYI, I'm starting RC2 right now.
> > >
> > > Stay tuned !
> > >
> > > Regards
> > > JB
> > >
> > > On 06/06/2018 10:44, Jean-Baptiste Onofré wrote:
> > > > Hi everyone,
> > > >
> > > > Please review and vote on the release candidate #1 for the
> > version
> > > > 2.5.0, as follows:
> > > >
> > > > [ ] +1, Approve the release
> > > > [ ] -1, Do not approve the release (please provide specific
> > comments)
> > > >
> > > > NB: this is the first release using Gradle, so don't be too
> > harsh ;) A
> > > > PR about the release guide will follow thanks to this
> release.
> > > >
> > > > The complete staging area is available for your review, which
> > > includes:
> > > > * JIRA release notes [1],
> > > > * the official Apache source release to be deployed to
> > > dist.apache.org <http://dist.apache.org> <
> http://dist.apache.org>
> > > > [2], which is signed with the key with fingerprint C8282E76
> [3],
> > > > * all artifacts to be deployed to the Maven Central
> > Repository [4],
> > > > * source code tag "v2.5.0-RC1" [5],
> > > > * website pull request listing the release and publishing
> > the API
> > > > reference manual [6].
> > > > * Java artifacts were built with Gradle 4.7 (wrapper) and
> > > OpenJDK/Oracle
> > > > JDK 1.8.0_172 (Oracle Corporation 25.172-b11).
> > > > * Python artifacts are deployed along with the source
> > release to the
> > > > dist.apache.org <http://dist.apache.org>
> > <http://dist.apache.org> [2].
> > > >
> > > > The vote will be open for at least 72 hours. It is adopted
> > by majority
> > > > approval, with at least 3 PMC affirmative votes.
> > > >
> > > > Thanks,
> > > > JB
> > > >
> > > > [1]
> > > >
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12342847
> > > > [2] https://dist.apache.org/repos/dist/dev/beam/2.5.0/
> > > > [3] https://dist.apache.org/repos/dist/release/beam/KEYS
> > > > [4]
> > >
> >
> https://repository.apache.org/content/repositories/orgapachebeam-1041/
> > > > [5] https://github.com/apache/beam/tree/v2.5.0-RC1
> > > > [6] https://github.com/apache/beam-site/pull/463
> > > >
> > >
> > > --
> > > Jean-Baptiste Onofré
> > > jbonofre@apache.org <ma...@apache.org>
> > <mailto:jbonofre@apache.org <ma...@apache.org>>
> > > http://blog.nanthrax.net
> > > Talend - http://www.talend.com
> > >
> >
> > --
> > Jean-Baptiste Onofré
> > jbonofre@apache.org <ma...@apache.org>
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
> >
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
Re: [VOTE] Apache Beam, version 2.5.0, release candidate #1
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
OK, I started the RC2, but I'm stopping the process to cut a new one.
Is it ok from your side ?
Regards
JB
On 15/06/2018 01:54, Charles Chen wrote:
> Looks like there is something wrong with PR 5636
> <https://github.com/apache/beam/pull/5636> which we cherry-picked
> above. It breaks leaderboard examples which previously passed. I've
> reopened the issue and will update this thread shortly.
>
> On Thu, Jun 14, 2018 at 12:55 PM Jean-Baptiste Onofré <jb@nanthrax.net
> <ma...@nanthrax.net>> wrote:
>
> Sure, just in time ;)
>
> Regards
> JB
>
> On 14/06/2018 20:58, Charles Chen wrote:
> > Can you also merge the CP https://github.com/apache/beam/pull/5636 for
> > https://issues.apache.org/jira/browse/BEAM-4549?
> >
> > On Thu, Jun 14, 2018 at 6:52 AM Jean-Baptiste Onofré
> <jb@nanthrax.net <ma...@nanthrax.net>
> > <mailto:jb@nanthrax.net <ma...@nanthrax.net>>> wrote:
> >
> > FYI, I'm starting RC2 right now.
> >
> > Stay tuned !
> >
> > Regards
> > JB
> >
> > On 06/06/2018 10:44, Jean-Baptiste Onofré wrote:
> > > Hi everyone,
> > >
> > > Please review and vote on the release candidate #1 for the
> version
> > > 2.5.0, as follows:
> > >
> > > [ ] +1, Approve the release
> > > [ ] -1, Do not approve the release (please provide specific
> comments)
> > >
> > > NB: this is the first release using Gradle, so don't be too
> harsh ;) A
> > > PR about the release guide will follow thanks to this release.
> > >
> > > The complete staging area is available for your review, which
> > includes:
> > > * JIRA release notes [1],
> > > * the official Apache source release to be deployed to
> > dist.apache.org <http://dist.apache.org> <http://dist.apache.org>
> > > [2], which is signed with the key with fingerprint C8282E76 [3],
> > > * all artifacts to be deployed to the Maven Central
> Repository [4],
> > > * source code tag "v2.5.0-RC1" [5],
> > > * website pull request listing the release and publishing
> the API
> > > reference manual [6].
> > > * Java artifacts were built with Gradle 4.7 (wrapper) and
> > OpenJDK/Oracle
> > > JDK 1.8.0_172 (Oracle Corporation 25.172-b11).
> > > * Python artifacts are deployed along with the source
> release to the
> > > dist.apache.org <http://dist.apache.org>
> <http://dist.apache.org> [2].
> > >
> > > The vote will be open for at least 72 hours. It is adopted
> by majority
> > > approval, with at least 3 PMC affirmative votes.
> > >
> > > Thanks,
> > > JB
> > >
> > > [1]
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12342847
> > > [2] https://dist.apache.org/repos/dist/dev/beam/2.5.0/
> > > [3] https://dist.apache.org/repos/dist/release/beam/KEYS
> > > [4]
> >
> https://repository.apache.org/content/repositories/orgapachebeam-1041/
> > > [5] https://github.com/apache/beam/tree/v2.5.0-RC1
> > > [6] https://github.com/apache/beam-site/pull/463
> > >
> >
> > --
> > Jean-Baptiste Onofré
> > jbonofre@apache.org <ma...@apache.org>
> <mailto:jbonofre@apache.org <ma...@apache.org>>
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
> >
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org <ma...@apache.org>
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: [VOTE] Apache Beam, version 2.5.0, release candidate #1
Posted by Charles Chen <cc...@google.com>.
Looks like there is something wrong with PR 5636
<https://github.com/apache/beam/pull/5636> which we cherry-picked above.
It breaks leaderboard examples which previously passed. I've reopened the
issue and will update this thread shortly.
On Thu, Jun 14, 2018 at 12:55 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> Sure, just in time ;)
>
> Regards
> JB
>
> On 14/06/2018 20:58, Charles Chen wrote:
> > Can you also merge the CP https://github.com/apache/beam/pull/5636 for
> > https://issues.apache.org/jira/browse/BEAM-4549?
> >
> > On Thu, Jun 14, 2018 at 6:52 AM Jean-Baptiste Onofré <jb@nanthrax.net
> > <ma...@nanthrax.net>> wrote:
> >
> > FYI, I'm starting RC2 right now.
> >
> > Stay tuned !
> >
> > Regards
> > JB
> >
> > On 06/06/2018 10:44, Jean-Baptiste Onofré wrote:
> > > Hi everyone,
> > >
> > > Please review and vote on the release candidate #1 for the version
> > > 2.5.0, as follows:
> > >
> > > [ ] +1, Approve the release
> > > [ ] -1, Do not approve the release (please provide specific
> comments)
> > >
> > > NB: this is the first release using Gradle, so don't be too harsh
> ;) A
> > > PR about the release guide will follow thanks to this release.
> > >
> > > The complete staging area is available for your review, which
> > includes:
> > > * JIRA release notes [1],
> > > * the official Apache source release to be deployed to
> > dist.apache.org <http://dist.apache.org>
> > > [2], which is signed with the key with fingerprint C8282E76 [3],
> > > * all artifacts to be deployed to the Maven Central Repository [4],
> > > * source code tag "v2.5.0-RC1" [5],
> > > * website pull request listing the release and publishing the API
> > > reference manual [6].
> > > * Java artifacts were built with Gradle 4.7 (wrapper) and
> > OpenJDK/Oracle
> > > JDK 1.8.0_172 (Oracle Corporation 25.172-b11).
> > > * Python artifacts are deployed along with the source release to
> the
> > > dist.apache.org <http://dist.apache.org> [2].
> > >
> > > The vote will be open for at least 72 hours. It is adopted by
> majority
> > > approval, with at least 3 PMC affirmative votes.
> > >
> > > Thanks,
> > > JB
> > >
> > > [1]
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12342847
> > > [2] https://dist.apache.org/repos/dist/dev/beam/2.5.0/
> > > [3] https://dist.apache.org/repos/dist/release/beam/KEYS
> > > [4]
> >
> https://repository.apache.org/content/repositories/orgapachebeam-1041/
> > > [5] https://github.com/apache/beam/tree/v2.5.0-RC1
> > > [6] https://github.com/apache/beam-site/pull/463
> > >
> >
> > --
> > Jean-Baptiste Onofré
> > jbonofre@apache.org <ma...@apache.org>
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
> >
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
Re: [VOTE] Apache Beam, version 2.5.0, release candidate #1
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Sure, just in time ;)
Regards
JB
On 14/06/2018 20:58, Charles Chen wrote:
> Can you also merge the CP https://github.com/apache/beam/pull/5636 for
> https://issues.apache.org/jira/browse/BEAM-4549?
>
> On Thu, Jun 14, 2018 at 6:52 AM Jean-Baptiste Onofré <jb@nanthrax.net
> <ma...@nanthrax.net>> wrote:
>
> FYI, I'm starting RC2 right now.
>
> Stay tuned !
>
> Regards
> JB
>
> On 06/06/2018 10:44, Jean-Baptiste Onofré wrote:
> > Hi everyone,
> >
> > Please review and vote on the release candidate #1 for the version
> > 2.5.0, as follows:
> >
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release (please provide specific comments)
> >
> > NB: this is the first release using Gradle, so don't be too harsh ;) A
> > PR about the release guide will follow thanks to this release.
> >
> > The complete staging area is available for your review, which
> includes:
> > * JIRA release notes [1],
> > * the official Apache source release to be deployed to
> dist.apache.org <http://dist.apache.org>
> > [2], which is signed with the key with fingerprint C8282E76 [3],
> > * all artifacts to be deployed to the Maven Central Repository [4],
> > * source code tag "v2.5.0-RC1" [5],
> > * website pull request listing the release and publishing the API
> > reference manual [6].
> > * Java artifacts were built with Gradle 4.7 (wrapper) and
> OpenJDK/Oracle
> > JDK 1.8.0_172 (Oracle Corporation 25.172-b11).
> > * Python artifacts are deployed along with the source release to the
> > dist.apache.org <http://dist.apache.org> [2].
> >
> > The vote will be open for at least 72 hours. It is adopted by majority
> > approval, with at least 3 PMC affirmative votes.
> >
> > Thanks,
> > JB
> >
> > [1]
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12342847
> > [2] https://dist.apache.org/repos/dist/dev/beam/2.5.0/
> > [3] https://dist.apache.org/repos/dist/release/beam/KEYS
> > [4]
> https://repository.apache.org/content/repositories/orgapachebeam-1041/
> > [5] https://github.com/apache/beam/tree/v2.5.0-RC1
> > [6] https://github.com/apache/beam-site/pull/463
> >
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org <ma...@apache.org>
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: [VOTE] Apache Beam, version 2.5.0, release candidate #1
Posted by Charles Chen <cc...@google.com>.
Can you also merge the CP https://github.com/apache/beam/pull/5636 for
https://issues.apache.org/jira/browse/BEAM-4549?
On Thu, Jun 14, 2018 at 6:52 AM Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> FYI, I'm starting RC2 right now.
>
> Stay tuned !
>
> Regards
> JB
>
> On 06/06/2018 10:44, Jean-Baptiste Onofré wrote:
> > Hi everyone,
> >
> > Please review and vote on the release candidate #1 for the version
> > 2.5.0, as follows:
> >
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release (please provide specific comments)
> >
> > NB: this is the first release using Gradle, so don't be too harsh ;) A
> > PR about the release guide will follow thanks to this release.
> >
> > The complete staging area is available for your review, which includes:
> > * JIRA release notes [1],
> > * the official Apache source release to be deployed to dist.apache.org
> > [2], which is signed with the key with fingerprint C8282E76 [3],
> > * all artifacts to be deployed to the Maven Central Repository [4],
> > * source code tag "v2.5.0-RC1" [5],
> > * website pull request listing the release and publishing the API
> > reference manual [6].
> > * Java artifacts were built with Gradle 4.7 (wrapper) and OpenJDK/Oracle
> > JDK 1.8.0_172 (Oracle Corporation 25.172-b11).
> > * Python artifacts are deployed along with the source release to the
> > dist.apache.org [2].
> >
> > The vote will be open for at least 72 hours. It is adopted by majority
> > approval, with at least 3 PMC affirmative votes.
> >
> > Thanks,
> > JB
> >
> > [1]
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12342847
> > [2] https://dist.apache.org/repos/dist/dev/beam/2.5.0/
> > [3] https://dist.apache.org/repos/dist/release/beam/KEYS
> > [4]
> https://repository.apache.org/content/repositories/orgapachebeam-1041/
> > [5] https://github.com/apache/beam/tree/v2.5.0-RC1
> > [6] https://github.com/apache/beam-site/pull/463
> >
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
Re: [VOTE] Apache Beam, version 2.5.0, release candidate #1
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
FYI, I'm starting RC2 right now.
Stay tuned !
Regards
JB
On 06/06/2018 10:44, Jean-Baptiste Onofré wrote:
> Hi everyone,
>
> Please review and vote on the release candidate #1 for the version
> 2.5.0, as follows:
>
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> NB: this is the first release using Gradle, so don't be too harsh ;) A
> PR about the release guide will follow thanks to this release.
>
> The complete staging area is available for your review, which includes:
> * JIRA release notes [1],
> * the official Apache source release to be deployed to dist.apache.org
> [2], which is signed with the key with fingerprint C8282E76 [3],
> * all artifacts to be deployed to the Maven Central Repository [4],
> * source code tag "v2.5.0-RC1" [5],
> * website pull request listing the release and publishing the API
> reference manual [6].
> * Java artifacts were built with Gradle 4.7 (wrapper) and OpenJDK/Oracle
> JDK 1.8.0_172 (Oracle Corporation 25.172-b11).
> * Python artifacts are deployed along with the source release to the
> dist.apache.org [2].
>
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
>
> Thanks,
> JB
>
> [1]
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12319527&version=12342847
> [2] https://dist.apache.org/repos/dist/dev/beam/2.5.0/
> [3] https://dist.apache.org/repos/dist/release/beam/KEYS
> [4] https://repository.apache.org/content/repositories/orgapachebeam-1041/
> [5] https://github.com/apache/beam/tree/v2.5.0-RC1
> [6] https://github.com/apache/beam-site/pull/463
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com