You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Scott Wegner <sw...@google.com> on 2018/04/11 17:25:19 UTC
Gradle Status [April 11]
Thanks everyone for the continued effort towards the Gradle migration. As a
high-level summary of our progress since Friday: we have a viable build,
with a number of minor issues that we're still working out. Please take a
look at the new documentation in our contribution guide and log any bugs
that you find.
Here's a more detailed view of improvements from just the past few days..
Release artifacts:
* Pom.xml generation logic now in master [1]
* Nightly snapshots are now produced using Gradle [2]
* Excluded modules propagated to dependencies when generating * pom.xml
* Artifact JARs are properly shaded [3]
* Working on fixing dependency scopes in generated pom [4]
PreCommits / Postcommits:
* All PreCommits and PostCommits migrated [5]; working on deflaking [6] [7]
[8] [9]
* Jenkins results now include JUnit test results [10] and build scan for
easier debugging [11]
* Spark ValidatesRunner PostCommit passes [12] [13]
* Flink ValidatesRunner PostCommit more reliable [14]
Documentation / IDE Setup:
* Contribution Guide [15] is now updated with Gradle commands [16] [17]
Performance Benchmarks:
* Working on getting Nexmark benchmarks migrated [18]
If I missed anything, please add it to this thread.
We are continuing to use this general roadmap:
(a) Publish release artifacts with Gradle (SNAPSHOT and signed releases)
(b) Postcommits migrated to Gradle
(c) Migrate documentation from maven to Gradle
(d) Migrate perfkit suites to use Gradle
Migration tasks are tracked as subtasks on BEAM-3249 [19]. Kenn has created
a separate issue to track post-migration cleanup items: BEAM-4045 [20].
Feel free to grab any unassigned items off of either list.
[1] https://github.com/apache/beam/pull/5054
[2] https://github.com/apache/beam/pull/5057
[3] https://github.com/apache/beam/pull/5087
[4] https://github.com/apache/beam/pull/5098
[5] https://github.com/apache/beam/pull/5047
[6] https://github.com/apache/beam/pull/5088
[7] https://github.com/apache/beam/pull/5086
[8] https://github.com/apache/beam/pull/5066
[9] https://github.com/apache/beam/pull/5059
[10] https://github.com/apache/beam/pull/5045
[11] https://github.com/apache/beam/pull/5091
[12] https://github.com/apache/beam/pull/5093
[13] https://github.com/apache/beam/pull/5069
[14] https://github.com/apache/beam/pull/5068
[15] https://beam.apache.org/contribute/contribution-guide/
[16] https://github.com/apache/beam-site/pull/412
[17] https://github.com/apache/beam-site/pull/414
[18] https://github.com/apache/beam/pull/5051
[19] https://issues.apache.org/jira/browse/BEAM-3249
[20] https://issues.apache.org/jira/browse/BEAM-4045
On Fri, Apr 6, 2018 at 9:32 AM Scott Wegner <sw...@google.com> wrote:
> I wanted to start a thread to summarize the current state of Gradle
> migration. We've made lots of good progress so far this week. Here's the
> status from what I can tell-- please add or correct anything I missed:
>
> * Release artifacts can be built and published for Snapshot and officlal
> releases [1]
> * Gradle-generated releases have been validated with the the Apache Beam
> archetype generation quickstart; still needs additional validation.
> * Generated release pom files have correct project metadata [2]
> * The python pre-commits are now working in Gradle [3]
> * Ismaël has started a collaborative doc of Gradle tips [4] as we all
> learn the new system-- please add your own. This will eventually feed into
> official documentation on the website.
> * Łukasz Gajowy is working on migrating performance testing framework [5]
> * Daniel is working on updating documentation to refer to Gradle instead
> of maven
>
> If I missed anything, please add it to this thread.
>
> The general roadmap we're working towards is:
> (a) Publish release artifacts with Gradle (SNAPSHOT and signed releases)
> (b) Postcommits migrated to Gradle
> (c) Migrate documentation from maven to Gradle
> (d) Migrate perfkit suites to use Gradle
>
> For those of you that are hacking: thanks for your help so far! Progress
> is being roughly tracked on the Kanban [6]; please make sure the issues
> assigned to you are up-to-date. Many of the changes are staged on
> lukecwik's local branch [7]; we'll work on merging them back soon.
>
>
> [1] https://github.com/lukecwik/incubator-beam/pull/7
> [2] https://github.com/lukecwik/incubator-beam/pull/3
> [3] https://github.com/apache/beam/pull/5032
> [4]
> https://docs.google.com/document/d/1wR56Jef3XIPwj4DFzQKznuGPM3JDfRDVkxzeDlbdVSQ/edit
> [5] https://github.com/apache/beam/pull/5003
> [6] https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=242
> [7] https://github.com/lukecwik/incubator-beam/tree/gradle
> --
>
>
> Got feedback? http://go/swegner-feedback
>
--
Got feedback? http://go/swegner-feedback
Re: Gradle Status [April 11]
Posted by Romain Manni-Bucau <rm...@gmail.com>.
just created one https://issues.apache.org/jira/browse/BEAM-4057
Romain Manni-Bucau
@rmannibucau | Blog | Old Blog | Github | LinkedIn | Book
2018-04-12 18:15 GMT+02:00 Ahmet Altay <al...@google.com>:
>> Found another blocker in current artifacts creations: there is not
> pom.xml and pom.properties in META-INF. This is used by tools +
> libraries + integrations so it is quite important to not break it
>
> Romain, is there a JIRA for this issues? If not could you create one please?
>
> On Thu, Apr 12, 2018 at 3:17 AM, Etienne Chauchot <ec...@apache.org>
> wrote:
>>
>> Nice !
>> thanks Kenn
>>
>> Le mercredi 11 avril 2018 à 18:21 +0000, Kenneth Knowles a écrit :
>>
>> Initial Nexmark+Gradle run is in, though a hiccup in the Spark runner +
>> Netty has been introduced since yesterday. Etienne mentioned he has worked
>> toward setting up periodic runs on all runners, so this should help get us
>> towards that. We'll probably prefer to build standalone fat jars for
>> selected runners and use those, which is pending unknown issues in the
>> shadow config leaving out required dependencies.
>>
>> Kenn
>>
>> On Wed, Apr 11, 2018 at 10:25 AM Scott Wegner <sw...@google.com> wrote:
>>
>> Thanks everyone for the continued effort towards the Gradle migration. As
>> a high-level summary of our progress since Friday: we have a viable build,
>> with a number of minor issues that we're still working out. Please take a
>> look at the new documentation in our contribution guide and log any bugs
>> that you find.
>>
>> Here's a more detailed view of improvements from just the past few days..
>>
>> Release artifacts:
>> * Pom.xml generation logic now in master [1]
>> * Nightly snapshots are now produced using Gradle [2]
>> * Excluded modules propagated to dependencies when generating * pom.xml
>> * Artifact JARs are properly shaded [3]
>> * Working on fixing dependency scopes in generated pom [4]
>> PreCommits / Postcommits:
>> * All PreCommits and PostCommits migrated [5]; working on deflaking [6]
>> [7] [8] [9]
>> * Jenkins results now include JUnit test results [10] and build scan for
>> easier debugging [11]
>> * Spark ValidatesRunner PostCommit passes [12] [13]
>> * Flink ValidatesRunner PostCommit more reliable [14]
>> Documentation / IDE Setup:
>> * Contribution Guide [15] is now updated with Gradle commands [16] [17]
>> Performance Benchmarks:
>> * Working on getting Nexmark benchmarks migrated [18]
>>
>> If I missed anything, please add it to this thread.
>>
>> We are continuing to use this general roadmap:
>> (a) Publish release artifacts with Gradle (SNAPSHOT and signed releases)
>> (b) Postcommits migrated to Gradle
>> (c) Migrate documentation from maven to Gradle
>> (d) Migrate perfkit suites to use Gradle
>>
>> Migration tasks are tracked as subtasks on BEAM-3249 [19]. Kenn has
>> created a separate issue to track post-migration cleanup items: BEAM-4045
>> [20]. Feel free to grab any unassigned items off of either list.
>>
>>
>> [1] https://github.com/apache/beam/pull/5054
>> [2] https://github.com/apache/beam/pull/5057
>> [3] https://github.com/apache/beam/pull/5087
>> [4] https://github.com/apache/beam/pull/5098
>> [5] https://github.com/apache/beam/pull/5047
>> [6] https://github.com/apache/beam/pull/5088
>> [7] https://github.com/apache/beam/pull/5086
>> [8] https://github.com/apache/beam/pull/5066
>> [9] https://github.com/apache/beam/pull/5059
>> [10] https://github.com/apache/beam/pull/5045
>> [11] https://github.com/apache/beam/pull/5091
>> [12] https://github.com/apache/beam/pull/5093
>> [13] https://github.com/apache/beam/pull/5069
>> [14] https://github.com/apache/beam/pull/5068
>> [15] https://beam.apache.org/contribute/contribution-guide/
>> [16] https://github.com/apache/beam-site/pull/412
>> [17] https://github.com/apache/beam-site/pull/414
>> [18] https://github.com/apache/beam/pull/5051
>> [19] https://issues.apache.org/jira/browse/BEAM-3249
>> [20] https://issues.apache.org/jira/browse/BEAM-4045
>>
>> On Fri, Apr 6, 2018 at 9:32 AM Scott Wegner <sw...@google.com> wrote:
>>
>> I wanted to start a thread to summarize the current state of Gradle
>> migration. We've made lots of good progress so far this week. Here's the
>> status from what I can tell-- please add or correct anything I missed:
>>
>> * Release artifacts can be built and published for Snapshot and officlal
>> releases [1]
>> * Gradle-generated releases have been validated with the the Apache Beam
>> archetype generation quickstart; still needs additional validation.
>> * Generated release pom files have correct project metadata [2]
>> * The python pre-commits are now working in Gradle [3]
>> * Ismaël has started a collaborative doc of Gradle tips [4] as we all
>> learn the new system-- please add your own. This will eventually feed into
>> official documentation on the website.
>> * Łukasz Gajowy is working on migrating performance testing framework [5]
>> * Daniel is working on updating documentation to refer to Gradle instead
>> of maven
>>
>> If I missed anything, please add it to this thread.
>>
>> The general roadmap we're working towards is:
>> (a) Publish release artifacts with Gradle (SNAPSHOT and signed releases)
>> (b) Postcommits migrated to Gradle
>> (c) Migrate documentation from maven to Gradle
>> (d) Migrate perfkit suites to use Gradle
>>
>> For those of you that are hacking: thanks for your help so far! Progress
>> is being roughly tracked on the Kanban [6]; please make sure the issues
>> assigned to you are up-to-date. Many of the changes are staged on lukecwik's
>> local branch [7]; we'll work on merging them back soon.
>>
>>
>> [1] https://github.com/lukecwik/incubator-beam/pull/7
>> [2] https://github.com/lukecwik/incubator-beam/pull/3
>> [3] https://github.com/apache/beam/pull/5032
>> [4]
>> https://docs.google.com/document/d/1wR56Jef3XIPwj4DFzQKznuGPM3JDfRDVkxzeDlbdVSQ/edit
>> [5] https://github.com/apache/beam/pull/5003
>> [6] https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=242
>> [7] https://github.com/lukecwik/incubator-beam/tree/gradle
>> --
>>
>>
>> Got feedback? http://go/swegner-feedback
>>
>> --
>>
>>
>> Got feedback? http://go/swegner-feedback
>>
>
Re: Gradle Status [April 11]
Posted by Ahmet Altay <al...@google.com>.
> Found another blocker in current artifacts creations: there is not
pom.xml and pom.properties in META-INF. This is used by tools +
libraries + integrations so it is quite important to not break it
Romain, is there a JIRA for this issues? If not could you create one please?
On Thu, Apr 12, 2018 at 3:17 AM, Etienne Chauchot <ec...@apache.org>
wrote:
> Nice !
> thanks Kenn
>
> Le mercredi 11 avril 2018 à 18:21 +0000, Kenneth Knowles a écrit :
>
> Initial Nexmark+Gradle run is in, though a hiccup in the Spark runner +
> Netty has been introduced since yesterday. Etienne mentioned he has worked
> toward setting up periodic runs on all runners, so this should help get us
> towards that. We'll probably prefer to build standalone fat jars for
> selected runners and use those, which is pending unknown issues in the
> shadow config leaving out required dependencies.
>
> Kenn
>
> On Wed, Apr 11, 2018 at 10:25 AM Scott Wegner <sw...@google.com> wrote:
>
> Thanks everyone for the continued effort towards the Gradle migration. As
> a high-level summary of our progress since Friday: we have a viable build,
> with a number of minor issues that we're still working out. Please take a
> look at the new documentation in our contribution guide and log any bugs
> that you find.
>
> Here's a more detailed view of improvements from just the past few days..
>
> Release artifacts:
> * Pom.xml generation logic now in master [1]
> * Nightly snapshots are now produced using Gradle [2]
> * Excluded modules propagated to dependencies when generating * pom.xml
> * Artifact JARs are properly shaded [3]
> * Working on fixing dependency scopes in generated pom [4]
> PreCommits / Postcommits:
> * All PreCommits and PostCommits migrated [5]; working on deflaking [6]
> [7] [8] [9]
> * Jenkins results now include JUnit test results [10] and build scan for
> easier debugging [11]
> * Spark ValidatesRunner PostCommit passes [12] [13]
> * Flink ValidatesRunner PostCommit more reliable [14]
> Documentation / IDE Setup:
> * Contribution Guide [15] is now updated with Gradle commands [16] [17]
> Performance Benchmarks:
> * Working on getting Nexmark benchmarks migrated [18]
>
> If I missed anything, please add it to this thread.
>
> We are continuing to use this general roadmap:
> (a) Publish release artifacts with Gradle (SNAPSHOT and signed releases)
> (b) Postcommits migrated to Gradle
> (c) Migrate documentation from maven to Gradle
> (d) Migrate perfkit suites to use Gradle
>
> Migration tasks are tracked as subtasks on BEAM-3249 [19]. Kenn has
> created a separate issue to track post-migration cleanup items:
> BEAM-4045 [20]. Feel free to grab any unassigned items off of either list.
>
>
> [1] https://github.com/apache/beam/pull/5054
> [2] https://github.com/apache/beam/pull/5057
> [3] https://github.com/apache/beam/pull/5087
> [4] https://github.com/apache/beam/pull/5098
> [5] https://github.com/apache/beam/pull/5047
> [6] https://github.com/apache/beam/pull/5088
> [7] https://github.com/apache/beam/pull/5086
> [8] https://github.com/apache/beam/pull/5066
> [9] https://github.com/apache/beam/pull/5059
> [10] https://github.com/apache/beam/pull/5045
> [11] https://github.com/apache/beam/pull/5091
> [12] https://github.com/apache/beam/pull/5093
> [13] https://github.com/apache/beam/pull/5069
> [14] https://github.com/apache/beam/pull/5068
> [15] https://beam.apache.org/contribute/contribution-guide/
> [16] https://github.com/apache/beam-site/pull/412
> [17] https://github.com/apache/beam-site/pull/414
> [18] https://github.com/apache/beam/pull/5051
> [19] https://issues.apache.org/jira/browse/BEAM-3249
> [20] https://issues.apache.org/jira/browse/BEAM-4045
>
> On Fri, Apr 6, 2018 at 9:32 AM Scott Wegner <sw...@google.com> wrote:
>
> I wanted to start a thread to summarize the current state of Gradle
> migration. We've made lots of good progress so far this week. Here's the
> status from what I can tell-- please add or correct anything I missed:
>
> * Release artifacts can be built and published for Snapshot and officlal
> releases [1]
> * Gradle-generated releases have been validated with the the Apache Beam
> archetype generation quickstart; still needs additional validation.
> * Generated release pom files have correct project metadata [2]
> * The python pre-commits are now working in Gradle [3]
> * Ismaël has started a collaborative doc of Gradle tips [4] as we all
> learn the new system-- please add your own. This will eventually feed into
> official documentation on the website.
> * Łukasz Gajowy is working on migrating performance testing framework [5]
> * Daniel is working on updating documentation to refer to Gradle instead
> of maven
>
> If I missed anything, please add it to this thread.
>
> The general roadmap we're working towards is:
> (a) Publish release artifacts with Gradle (SNAPSHOT and signed releases)
> (b) Postcommits migrated to Gradle
> (c) Migrate documentation from maven to Gradle
> (d) Migrate perfkit suites to use Gradle
>
> For those of you that are hacking: thanks for your help so far! Progress
> is being roughly tracked on the Kanban [6]; please make sure the issues
> assigned to you are up-to-date. Many of the changes are staged on
> lukecwik's local branch [7]; we'll work on merging them back soon.
>
>
> [1] https://github.com/lukecwik/incubator-beam/pull/7
> [2] https://github.com/lukecwik/incubator-beam/pull/3
> [3] https://github.com/apache/beam/pull/5032
> [4] https://docs.google.com/document/d/1wR56Jef3XIPwj4DFzQKznuGPM3JDf
> RDVkxzeDlbdVSQ/edit
> [5] https://github.com/apache/beam/pull/5003
> [6] https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=242
> [7] https://github.com/lukecwik/incubator-beam/tree/gradle
> --
>
>
> Got feedback? http://go/swegner-feedback
>
> --
>
>
> Got feedback? http://go/swegner-feedback
>
>
Re: Gradle Status [April 11]
Posted by Etienne Chauchot <ec...@apache.org>.
Nice !
thanks Kenn
Le mercredi 11 avril 2018 à 18:21 +0000, Kenneth Knowles a écrit :
> Initial Nexmark+Gradle run is in, though a hiccup in the Spark runner + Netty has been introduced since yesterday.
> Etienne mentioned he has worked toward setting up periodic runs on all runners, so this should help get us towards
> that. We'll probably prefer to build standalone fat jars for selected runners and use those, which is pending unknown
> issues in the shadow config leaving out required dependencies.
>
> Kenn
>
> On Wed, Apr 11, 2018 at 10:25 AM Scott Wegner <sw...@google.com> wrote:
> > Thanks everyone for the continued effort towards the Gradle migration. As a high-level summary of our progress since
> > Friday: we have a viable build, with a number of minor issues that we're still working out. Please take a look at
> > the new documentation in our contribution guide and log any bugs that you find.
> >
> > Here's a more detailed view of improvements from just the past few days..
> >
> > Release artifacts:
> > * Pom.xml generation logic now in master [1]
> > * Nightly snapshots are now produced using Gradle [2]
> > * Excluded modules propagated to dependencies when generating * pom.xml
> > * Artifact JARs are properly shaded [3]
> > * Working on fixing dependency scopes in generated pom [4]
> > PreCommits / Postcommits:
> > * All PreCommits and PostCommits migrated [5]; working on deflaking [6] [7] [8] [9]
> > * Jenkins results now include JUnit test results [10] and build scan for easier debugging [11]
> > * Spark ValidatesRunner PostCommit passes [12] [13]
> > * Flink ValidatesRunner PostCommit more reliable [14]
> > Documentation / IDE Setup:
> > * Contribution Guide [15] is now updated with Gradle commands [16] [17]
> > Performance Benchmarks:
> > * Working on getting Nexmark benchmarks migrated [18]
> >
> > If I missed anything, please add it to this thread.
> >
> > We are continuing to use this general roadmap:
> > (a) Publish release artifacts with Gradle (SNAPSHOT and signed releases)
> > (b) Postcommits migrated to Gradle
> > (c) Migrate documentation from maven to Gradle
> > (d) Migrate perfkit suites to use Gradle
> >
> > Migration tasks are tracked as subtasks on BEAM-3249 [19]. Kenn has created a separate issue to track post-migration
> > cleanup items: BEAM-4045 [20]. Feel free to grab any unassigned items off of either list.
> >
> >
> > [1] https://github.com/apache/beam/pull/5054
> > [2] https://github.com/apache/beam/pull/5057
> > [3] https://github.com/apache/beam/pull/5087
> > [4] https://github.com/apache/beam/pull/5098
> > [5] https://github.com/apache/beam/pull/5047
> > [6] https://github.com/apache/beam/pull/5088
> > [7] https://github.com/apache/beam/pull/5086
> > [8] https://github.com/apache/beam/pull/5066
> > [9] https://github.com/apache/beam/pull/5059
> > [10] https://github.com/apache/beam/pull/5045
> > [11] https://github.com/apache/beam/pull/5091
> > [12] https://github.com/apache/beam/pull/5093
> > [13] https://github.com/apache/beam/pull/5069
> > [14] https://github.com/apache/beam/pull/5068
> > [15] https://beam.apache.org/contribute/contribution-guide/
> > [16] https://github.com/apache/beam-site/pull/412
> > [17] https://github.com/apache/beam-site/pull/414
> > [18] https://github.com/apache/beam/pull/5051
> > [19] https://issues.apache.org/jira/browse/BEAM-3249
> > [20] https://issues.apache.org/jira/browse/BEAM-4045
> >
> > On Fri, Apr 6, 2018 at 9:32 AM Scott Wegner <sw...@google.com> wrote:
> > > I wanted to start a thread to summarize the current state of Gradle migration. We've made lots of good progress so
> > > far this week. Here's the status from what I can tell-- please add or correct anything I missed:
> > >
> > > * Release artifacts can be built and published for Snapshot and officlal releases [1]
> > > * Gradle-generated releases have been validated with the the Apache Beam archetype generation quickstart; still
> > > needs additional validation.
> > > * Generated release pom files have correct project metadata [2]
> > > * The python pre-commits are now working in Gradle [3]
> > > * Ismaël has started a collaborative doc of Gradle tips [4] as we all learn the new system-- please add your own.
> > > This will eventually feed into official documentation on the website.
> > > * Łukasz Gajowy is working on migrating performance testing framework [5]
> > > * Daniel is working on updating documentation to refer to Gradle instead of maven
> > >
> > > If I missed anything, please add it to this thread.
> > >
> > > The general roadmap we're working towards is:
> > > (a) Publish release artifacts with Gradle (SNAPSHOT and signed releases)
> > > (b) Postcommits migrated to Gradle
> > > (c) Migrate documentation from maven to Gradle
> > > (d) Migrate perfkit suites to use Gradle
> > >
> > > For those of you that are hacking: thanks for your help so far! Progress is being roughly tracked on the Kanban
> > > [6]; please make sure the issues assigned to you are up-to-date. Many of the changes are staged on lukecwik's
> > > local branch [7]; we'll work on merging them back soon.
> > >
> > >
> > > [1] https://github.com/lukecwik/incubator-beam/pull/7
> > > [2] https://github.com/lukecwik/incubator-beam/pull/3
> > > [3] https://github.com/apache/beam/pull/5032
> > > [4] https://docs.google.com/document/d/1wR56Jef3XIPwj4DFzQKznuGPM3JDfRDVkxzeDlbdVSQ/edit
> > > [5] https://github.com/apache/beam/pull/5003
> > > [6] https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=242
> > > [7] https://github.com/lukecwik/incubator-beam/tree/gradle
> > > --
> > >
> > >
> > > Got feedback? http://go/swegner-feedback
> > >
> > --
> >
> >
> > Got feedback? http://go/swegner-feedback
> >
Re: Gradle Status [April 11]
Posted by Romain Manni-Bucau <rm...@gmail.com>.
Found another blocker in current artifacts creations: there is not
pom.xml and pom.properties in META-INF. This is used by tools +
libraries + integrations so it is quite important to not break it.
Romain Manni-Bucau
@rmannibucau | Blog | Old Blog | Github | LinkedIn | Book
2018-04-12 0:59 GMT+02:00 Kenneth Knowles <kl...@google.com>:
> There are plenty of technical hurdles since Bazel is best suited to an
> isolated monorepo. We could probably have overcome those problems with
> effort. I got through a few of them in the short sprint I did. Ultimately it
> was a community-driven decision: more people (as in "more than just me" :-)
> jumped on board and got excited about building out the Gradle build.
>
> Kenn
>
> On Wed, Apr 11, 2018 at 3:17 PM Nathan Fisher <nf...@junctionbox.ca>
> wrote:
>>
>> Is there a document or similar outlining the decision behind Gradle over
>> Bazel?
>>
>> For purely intellectual curiosity I’m curious what tradeoffs/benefits were
>> considered when evaluating the two or if it was more a matter of
>> community/contributor familiarity.
>>
>> I found this thread which seems to imply issues around dependency
>> management;
>>
>>
>> https://lists.apache.org/thread.html/bba0a89f2561fb2f7150a8381c1eb3923fa46299f3b35ff1304d7c46@%3Cdev.beam.apache.org%3E
>>
>> On Wed, Apr 11, 2018 at 3:21 PM, Kenneth Knowles <kl...@google.com> wrote:
>>>
>>> Initial Nexmark+Gradle run is in, though a hiccup in the Spark runner +
>>> Netty has been introduced since yesterday. Etienne mentioned he has worked
>>> toward setting up periodic runs on all runners, so this should help get us
>>> towards that. We'll probably prefer to build standalone fat jars for
>>> selected runners and use those, which is pending unknown issues in the
>>> shadow config leaving out required dependencies.
>>>
>>> Kenn
>>>
>>> On Wed, Apr 11, 2018 at 10:25 AM Scott Wegner <sw...@google.com> wrote:
>>>>
>>>> Thanks everyone for the continued effort towards the Gradle migration.
>>>> As a high-level summary of our progress since Friday: we have a viable
>>>> build, with a number of minor issues that we're still working out. Please
>>>> take a look at the new documentation in our contribution guide and log any
>>>> bugs that you find.
>>>>
>>>> Here's a more detailed view of improvements from just the past few
>>>> days..
>>>>
>>>> Release artifacts:
>>>> * Pom.xml generation logic now in master [1]
>>>> * Nightly snapshots are now produced using Gradle [2]
>>>> * Excluded modules propagated to dependencies when generating * pom.xml
>>>> * Artifact JARs are properly shaded [3]
>>>> * Working on fixing dependency scopes in generated pom [4]
>>>> PreCommits / Postcommits:
>>>> * All PreCommits and PostCommits migrated [5]; working on deflaking [6]
>>>> [7] [8] [9]
>>>> * Jenkins results now include JUnit test results [10] and build scan for
>>>> easier debugging [11]
>>>> * Spark ValidatesRunner PostCommit passes [12] [13]
>>>> * Flink ValidatesRunner PostCommit more reliable [14]
>>>> Documentation / IDE Setup:
>>>> * Contribution Guide [15] is now updated with Gradle commands [16] [17]
>>>> Performance Benchmarks:
>>>> * Working on getting Nexmark benchmarks migrated [18]
>>>>
>>>> If I missed anything, please add it to this thread.
>>>>
>>>> We are continuing to use this general roadmap:
>>>> (a) Publish release artifacts with Gradle (SNAPSHOT and signed releases)
>>>> (b) Postcommits migrated to Gradle
>>>> (c) Migrate documentation from maven to Gradle
>>>> (d) Migrate perfkit suites to use Gradle
>>>>
>>>> Migration tasks are tracked as subtasks on BEAM-3249 [19]. Kenn has
>>>> created a separate issue to track post-migration cleanup items: BEAM-4045
>>>> [20]. Feel free to grab any unassigned items off of either list.
>>>>
>>>>
>>>> [1] https://github.com/apache/beam/pull/5054
>>>> [2] https://github.com/apache/beam/pull/5057
>>>> [3] https://github.com/apache/beam/pull/5087
>>>> [4] https://github.com/apache/beam/pull/5098
>>>> [5] https://github.com/apache/beam/pull/5047
>>>> [6] https://github.com/apache/beam/pull/5088
>>>> [7] https://github.com/apache/beam/pull/5086
>>>> [8] https://github.com/apache/beam/pull/5066
>>>> [9] https://github.com/apache/beam/pull/5059
>>>> [10] https://github.com/apache/beam/pull/5045
>>>> [11] https://github.com/apache/beam/pull/5091
>>>> [12] https://github.com/apache/beam/pull/5093
>>>> [13] https://github.com/apache/beam/pull/5069
>>>> [14] https://github.com/apache/beam/pull/5068
>>>> [15] https://beam.apache.org/contribute/contribution-guide/
>>>> [16] https://github.com/apache/beam-site/pull/412
>>>> [17] https://github.com/apache/beam-site/pull/414
>>>> [18] https://github.com/apache/beam/pull/5051
>>>> [19] https://issues.apache.org/jira/browse/BEAM-3249
>>>> [20] https://issues.apache.org/jira/browse/BEAM-4045
>>>>
>>>> On Fri, Apr 6, 2018 at 9:32 AM Scott Wegner <sw...@google.com> wrote:
>>>>>
>>>>> I wanted to start a thread to summarize the current state of Gradle
>>>>> migration. We've made lots of good progress so far this week. Here's the
>>>>> status from what I can tell-- please add or correct anything I missed:
>>>>>
>>>>> * Release artifacts can be built and published for Snapshot and
>>>>> officlal releases [1]
>>>>> * Gradle-generated releases have been validated with the the Apache
>>>>> Beam archetype generation quickstart; still needs additional validation.
>>>>> * Generated release pom files have correct project metadata [2]
>>>>> * The python pre-commits are now working in Gradle [3]
>>>>> * Ismaël has started a collaborative doc of Gradle tips [4] as we all
>>>>> learn the new system-- please add your own. This will eventually feed into
>>>>> official documentation on the website.
>>>>> * Łukasz Gajowy is working on migrating performance testing framework
>>>>> [5]
>>>>> * Daniel is working on updating documentation to refer to Gradle
>>>>> instead of maven
>>>>>
>>>>> If I missed anything, please add it to this thread.
>>>>>
>>>>> The general roadmap we're working towards is:
>>>>> (a) Publish release artifacts with Gradle (SNAPSHOT and signed
>>>>> releases)
>>>>> (b) Postcommits migrated to Gradle
>>>>> (c) Migrate documentation from maven to Gradle
>>>>> (d) Migrate perfkit suites to use Gradle
>>>>>
>>>>> For those of you that are hacking: thanks for your help so far!
>>>>> Progress is being roughly tracked on the Kanban [6]; please make sure the
>>>>> issues assigned to you are up-to-date. Many of the changes are staged on
>>>>> lukecwik's local branch [7]; we'll work on merging them back soon.
>>>>>
>>>>>
>>>>> [1] https://github.com/lukecwik/incubator-beam/pull/7
>>>>> [2] https://github.com/lukecwik/incubator-beam/pull/3
>>>>> [3] https://github.com/apache/beam/pull/5032
>>>>> [4]
>>>>> https://docs.google.com/document/d/1wR56Jef3XIPwj4DFzQKznuGPM3JDfRDVkxzeDlbdVSQ/edit
>>>>> [5] https://github.com/apache/beam/pull/5003
>>>>> [6] https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=242
>>>>> [7] https://github.com/lukecwik/incubator-beam/tree/gradle
>>>>> --
>>>>>
>>>>>
>>>>> Got feedback? http://go/swegner-feedback
>>>>
>>>> --
>>>>
>>>>
>>>> Got feedback? http://go/swegner-feedback
>>
>> --
>> - sent from my mobile
Re: Gradle Status [April 11]
Posted by Nathan Fisher <nf...@junctionbox.ca>.
Great thanks for that! Makes perfect sense.
On Wed, Apr 11, 2018 at 7:59 PM, Kenneth Knowles <kl...@google.com> wrote:
> There are plenty of technical hurdles since Bazel is best suited to an
> isolated monorepo. We could probably have overcome those problems with
> effort. I got through a few of them in the short sprint I did. Ultimately
> it was a community-driven decision: more people (as in "more than just me"
> :-) jumped on board and got excited about building out the Gradle build.
>
> Kenn
>
> On Wed, Apr 11, 2018 at 3:17 PM Nathan Fisher <nf...@junctionbox.ca>
> wrote:
>
>> Is there a document or similar outlining the decision behind Gradle over
>> Bazel?
>>
>> For purely intellectual curiosity I’m curious what tradeoffs/benefits
>> were considered when evaluating the two or if it was more a matter of
>> community/contributor familiarity.
>>
>> I found this thread which seems to imply issues around dependency
>> management;
>>
>>
>> https://lists.apache.org/thread.html/bba0a89f2561fb2f7150a8381c1eb3923fa46299f3b35ff1304d7c46@%3Cdev.beam.apache.org%3E
>>
>> On Wed, Apr 11, 2018 at 3:21 PM, Kenneth Knowles <kl...@google.com> wrote:
>>
>>> Initial Nexmark+Gradle run is in, though a hiccup in the Spark runner +
>>> Netty has been introduced since yesterday. Etienne mentioned he has worked
>>> toward setting up periodic runs on all runners, so this should help get us
>>> towards that. We'll probably prefer to build standalone fat jars for
>>> selected runners and use those, which is pending unknown issues in the
>>> shadow config leaving out required dependencies.
>>>
>>> Kenn
>>>
>>> On Wed, Apr 11, 2018 at 10:25 AM Scott Wegner <sw...@google.com>
>>> wrote:
>>>
>>>> Thanks everyone for the continued effort towards the Gradle migration.
>>>> As a high-level summary of our progress since Friday: we have a viable
>>>> build, with a number of minor issues that we're still working out. Please
>>>> take a look at the new documentation in our contribution guide and log any
>>>> bugs that you find.
>>>>
>>>> Here's a more detailed view of improvements from just the past few
>>>> days..
>>>>
>>>> Release artifacts:
>>>> * Pom.xml generation logic now in master [1]
>>>> * Nightly snapshots are now produced using Gradle [2]
>>>> * Excluded modules propagated to dependencies when generating * pom.xml
>>>> * Artifact JARs are properly shaded [3]
>>>> * Working on fixing dependency scopes in generated pom [4]
>>>> PreCommits / Postcommits:
>>>> * All PreCommits and PostCommits migrated [5]; working on deflaking [6]
>>>> [7] [8] [9]
>>>> * Jenkins results now include JUnit test results [10] and build scan
>>>> for easier debugging [11]
>>>> * Spark ValidatesRunner PostCommit passes [12] [13]
>>>> * Flink ValidatesRunner PostCommit more reliable [14]
>>>> Documentation / IDE Setup:
>>>> * Contribution Guide [15] is now updated with Gradle commands [16] [17]
>>>> Performance Benchmarks:
>>>> * Working on getting Nexmark benchmarks migrated [18]
>>>>
>>>> If I missed anything, please add it to this thread.
>>>>
>>>> We are continuing to use this general roadmap:
>>>> (a) Publish release artifacts with Gradle (SNAPSHOT and signed releases)
>>>> (b) Postcommits migrated to Gradle
>>>> (c) Migrate documentation from maven to Gradle
>>>> (d) Migrate perfkit suites to use Gradle
>>>>
>>>> Migration tasks are tracked as subtasks on BEAM-3249 [19]. Kenn has
>>>> created a separate issue to track post-migration cleanup items:
>>>> BEAM-4045 [20]. Feel free to grab any unassigned items off of either list.
>>>>
>>>>
>>>> [1] https://github.com/apache/beam/pull/5054
>>>> [2] https://github.com/apache/beam/pull/5057
>>>> [3] https://github.com/apache/beam/pull/5087
>>>> [4] https://github.com/apache/beam/pull/5098
>>>> [5] https://github.com/apache/beam/pull/5047
>>>> [6] https://github.com/apache/beam/pull/5088
>>>> [7] https://github.com/apache/beam/pull/5086
>>>> [8] https://github.com/apache/beam/pull/5066
>>>> [9] https://github.com/apache/beam/pull/5059
>>>> [10] https://github.com/apache/beam/pull/5045
>>>> [11] https://github.com/apache/beam/pull/5091
>>>> [12] https://github.com/apache/beam/pull/5093
>>>> [13] https://github.com/apache/beam/pull/5069
>>>> [14] https://github.com/apache/beam/pull/5068
>>>> [15] https://beam.apache.org/contribute/contribution-guide/
>>>> [16] https://github.com/apache/beam-site/pull/412
>>>> [17] https://github.com/apache/beam-site/pull/414
>>>> [18] https://github.com/apache/beam/pull/5051
>>>> [19] https://issues.apache.org/jira/browse/BEAM-3249
>>>> [20] https://issues.apache.org/jira/browse/BEAM-4045
>>>>
>>>> On Fri, Apr 6, 2018 at 9:32 AM Scott Wegner <sw...@google.com> wrote:
>>>>
>>>>> I wanted to start a thread to summarize the current state of Gradle
>>>>> migration. We've made lots of good progress so far this week. Here's the
>>>>> status from what I can tell-- please add or correct anything I missed:
>>>>>
>>>>> * Release artifacts can be built and published for Snapshot and
>>>>> officlal releases [1]
>>>>> * Gradle-generated releases have been validated with the the Apache
>>>>> Beam archetype generation quickstart; still needs additional validation.
>>>>> * Generated release pom files have correct project metadata [2]
>>>>> * The python pre-commits are now working in Gradle [3]
>>>>> * Ismaël has started a collaborative doc of Gradle tips [4] as we all
>>>>> learn the new system-- please add your own. This will eventually feed into
>>>>> official documentation on the website.
>>>>> * Łukasz Gajowy is working on migrating performance testing framework
>>>>> [5]
>>>>> * Daniel is working on updating documentation to refer to Gradle
>>>>> instead of maven
>>>>>
>>>>> If I missed anything, please add it to this thread.
>>>>>
>>>>> The general roadmap we're working towards is:
>>>>> (a) Publish release artifacts with Gradle (SNAPSHOT and signed
>>>>> releases)
>>>>> (b) Postcommits migrated to Gradle
>>>>> (c) Migrate documentation from maven to Gradle
>>>>> (d) Migrate perfkit suites to use Gradle
>>>>>
>>>>> For those of you that are hacking: thanks for your help so far!
>>>>> Progress is being roughly tracked on the Kanban [6]; please make sure the
>>>>> issues assigned to you are up-to-date. Many of the changes are staged on
>>>>> lukecwik's local branch [7]; we'll work on merging them back soon.
>>>>>
>>>>>
>>>>> [1] https://github.com/lukecwik/incubator-beam/pull/7
>>>>> [2] https://github.com/lukecwik/incubator-beam/pull/3
>>>>> [3] https://github.com/apache/beam/pull/5032
>>>>> [4]
>>>>> https://docs.google.com/document/d/1wR56Jef3XIPwj4DFzQKznuGPM3JDfRDVkxzeDlbdVSQ/edit
>>>>> [5] https://github.com/apache/beam/pull/5003
>>>>> [6]
>>>>> https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=242
>>>>> [7] https://github.com/lukecwik/incubator-beam/tree/gradle
>>>>> --
>>>>>
>>>>>
>>>>> Got feedback? http://go/swegner-feedback
>>>>>
>>>> --
>>>>
>>>>
>>>> Got feedback? http://go/swegner-feedback
>>>>
>>> --
>> - sent from my mobile
>>
> --
- sent from my mobile
Re: Gradle Status [April 11]
Posted by Kenneth Knowles <kl...@google.com>.
There are plenty of technical hurdles since Bazel is best suited to an
isolated monorepo. We could probably have overcome those problems with
effort. I got through a few of them in the short sprint I did. Ultimately
it was a community-driven decision: more people (as in "more than just me"
:-) jumped on board and got excited about building out the Gradle build.
Kenn
On Wed, Apr 11, 2018 at 3:17 PM Nathan Fisher <nf...@junctionbox.ca>
wrote:
> Is there a document or similar outlining the decision behind Gradle over
> Bazel?
>
> For purely intellectual curiosity I’m curious what tradeoffs/benefits were
> considered when evaluating the two or if it was more a matter of
> community/contributor familiarity.
>
> I found this thread which seems to imply issues around dependency
> management;
>
>
> https://lists.apache.org/thread.html/bba0a89f2561fb2f7150a8381c1eb3923fa46299f3b35ff1304d7c46@%3Cdev.beam.apache.org%3E
>
> On Wed, Apr 11, 2018 at 3:21 PM, Kenneth Knowles <kl...@google.com> wrote:
>
>> Initial Nexmark+Gradle run is in, though a hiccup in the Spark runner +
>> Netty has been introduced since yesterday. Etienne mentioned he has worked
>> toward setting up periodic runs on all runners, so this should help get us
>> towards that. We'll probably prefer to build standalone fat jars for
>> selected runners and use those, which is pending unknown issues in the
>> shadow config leaving out required dependencies.
>>
>> Kenn
>>
>> On Wed, Apr 11, 2018 at 10:25 AM Scott Wegner <sw...@google.com> wrote:
>>
>>> Thanks everyone for the continued effort towards the Gradle migration.
>>> As a high-level summary of our progress since Friday: we have a viable
>>> build, with a number of minor issues that we're still working out. Please
>>> take a look at the new documentation in our contribution guide and log any
>>> bugs that you find.
>>>
>>> Here's a more detailed view of improvements from just the past few days..
>>>
>>> Release artifacts:
>>> * Pom.xml generation logic now in master [1]
>>> * Nightly snapshots are now produced using Gradle [2]
>>> * Excluded modules propagated to dependencies when generating * pom.xml
>>> * Artifact JARs are properly shaded [3]
>>> * Working on fixing dependency scopes in generated pom [4]
>>> PreCommits / Postcommits:
>>> * All PreCommits and PostCommits migrated [5]; working on deflaking [6]
>>> [7] [8] [9]
>>> * Jenkins results now include JUnit test results [10] and build scan for
>>> easier debugging [11]
>>> * Spark ValidatesRunner PostCommit passes [12] [13]
>>> * Flink ValidatesRunner PostCommit more reliable [14]
>>> Documentation / IDE Setup:
>>> * Contribution Guide [15] is now updated with Gradle commands [16] [17]
>>> Performance Benchmarks:
>>> * Working on getting Nexmark benchmarks migrated [18]
>>>
>>> If I missed anything, please add it to this thread.
>>>
>>> We are continuing to use this general roadmap:
>>> (a) Publish release artifacts with Gradle (SNAPSHOT and signed releases)
>>> (b) Postcommits migrated to Gradle
>>> (c) Migrate documentation from maven to Gradle
>>> (d) Migrate perfkit suites to use Gradle
>>>
>>> Migration tasks are tracked as subtasks on BEAM-3249 [19]. Kenn has
>>> created a separate issue to track post-migration cleanup items:
>>> BEAM-4045 [20]. Feel free to grab any unassigned items off of either list.
>>>
>>>
>>> [1] https://github.com/apache/beam/pull/5054
>>> [2] https://github.com/apache/beam/pull/5057
>>> [3] https://github.com/apache/beam/pull/5087
>>> [4] https://github.com/apache/beam/pull/5098
>>> [5] https://github.com/apache/beam/pull/5047
>>> [6] https://github.com/apache/beam/pull/5088
>>> [7] https://github.com/apache/beam/pull/5086
>>> [8] https://github.com/apache/beam/pull/5066
>>> [9] https://github.com/apache/beam/pull/5059
>>> [10] https://github.com/apache/beam/pull/5045
>>> [11] https://github.com/apache/beam/pull/5091
>>> [12] https://github.com/apache/beam/pull/5093
>>> [13] https://github.com/apache/beam/pull/5069
>>> [14] https://github.com/apache/beam/pull/5068
>>> [15] https://beam.apache.org/contribute/contribution-guide/
>>> [16] https://github.com/apache/beam-site/pull/412
>>> [17] https://github.com/apache/beam-site/pull/414
>>> [18] https://github.com/apache/beam/pull/5051
>>> [19] https://issues.apache.org/jira/browse/BEAM-3249
>>> [20] https://issues.apache.org/jira/browse/BEAM-4045
>>>
>>> On Fri, Apr 6, 2018 at 9:32 AM Scott Wegner <sw...@google.com> wrote:
>>>
>>>> I wanted to start a thread to summarize the current state of Gradle
>>>> migration. We've made lots of good progress so far this week. Here's the
>>>> status from what I can tell-- please add or correct anything I missed:
>>>>
>>>> * Release artifacts can be built and published for Snapshot and
>>>> officlal releases [1]
>>>> * Gradle-generated releases have been validated with the the Apache
>>>> Beam archetype generation quickstart; still needs additional validation.
>>>> * Generated release pom files have correct project metadata [2]
>>>> * The python pre-commits are now working in Gradle [3]
>>>> * Ismaël has started a collaborative doc of Gradle tips [4] as we all
>>>> learn the new system-- please add your own. This will eventually feed into
>>>> official documentation on the website.
>>>> * Łukasz Gajowy is working on migrating performance testing framework
>>>> [5]
>>>> * Daniel is working on updating documentation to refer to Gradle
>>>> instead of maven
>>>>
>>>> If I missed anything, please add it to this thread.
>>>>
>>>> The general roadmap we're working towards is:
>>>> (a) Publish release artifacts with Gradle (SNAPSHOT and signed releases)
>>>> (b) Postcommits migrated to Gradle
>>>> (c) Migrate documentation from maven to Gradle
>>>> (d) Migrate perfkit suites to use Gradle
>>>>
>>>> For those of you that are hacking: thanks for your help so far!
>>>> Progress is being roughly tracked on the Kanban [6]; please make sure the
>>>> issues assigned to you are up-to-date. Many of the changes are staged on
>>>> lukecwik's local branch [7]; we'll work on merging them back soon.
>>>>
>>>>
>>>> [1] https://github.com/lukecwik/incubator-beam/pull/7
>>>> [2] https://github.com/lukecwik/incubator-beam/pull/3
>>>> [3] https://github.com/apache/beam/pull/5032
>>>> [4]
>>>> https://docs.google.com/document/d/1wR56Jef3XIPwj4DFzQKznuGPM3JDfRDVkxzeDlbdVSQ/edit
>>>> [5] https://github.com/apache/beam/pull/5003
>>>> [6] https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=242
>>>>
>>>> [7] https://github.com/lukecwik/incubator-beam/tree/gradle
>>>> --
>>>>
>>>>
>>>> Got feedback? http://go/swegner-feedback
>>>>
>>> --
>>>
>>>
>>> Got feedback? http://go/swegner-feedback
>>>
>> --
> - sent from my mobile
>
Re: Gradle Status [April 11]
Posted by Nathan Fisher <nf...@junctionbox.ca>.
Is there a document or similar outlining the decision behind Gradle over
Bazel?
For purely intellectual curiosity I’m curious what tradeoffs/benefits were
considered when evaluating the two or if it was more a matter of
community/contributor familiarity.
I found this thread which seems to imply issues around dependency
management;
https://lists.apache.org/thread.html/bba0a89f2561fb2f7150a8381c1eb3923fa46299f3b35ff1304d7c46@%3Cdev.beam.apache.org%3E
On Wed, Apr 11, 2018 at 3:21 PM, Kenneth Knowles <kl...@google.com> wrote:
> Initial Nexmark+Gradle run is in, though a hiccup in the Spark runner +
> Netty has been introduced since yesterday. Etienne mentioned he has worked
> toward setting up periodic runs on all runners, so this should help get us
> towards that. We'll probably prefer to build standalone fat jars for
> selected runners and use those, which is pending unknown issues in the
> shadow config leaving out required dependencies.
>
> Kenn
>
> On Wed, Apr 11, 2018 at 10:25 AM Scott Wegner <sw...@google.com> wrote:
>
>> Thanks everyone for the continued effort towards the Gradle migration. As
>> a high-level summary of our progress since Friday: we have a viable build,
>> with a number of minor issues that we're still working out. Please take a
>> look at the new documentation in our contribution guide and log any bugs
>> that you find.
>>
>> Here's a more detailed view of improvements from just the past few days..
>>
>> Release artifacts:
>> * Pom.xml generation logic now in master [1]
>> * Nightly snapshots are now produced using Gradle [2]
>> * Excluded modules propagated to dependencies when generating * pom.xml
>> * Artifact JARs are properly shaded [3]
>> * Working on fixing dependency scopes in generated pom [4]
>> PreCommits / Postcommits:
>> * All PreCommits and PostCommits migrated [5]; working on deflaking [6]
>> [7] [8] [9]
>> * Jenkins results now include JUnit test results [10] and build scan for
>> easier debugging [11]
>> * Spark ValidatesRunner PostCommit passes [12] [13]
>> * Flink ValidatesRunner PostCommit more reliable [14]
>> Documentation / IDE Setup:
>> * Contribution Guide [15] is now updated with Gradle commands [16] [17]
>> Performance Benchmarks:
>> * Working on getting Nexmark benchmarks migrated [18]
>>
>> If I missed anything, please add it to this thread.
>>
>> We are continuing to use this general roadmap:
>> (a) Publish release artifacts with Gradle (SNAPSHOT and signed releases)
>> (b) Postcommits migrated to Gradle
>> (c) Migrate documentation from maven to Gradle
>> (d) Migrate perfkit suites to use Gradle
>>
>> Migration tasks are tracked as subtasks on BEAM-3249 [19]. Kenn has
>> created a separate issue to track post-migration cleanup items:
>> BEAM-4045 [20]. Feel free to grab any unassigned items off of either list.
>>
>>
>> [1] https://github.com/apache/beam/pull/5054
>> [2] https://github.com/apache/beam/pull/5057
>> [3] https://github.com/apache/beam/pull/5087
>> [4] https://github.com/apache/beam/pull/5098
>> [5] https://github.com/apache/beam/pull/5047
>> [6] https://github.com/apache/beam/pull/5088
>> [7] https://github.com/apache/beam/pull/5086
>> [8] https://github.com/apache/beam/pull/5066
>> [9] https://github.com/apache/beam/pull/5059
>> [10] https://github.com/apache/beam/pull/5045
>> [11] https://github.com/apache/beam/pull/5091
>> [12] https://github.com/apache/beam/pull/5093
>> [13] https://github.com/apache/beam/pull/5069
>> [14] https://github.com/apache/beam/pull/5068
>> [15] https://beam.apache.org/contribute/contribution-guide/
>> [16] https://github.com/apache/beam-site/pull/412
>> [17] https://github.com/apache/beam-site/pull/414
>> [18] https://github.com/apache/beam/pull/5051
>> [19] https://issues.apache.org/jira/browse/BEAM-3249
>> [20] https://issues.apache.org/jira/browse/BEAM-4045
>>
>> On Fri, Apr 6, 2018 at 9:32 AM Scott Wegner <sw...@google.com> wrote:
>>
>>> I wanted to start a thread to summarize the current state of Gradle
>>> migration. We've made lots of good progress so far this week. Here's the
>>> status from what I can tell-- please add or correct anything I missed:
>>>
>>> * Release artifacts can be built and published for Snapshot and officlal
>>> releases [1]
>>> * Gradle-generated releases have been validated with the the Apache Beam
>>> archetype generation quickstart; still needs additional validation.
>>> * Generated release pom files have correct project metadata [2]
>>> * The python pre-commits are now working in Gradle [3]
>>> * Ismaël has started a collaborative doc of Gradle tips [4] as we all
>>> learn the new system-- please add your own. This will eventually feed into
>>> official documentation on the website.
>>> * Łukasz Gajowy is working on migrating performance testing framework [5]
>>> * Daniel is working on updating documentation to refer to Gradle instead
>>> of maven
>>>
>>> If I missed anything, please add it to this thread.
>>>
>>> The general roadmap we're working towards is:
>>> (a) Publish release artifacts with Gradle (SNAPSHOT and signed releases)
>>> (b) Postcommits migrated to Gradle
>>> (c) Migrate documentation from maven to Gradle
>>> (d) Migrate perfkit suites to use Gradle
>>>
>>> For those of you that are hacking: thanks for your help so far! Progress
>>> is being roughly tracked on the Kanban [6]; please make sure the issues
>>> assigned to you are up-to-date. Many of the changes are staged on
>>> lukecwik's local branch [7]; we'll work on merging them back soon.
>>>
>>>
>>> [1] https://github.com/lukecwik/incubator-beam/pull/7
>>> [2] https://github.com/lukecwik/incubator-beam/pull/3
>>> [3] https://github.com/apache/beam/pull/5032
>>> [4]
>>> https://docs.google.com/document/d/1wR56Jef3XIPwj4DFzQKznuGPM3JDfRDVkxzeDlbdVSQ/edit
>>> [5] https://github.com/apache/beam/pull/5003
>>> [6] https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=242
>>> [7] https://github.com/lukecwik/incubator-beam/tree/gradle
>>> --
>>>
>>>
>>> Got feedback? http://go/swegner-feedback
>>>
>> --
>>
>>
>> Got feedback? http://go/swegner-feedback
>>
> --
- sent from my mobile
Re: Gradle Status [April 11]
Posted by Kenneth Knowles <kl...@google.com>.
Initial Nexmark+Gradle run is in, though a hiccup in the Spark runner +
Netty has been introduced since yesterday. Etienne mentioned he has worked
toward setting up periodic runs on all runners, so this should help get us
towards that. We'll probably prefer to build standalone fat jars for
selected runners and use those, which is pending unknown issues in the
shadow config leaving out required dependencies.
Kenn
On Wed, Apr 11, 2018 at 10:25 AM Scott Wegner <sw...@google.com> wrote:
> Thanks everyone for the continued effort towards the Gradle migration. As
> a high-level summary of our progress since Friday: we have a viable build,
> with a number of minor issues that we're still working out. Please take a
> look at the new documentation in our contribution guide and log any bugs
> that you find.
>
> Here's a more detailed view of improvements from just the past few days..
>
> Release artifacts:
> * Pom.xml generation logic now in master [1]
> * Nightly snapshots are now produced using Gradle [2]
> * Excluded modules propagated to dependencies when generating * pom.xml
> * Artifact JARs are properly shaded [3]
> * Working on fixing dependency scopes in generated pom [4]
> PreCommits / Postcommits:
> * All PreCommits and PostCommits migrated [5]; working on deflaking [6]
> [7] [8] [9]
> * Jenkins results now include JUnit test results [10] and build scan for
> easier debugging [11]
> * Spark ValidatesRunner PostCommit passes [12] [13]
> * Flink ValidatesRunner PostCommit more reliable [14]
> Documentation / IDE Setup:
> * Contribution Guide [15] is now updated with Gradle commands [16] [17]
> Performance Benchmarks:
> * Working on getting Nexmark benchmarks migrated [18]
>
> If I missed anything, please add it to this thread.
>
> We are continuing to use this general roadmap:
> (a) Publish release artifacts with Gradle (SNAPSHOT and signed releases)
> (b) Postcommits migrated to Gradle
> (c) Migrate documentation from maven to Gradle
> (d) Migrate perfkit suites to use Gradle
>
> Migration tasks are tracked as subtasks on BEAM-3249 [19]. Kenn has
> created a separate issue to track post-migration cleanup items:
> BEAM-4045 [20]. Feel free to grab any unassigned items off of either list.
>
>
> [1] https://github.com/apache/beam/pull/5054
> [2] https://github.com/apache/beam/pull/5057
> [3] https://github.com/apache/beam/pull/5087
> [4] https://github.com/apache/beam/pull/5098
> [5] https://github.com/apache/beam/pull/5047
> [6] https://github.com/apache/beam/pull/5088
> [7] https://github.com/apache/beam/pull/5086
> [8] https://github.com/apache/beam/pull/5066
> [9] https://github.com/apache/beam/pull/5059
> [10] https://github.com/apache/beam/pull/5045
> [11] https://github.com/apache/beam/pull/5091
> [12] https://github.com/apache/beam/pull/5093
> [13] https://github.com/apache/beam/pull/5069
> [14] https://github.com/apache/beam/pull/5068
> [15] https://beam.apache.org/contribute/contribution-guide/
> [16] https://github.com/apache/beam-site/pull/412
> [17] https://github.com/apache/beam-site/pull/414
> [18] https://github.com/apache/beam/pull/5051
> [19] https://issues.apache.org/jira/browse/BEAM-3249
> [20] https://issues.apache.org/jira/browse/BEAM-4045
>
> On Fri, Apr 6, 2018 at 9:32 AM Scott Wegner <sw...@google.com> wrote:
>
>> I wanted to start a thread to summarize the current state of Gradle
>> migration. We've made lots of good progress so far this week. Here's the
>> status from what I can tell-- please add or correct anything I missed:
>>
>> * Release artifacts can be built and published for Snapshot and officlal
>> releases [1]
>> * Gradle-generated releases have been validated with the the Apache Beam
>> archetype generation quickstart; still needs additional validation.
>> * Generated release pom files have correct project metadata [2]
>> * The python pre-commits are now working in Gradle [3]
>> * Ismaël has started a collaborative doc of Gradle tips [4] as we all
>> learn the new system-- please add your own. This will eventually feed into
>> official documentation on the website.
>> * Łukasz Gajowy is working on migrating performance testing framework [5]
>> * Daniel is working on updating documentation to refer to Gradle instead
>> of maven
>>
>> If I missed anything, please add it to this thread.
>>
>> The general roadmap we're working towards is:
>> (a) Publish release artifacts with Gradle (SNAPSHOT and signed releases)
>> (b) Postcommits migrated to Gradle
>> (c) Migrate documentation from maven to Gradle
>> (d) Migrate perfkit suites to use Gradle
>>
>> For those of you that are hacking: thanks for your help so far! Progress
>> is being roughly tracked on the Kanban [6]; please make sure the issues
>> assigned to you are up-to-date. Many of the changes are staged on
>> lukecwik's local branch [7]; we'll work on merging them back soon.
>>
>>
>> [1] https://github.com/lukecwik/incubator-beam/pull/7
>> [2] https://github.com/lukecwik/incubator-beam/pull/3
>> [3] https://github.com/apache/beam/pull/5032
>> [4]
>> https://docs.google.com/document/d/1wR56Jef3XIPwj4DFzQKznuGPM3JDfRDVkxzeDlbdVSQ/edit
>> [5] https://github.com/apache/beam/pull/5003
>> [6] https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=242
>> [7] https://github.com/lukecwik/incubator-beam/tree/gradle
>> --
>>
>>
>> Got feedback? http://go/swegner-feedback
>>
> --
>
>
> Got feedback? http://go/swegner-feedback
>