You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Michael Luckey <ad...@gmail.com> on 2018/10/28 16:57:18 UTC
[Build-System] Facilitate up-to-date checks on test/shadowTest jar creation
Hi,
currently we bundle our tests within packageTests task [1] and additionally
create a shadowed version with shadowTestJar.
Unfortunately both task build the jar with the same classifier, which
results in each task overwriting the others output, preventing any
up-to-date checks from working.
Changing the build slightly to produce different artifacts
---
a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
+++
b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
@@ -588,7 +588,7 @@ class BeamModulePlugin implements Plugin<Project> {
// Ensure that tests are packaged and part of the artifact set.
project.task('packageTests', type: Jar) {
- classifier = 'tests'
+ classifier = 'tests-unshaded'
from project.sourceSets.test.output
}
project.artifacts.archives project.packageTests
and so enabling up-to-date checking cuts build time [3] to half (on a build
ignoring python/go/website where up-to-date checks currently do not work
anyway) [4].
As I currently do not fully understand all consequences, I am wondering,
whether we could apply that simple fix or wether there is some hidden
magic, which relies on both artifacts be named the same.
Anyone able to provide further insights here?
Thx,
michel
[1]
https://github.com/apache/beam/blob/master/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L609-L614
[2]
https://github.com/apache/beam/blob/master/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L766-L780
[3]
Profiled build: -x :beam-sdks-python:build -x :beam-sdks-go:build -x
:beam-website:build build
DescriptionDuration
Total Build Time 1m28.94s
Startup 1.657s
Settings and BuildSrc 1.870s
Loading Projects 0.035s
Configuring Projects 4.565s
Task Execution 1m17.11s
[4]
Profiled build: -x :beam-sdks-python:build -x :beam-sdks-go:build -x
:beam-website:build build
DescriptionDuration
Total Build Time 46.781s
Startup 1.633s
Settings and BuildSrc 2.004s
Loading Projects 0.061s
Configuring Projects 4.883s
Task Execution 34.121s
Re: [Build-System] Facilitate up-to-date checks on test/shadowTest
jar creation
Posted by Michael Luckey <ad...@gmail.com>.
Oh wow! Thanks for taking over so fast :)
Would really helpful if that would fix some flakiness also.
michel
On Sun, Oct 28, 2018 at 9:37 PM Kenneth Knowles <ke...@apache.org> wrote:
> Opened https://github.com/apache/beam/pull/6864 with your change but
> would love to get you GitHub contributor credit in the metrics FWIW.
>
> Kenn
>
> On Sun, Oct 28, 2018 at 12:52 PM Kenneth Knowles <ke...@apache.org> wrote:
>
>> Nice!
>>
>> Do you have a PR open? Let's get this change in immediately.
>>
>> This is probably the root cause of the flood of failures in
>> https://issues.apache.org/jira/browse/BEAM-5035 /
>> https://issues.apache.org/jira/browse/BEAM-5207 /
>> https://issues.apache.org/jira/browse/BEAM-5116 that led to us disabling
>> parallel builds.
>>
>> I've been slowly working towards two other independently useful changes
>> that I hoped would help ((1) don't depend on test jars and (2) vendor
>> dependencies so we just don't shade at all). Your solution will get us a
>> fast build sooner.
>>
>> Kenn
>>
>> On Sun, Oct 28, 2018 at 9:57 AM Michael Luckey <ad...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> currently we bundle our tests within packageTests task [1] and
>>> additionally create a shadowed version with shadowTestJar.
>>>
>>> Unfortunately both task build the jar with the same classifier, which
>>> results in each task overwriting the others output, preventing any
>>> up-to-date checks from working.
>>>
>>> Changing the build slightly to produce different artifacts
>>>
>>> ---
>>> a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
>>>
>>> +++
>>> b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
>>>
>>> @@ -588,7 +588,7 @@ class BeamModulePlugin implements Plugin<Project> {
>>>
>>>
>>>
>>> // Ensure that tests are packaged and part of the artifact set.
>>>
>>> project.task('packageTests', type: Jar) {
>>>
>>> - classifier = 'tests'
>>>
>>> + classifier = 'tests-unshaded'
>>>
>>> from project.sourceSets.test.output
>>>
>>> }
>>>
>>> project.artifacts.archives project.packageTests
>>>
>>> and so enabling up-to-date checking cuts build time [3] to half (on a
>>> build ignoring python/go/website where up-to-date checks currently do not
>>> work anyway) [4].
>>>
>>> As I currently do not fully understand all consequences, I am wondering,
>>> whether we could apply that simple fix or wether there is some hidden
>>> magic, which relies on both artifacts be named the same.
>>>
>>> Anyone able to provide further insights here?
>>>
>>> Thx,
>>>
>>> michel
>>>
>>>
>>> [1]
>>> https://github.com/apache/beam/blob/master/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L609-L614
>>> [2]
>>> https://github.com/apache/beam/blob/master/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L766-L780
>>> [3]
>>> Profiled build: -x :beam-sdks-python:build -x :beam-sdks-go:build -x
>>> :beam-website:build build
>>> DescriptionDuration
>>> Total Build Time 1m28.94s
>>> Startup 1.657s
>>> Settings and BuildSrc 1.870s
>>> Loading Projects 0.035s
>>> Configuring Projects 4.565s
>>> Task Execution 1m17.11s
>>> [4]
>>> Profiled build: -x :beam-sdks-python:build -x :beam-sdks-go:build -x
>>> :beam-website:build build
>>>
>>> DescriptionDuration
>>> Total Build Time 46.781s
>>> Startup 1.633s
>>> Settings and BuildSrc 2.004s
>>> Loading Projects 0.061s
>>> Configuring Projects 4.883s
>>> Task Execution 34.121s
>>>
>>>
>>>
>>>
Re: [Build-System] Facilitate up-to-date checks on test/shadowTest
jar creation
Posted by Kenneth Knowles <ke...@apache.org>.
Opened https://github.com/apache/beam/pull/6864 with your change but would
love to get you GitHub contributor credit in the metrics FWIW.
Kenn
On Sun, Oct 28, 2018 at 12:52 PM Kenneth Knowles <ke...@apache.org> wrote:
> Nice!
>
> Do you have a PR open? Let's get this change in immediately.
>
> This is probably the root cause of the flood of failures in
> https://issues.apache.org/jira/browse/BEAM-5035 /
> https://issues.apache.org/jira/browse/BEAM-5207 /
> https://issues.apache.org/jira/browse/BEAM-5116 that led to us disabling
> parallel builds.
>
> I've been slowly working towards two other independently useful changes
> that I hoped would help ((1) don't depend on test jars and (2) vendor
> dependencies so we just don't shade at all). Your solution will get us a
> fast build sooner.
>
> Kenn
>
> On Sun, Oct 28, 2018 at 9:57 AM Michael Luckey <ad...@gmail.com>
> wrote:
>
>> Hi,
>>
>> currently we bundle our tests within packageTests task [1] and
>> additionally create a shadowed version with shadowTestJar.
>>
>> Unfortunately both task build the jar with the same classifier, which
>> results in each task overwriting the others output, preventing any
>> up-to-date checks from working.
>>
>> Changing the build slightly to produce different artifacts
>>
>> ---
>> a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
>>
>> +++
>> b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
>>
>> @@ -588,7 +588,7 @@ class BeamModulePlugin implements Plugin<Project> {
>>
>>
>>
>> // Ensure that tests are packaged and part of the artifact set.
>>
>> project.task('packageTests', type: Jar) {
>>
>> - classifier = 'tests'
>>
>> + classifier = 'tests-unshaded'
>>
>> from project.sourceSets.test.output
>>
>> }
>>
>> project.artifacts.archives project.packageTests
>>
>> and so enabling up-to-date checking cuts build time [3] to half (on a
>> build ignoring python/go/website where up-to-date checks currently do not
>> work anyway) [4].
>>
>> As I currently do not fully understand all consequences, I am wondering,
>> whether we could apply that simple fix or wether there is some hidden
>> magic, which relies on both artifacts be named the same.
>>
>> Anyone able to provide further insights here?
>>
>> Thx,
>>
>> michel
>>
>>
>> [1]
>> https://github.com/apache/beam/blob/master/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L609-L614
>> [2]
>> https://github.com/apache/beam/blob/master/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L766-L780
>> [3]
>> Profiled build: -x :beam-sdks-python:build -x :beam-sdks-go:build -x
>> :beam-website:build build
>> DescriptionDuration
>> Total Build Time 1m28.94s
>> Startup 1.657s
>> Settings and BuildSrc 1.870s
>> Loading Projects 0.035s
>> Configuring Projects 4.565s
>> Task Execution 1m17.11s
>> [4]
>> Profiled build: -x :beam-sdks-python:build -x :beam-sdks-go:build -x
>> :beam-website:build build
>>
>> DescriptionDuration
>> Total Build Time 46.781s
>> Startup 1.633s
>> Settings and BuildSrc 2.004s
>> Loading Projects 0.061s
>> Configuring Projects 4.883s
>> Task Execution 34.121s
>>
>>
>>
>>
Re: [Build-System] Facilitate up-to-date checks on test/shadowTest
jar creation
Posted by Kenneth Knowles <ke...@apache.org>.
Nice!
Do you have a PR open? Let's get this change in immediately.
This is probably the root cause of the flood of failures in
https://issues.apache.org/jira/browse/BEAM-5035 /
https://issues.apache.org/jira/browse/BEAM-5207 /
https://issues.apache.org/jira/browse/BEAM-5116 that led to us disabling
parallel builds.
I've been slowly working towards two other independently useful changes
that I hoped would help ((1) don't depend on test jars and (2) vendor
dependencies so we just don't shade at all). Your solution will get us a
fast build sooner.
Kenn
On Sun, Oct 28, 2018 at 9:57 AM Michael Luckey <ad...@gmail.com> wrote:
> Hi,
>
> currently we bundle our tests within packageTests task [1] and
> additionally create a shadowed version with shadowTestJar.
>
> Unfortunately both task build the jar with the same classifier, which
> results in each task overwriting the others output, preventing any
> up-to-date checks from working.
>
> Changing the build slightly to produce different artifacts
>
> ---
> a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
>
> +++
> b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
>
> @@ -588,7 +588,7 @@ class BeamModulePlugin implements Plugin<Project> {
>
>
>
> // Ensure that tests are packaged and part of the artifact set.
>
> project.task('packageTests', type: Jar) {
>
> - classifier = 'tests'
>
> + classifier = 'tests-unshaded'
>
> from project.sourceSets.test.output
>
> }
>
> project.artifacts.archives project.packageTests
>
> and so enabling up-to-date checking cuts build time [3] to half (on a
> build ignoring python/go/website where up-to-date checks currently do not
> work anyway) [4].
>
> As I currently do not fully understand all consequences, I am wondering,
> whether we could apply that simple fix or wether there is some hidden
> magic, which relies on both artifacts be named the same.
>
> Anyone able to provide further insights here?
>
> Thx,
>
> michel
>
>
> [1]
> https://github.com/apache/beam/blob/master/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L609-L614
> [2]
> https://github.com/apache/beam/blob/master/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L766-L780
> [3]
> Profiled build: -x :beam-sdks-python:build -x :beam-sdks-go:build -x
> :beam-website:build build
> DescriptionDuration
> Total Build Time 1m28.94s
> Startup 1.657s
> Settings and BuildSrc 1.870s
> Loading Projects 0.035s
> Configuring Projects 4.565s
> Task Execution 1m17.11s
> [4]
> Profiled build: -x :beam-sdks-python:build -x :beam-sdks-go:build -x
> :beam-website:build build
>
> DescriptionDuration
> Total Build Time 46.781s
> Startup 1.633s
> Settings and BuildSrc 2.004s
> Loading Projects 0.061s
> Configuring Projects 4.883s
> Task Execution 34.121s
>
>
>
>