You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Chamikara Jayalath <ch...@google.com> on 2020/09/29 05:34:14 UTC
Behavior change for Gradle build target
Hi,
I just noticed that the result jars produced by command "./gradlew
:sdks:java:io:expansion-service:build" went from both shaded [1] and
unshaded [2] jars for Beam 2.24.0 branch to just the unshaded jar [3] for
Beam 2.25.0 branch.
Does anyone know what resulted in this behaviour change ?
Also, does this mean that the jars released for Beam 2.25.0 will be
different ? (if so I believe that will break cross-language transforms that
will try to download the shaded jar for expansion).
[1] beam-sdks-java-io-expansion-service-2.24.0-SNAPSHOT.jar
[2] beam-sdks-java-io-expansion-service-2.24.0-SNAPSHOT-unshaded.jar
[3] beam-sdks-java-io-expansion-service-2.25.0-SNAPSHOT-unshaded.jar
Re: Behavior change for Gradle build target
Posted by Brian Hulette <bh...@google.com>.
I opened up BEAM-10986 [1] to track this, let's move discussion there.
[1] https://issues.apache.org/jira/browse/BEAM-10986
On Tue, Sep 29, 2020 at 2:40 PM Brian Hulette <bh...@google.com> wrote:
> A bisect found [1] to be the culprit, it upgrades the shadow plugin. I
> suspect it's related to this change in shadow 4.0.4 [2], but I'm not sure:
>
> > When using shadow, application, and maven plugins together, remove
> shadowDistZip and shadowDistTar from
> > configurations.archives so they are not published or installed by
> default with the uploadArchives or install
> > tasks. #347
>
> I think we may want to rollback the shadow plugin upgrade and cherry-pick
> onto the 2.25.0 branch until we can debug.
>
> [1] https://github.com/apache/beam/pull/12821
> [2] https://github.com/johnrengelman/shadow/releases/tag/4.0.4
>
> On Tue, Sep 29, 2020 at 1:45 PM Brian Hulette <bh...@google.com> wrote:
>
>> I think that output is still incorrect for
>> :sdks:java:io:expansion-service, there should be both shaded and unshaded
>> in ./sdks/java/io/expansion-service/build/libs
>>
>> On Tue, Sep 29, 2020 at 1:08 PM Tyson Hamilton <ty...@google.com>
>> wrote:
>>
>>> Or, possibly that PR already fixed the issue. Could you sync, retry
>>> again? Here is what I found after running the gradle task you mentioned:
>>>
>>> ttysonjh@tysonjh:~/Development/beam$ find . -name
>>> "*expansion-service*SNAPSHOT*"
>>>
>>> ./sdks/java/expansion-service/build/libs/beam-sdks-java-expansion-service-2.26.0-SNAPSHOT.jar
>>>
>>> ./sdks/java/io/expansion-service/build/libs/beam-sdks-java-io-expansion-service-2.26.0-SNAPSHOT-unshaded.jar
>>>
>>> ./sdks/java/io/expansion-service/build/libs/beam-sdks-java-io-expansion-service-2.26.0-SNAPSHOT-tests-unshaded.jar
>>>
>>> ./sdks/java/io/expansion-service/build/distributions/expansion-service-2.26.0-SNAPSHOT.tar
>>>
>>> ./sdks/java/io/expansion-service/build/distributions/expansion-service-2.26.0-SNAPSHOT.zip
>>>
>>>
>>> On Tue, Sep 29, 2020 at 12:59 PM Tyson Hamilton <ty...@google.com>
>>> wrote:
>>>
>>>> Hm. It sounds like it is possibly related to the gradle upgrade to 6? I
>>>> had a similar issue that I fixed for the Java nightly snapshot build in
>>>> PR#12947 (https://github.com/apache/beam/pull/12947). It may require
>>>> similar changes to the gradle.build file to generate the jar.
>>>>
>>>> On Tue, Sep 29, 2020 at 8:20 AM Chamikara Jayalath <
>>>> chamikara@google.com> wrote:
>>>>
>>>>> +Brian Hulette <bh...@google.com> for any follow ups here since
>>>>> I'll be OOO for a few days.
>>>>>
>>>>> Thanks,
>>>>> Cham
>>>>>
>>>>> On Mon, Sep 28, 2020 at 10:34 PM Chamikara Jayalath <
>>>>> chamikara@google.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I just noticed that the result jars produced by command "./gradlew
>>>>>> :sdks:java:io:expansion-service:build" went from both shaded [1] and
>>>>>> unshaded [2] jars for Beam 2.24.0 branch to just the unshaded jar [3] for
>>>>>> Beam 2.25.0 branch.
>>>>>>
>>>>>> Does anyone know what resulted in this behaviour change ?
>>>>>>
>>>>>> Also, does this mean that the jars released for Beam 2.25.0 will be
>>>>>> different ? (if so I believe that will break cross-language transforms that
>>>>>> will try to download the shaded jar for expansion).
>>>>>>
>>>>>> [1] beam-sdks-java-io-expansion-service-2.24.0-SNAPSHOT.jar
>>>>>> [2] beam-sdks-java-io-expansion-service-2.24.0-SNAPSHOT-unshaded.jar
>>>>>> [3] beam-sdks-java-io-expansion-service-2.25.0-SNAPSHOT-unshaded.jar
>>>>>>
>>>>>
Re: Behavior change for Gradle build target
Posted by Brian Hulette <bh...@google.com>.
A bisect found [1] to be the culprit, it upgrades the shadow plugin. I
suspect it's related to this change in shadow 4.0.4 [2], but I'm not sure:
> When using shadow, application, and maven plugins together, remove
shadowDistZip and shadowDistTar from
> configurations.archives so they are not published or installed by default
with the uploadArchives or install
> tasks. #347
I think we may want to rollback the shadow plugin upgrade and cherry-pick
onto the 2.25.0 branch until we can debug.
[1] https://github.com/apache/beam/pull/12821
[2] https://github.com/johnrengelman/shadow/releases/tag/4.0.4
On Tue, Sep 29, 2020 at 1:45 PM Brian Hulette <bh...@google.com> wrote:
> I think that output is still incorrect for
> :sdks:java:io:expansion-service, there should be both shaded and unshaded
> in ./sdks/java/io/expansion-service/build/libs
>
> On Tue, Sep 29, 2020 at 1:08 PM Tyson Hamilton <ty...@google.com> wrote:
>
>> Or, possibly that PR already fixed the issue. Could you sync, retry
>> again? Here is what I found after running the gradle task you mentioned:
>>
>> ttysonjh@tysonjh:~/Development/beam$ find . -name
>> "*expansion-service*SNAPSHOT*"
>>
>> ./sdks/java/expansion-service/build/libs/beam-sdks-java-expansion-service-2.26.0-SNAPSHOT.jar
>>
>> ./sdks/java/io/expansion-service/build/libs/beam-sdks-java-io-expansion-service-2.26.0-SNAPSHOT-unshaded.jar
>>
>> ./sdks/java/io/expansion-service/build/libs/beam-sdks-java-io-expansion-service-2.26.0-SNAPSHOT-tests-unshaded.jar
>>
>> ./sdks/java/io/expansion-service/build/distributions/expansion-service-2.26.0-SNAPSHOT.tar
>>
>> ./sdks/java/io/expansion-service/build/distributions/expansion-service-2.26.0-SNAPSHOT.zip
>>
>>
>> On Tue, Sep 29, 2020 at 12:59 PM Tyson Hamilton <ty...@google.com>
>> wrote:
>>
>>> Hm. It sounds like it is possibly related to the gradle upgrade to 6? I
>>> had a similar issue that I fixed for the Java nightly snapshot build in
>>> PR#12947 (https://github.com/apache/beam/pull/12947). It may require
>>> similar changes to the gradle.build file to generate the jar.
>>>
>>> On Tue, Sep 29, 2020 at 8:20 AM Chamikara Jayalath <ch...@google.com>
>>> wrote:
>>>
>>>> +Brian Hulette <bh...@google.com> for any follow ups here since
>>>> I'll be OOO for a few days.
>>>>
>>>> Thanks,
>>>> Cham
>>>>
>>>> On Mon, Sep 28, 2020 at 10:34 PM Chamikara Jayalath <
>>>> chamikara@google.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I just noticed that the result jars produced by command "./gradlew
>>>>> :sdks:java:io:expansion-service:build" went from both shaded [1] and
>>>>> unshaded [2] jars for Beam 2.24.0 branch to just the unshaded jar [3] for
>>>>> Beam 2.25.0 branch.
>>>>>
>>>>> Does anyone know what resulted in this behaviour change ?
>>>>>
>>>>> Also, does this mean that the jars released for Beam 2.25.0 will be
>>>>> different ? (if so I believe that will break cross-language transforms that
>>>>> will try to download the shaded jar for expansion).
>>>>>
>>>>> [1] beam-sdks-java-io-expansion-service-2.24.0-SNAPSHOT.jar
>>>>> [2] beam-sdks-java-io-expansion-service-2.24.0-SNAPSHOT-unshaded.jar
>>>>> [3] beam-sdks-java-io-expansion-service-2.25.0-SNAPSHOT-unshaded.jar
>>>>>
>>>>
Re: Behavior change for Gradle build target
Posted by Brian Hulette <bh...@google.com>.
I think that output is still incorrect for :sdks:java:io:expansion-service,
there should be both shaded and unshaded in
./sdks/java/io/expansion-service/build/libs
On Tue, Sep 29, 2020 at 1:08 PM Tyson Hamilton <ty...@google.com> wrote:
> Or, possibly that PR already fixed the issue. Could you sync, retry again?
> Here is what I found after running the gradle task you mentioned:
>
> ttysonjh@tysonjh:~/Development/beam$ find . -name
> "*expansion-service*SNAPSHOT*"
>
> ./sdks/java/expansion-service/build/libs/beam-sdks-java-expansion-service-2.26.0-SNAPSHOT.jar
>
> ./sdks/java/io/expansion-service/build/libs/beam-sdks-java-io-expansion-service-2.26.0-SNAPSHOT-unshaded.jar
>
> ./sdks/java/io/expansion-service/build/libs/beam-sdks-java-io-expansion-service-2.26.0-SNAPSHOT-tests-unshaded.jar
>
> ./sdks/java/io/expansion-service/build/distributions/expansion-service-2.26.0-SNAPSHOT.tar
>
> ./sdks/java/io/expansion-service/build/distributions/expansion-service-2.26.0-SNAPSHOT.zip
>
>
> On Tue, Sep 29, 2020 at 12:59 PM Tyson Hamilton <ty...@google.com>
> wrote:
>
>> Hm. It sounds like it is possibly related to the gradle upgrade to 6? I
>> had a similar issue that I fixed for the Java nightly snapshot build in
>> PR#12947 (https://github.com/apache/beam/pull/12947). It may require
>> similar changes to the gradle.build file to generate the jar.
>>
>> On Tue, Sep 29, 2020 at 8:20 AM Chamikara Jayalath <ch...@google.com>
>> wrote:
>>
>>> +Brian Hulette <bh...@google.com> for any follow ups here since I'll
>>> be OOO for a few days.
>>>
>>> Thanks,
>>> Cham
>>>
>>> On Mon, Sep 28, 2020 at 10:34 PM Chamikara Jayalath <
>>> chamikara@google.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I just noticed that the result jars produced by command "./gradlew
>>>> :sdks:java:io:expansion-service:build" went from both shaded [1] and
>>>> unshaded [2] jars for Beam 2.24.0 branch to just the unshaded jar [3] for
>>>> Beam 2.25.0 branch.
>>>>
>>>> Does anyone know what resulted in this behaviour change ?
>>>>
>>>> Also, does this mean that the jars released for Beam 2.25.0 will be
>>>> different ? (if so I believe that will break cross-language transforms that
>>>> will try to download the shaded jar for expansion).
>>>>
>>>> [1] beam-sdks-java-io-expansion-service-2.24.0-SNAPSHOT.jar
>>>> [2] beam-sdks-java-io-expansion-service-2.24.0-SNAPSHOT-unshaded.jar
>>>> [3] beam-sdks-java-io-expansion-service-2.25.0-SNAPSHOT-unshaded.jar
>>>>
>>>
Re: Behavior change for Gradle build target
Posted by Tyson Hamilton <ty...@google.com>.
Or, possibly that PR already fixed the issue. Could you sync, retry again?
Here is what I found after running the gradle task you mentioned:
ttysonjh@tysonjh:~/Development/beam$ find . -name
"*expansion-service*SNAPSHOT*"
./sdks/java/expansion-service/build/libs/beam-sdks-java-expansion-service-2.26.0-SNAPSHOT.jar
./sdks/java/io/expansion-service/build/libs/beam-sdks-java-io-expansion-service-2.26.0-SNAPSHOT-unshaded.jar
./sdks/java/io/expansion-service/build/libs/beam-sdks-java-io-expansion-service-2.26.0-SNAPSHOT-tests-unshaded.jar
./sdks/java/io/expansion-service/build/distributions/expansion-service-2.26.0-SNAPSHOT.tar
./sdks/java/io/expansion-service/build/distributions/expansion-service-2.26.0-SNAPSHOT.zip
On Tue, Sep 29, 2020 at 12:59 PM Tyson Hamilton <ty...@google.com> wrote:
> Hm. It sounds like it is possibly related to the gradle upgrade to 6? I
> had a similar issue that I fixed for the Java nightly snapshot build in
> PR#12947 (https://github.com/apache/beam/pull/12947). It may require
> similar changes to the gradle.build file to generate the jar.
>
> On Tue, Sep 29, 2020 at 8:20 AM Chamikara Jayalath <ch...@google.com>
> wrote:
>
>> +Brian Hulette <bh...@google.com> for any follow ups here since I'll
>> be OOO for a few days.
>>
>> Thanks,
>> Cham
>>
>> On Mon, Sep 28, 2020 at 10:34 PM Chamikara Jayalath <ch...@google.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I just noticed that the result jars produced by command "./gradlew
>>> :sdks:java:io:expansion-service:build" went from both shaded [1] and
>>> unshaded [2] jars for Beam 2.24.0 branch to just the unshaded jar [3] for
>>> Beam 2.25.0 branch.
>>>
>>> Does anyone know what resulted in this behaviour change ?
>>>
>>> Also, does this mean that the jars released for Beam 2.25.0 will be
>>> different ? (if so I believe that will break cross-language transforms that
>>> will try to download the shaded jar for expansion).
>>>
>>> [1] beam-sdks-java-io-expansion-service-2.24.0-SNAPSHOT.jar
>>> [2] beam-sdks-java-io-expansion-service-2.24.0-SNAPSHOT-unshaded.jar
>>> [3] beam-sdks-java-io-expansion-service-2.25.0-SNAPSHOT-unshaded.jar
>>>
>>
Re: Behavior change for Gradle build target
Posted by Tyson Hamilton <ty...@google.com>.
Hm. It sounds like it is possibly related to the gradle upgrade to 6? I had
a similar issue that I fixed for the Java nightly snapshot build in
PR#12947 (https://github.com/apache/beam/pull/12947). It may require
similar changes to the gradle.build file to generate the jar.
On Tue, Sep 29, 2020 at 8:20 AM Chamikara Jayalath <ch...@google.com>
wrote:
> +Brian Hulette <bh...@google.com> for any follow ups here since I'll
> be OOO for a few days.
>
> Thanks,
> Cham
>
> On Mon, Sep 28, 2020 at 10:34 PM Chamikara Jayalath <ch...@google.com>
> wrote:
>
>> Hi,
>>
>> I just noticed that the result jars produced by command "./gradlew
>> :sdks:java:io:expansion-service:build" went from both shaded [1] and
>> unshaded [2] jars for Beam 2.24.0 branch to just the unshaded jar [3] for
>> Beam 2.25.0 branch.
>>
>> Does anyone know what resulted in this behaviour change ?
>>
>> Also, does this mean that the jars released for Beam 2.25.0 will be
>> different ? (if so I believe that will break cross-language transforms that
>> will try to download the shaded jar for expansion).
>>
>> [1] beam-sdks-java-io-expansion-service-2.24.0-SNAPSHOT.jar
>> [2] beam-sdks-java-io-expansion-service-2.24.0-SNAPSHOT-unshaded.jar
>> [3] beam-sdks-java-io-expansion-service-2.25.0-SNAPSHOT-unshaded.jar
>>
>
Re: Behavior change for Gradle build target
Posted by Chamikara Jayalath <ch...@google.com>.
+Brian Hulette <bh...@google.com> for any follow ups here since I'll be
OOO for a few days.
Thanks,
Cham
On Mon, Sep 28, 2020 at 10:34 PM Chamikara Jayalath <ch...@google.com>
wrote:
> Hi,
>
> I just noticed that the result jars produced by command "./gradlew
> :sdks:java:io:expansion-service:build" went from both shaded [1] and
> unshaded [2] jars for Beam 2.24.0 branch to just the unshaded jar [3] for
> Beam 2.25.0 branch.
>
> Does anyone know what resulted in this behaviour change ?
>
> Also, does this mean that the jars released for Beam 2.25.0 will be
> different ? (if so I believe that will break cross-language transforms that
> will try to download the shaded jar for expansion).
>
> [1] beam-sdks-java-io-expansion-service-2.24.0-SNAPSHOT.jar
> [2] beam-sdks-java-io-expansion-service-2.24.0-SNAPSHOT-unshaded.jar
> [3] beam-sdks-java-io-expansion-service-2.25.0-SNAPSHOT-unshaded.jar
>