You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Pablo Estrada <pa...@google.com> on 2018/05/17 21:45:19 UTC
Java PreCommit seems broken
I'm seeing failures on Maven Archetype-related tests.
Build Scan of a sample run: https://scans.gradle.com/s/kr23q43mh6fmk
And the failure is here specifically:
https://scans.gradle.com/s/kr23q43mh6fmk/console-log?task=:beam-sdks-java-maven-archetypes-examples:generateAndBuildArchetypeTest#L116
Does anyone know why this might be happening?
Best
-P.
--
Got feedback? go/pabloem-feedback
Re: Java PreCommit seems broken
Posted by Scott Wegner <sw...@google.com>.
I've logged BEAM-4382 [1] to decouple maven archetype generation from the
rest of the Maven build. Luke, would you mind adding any context you have
about generating archetypes from Gradle? From a quick search I couldn't
find a native Gradle plugin, but perhaps the logic is simple enough to roll
our own logic.
[1] https://issues.apache.org/jira/browse/BEAM-4382
On Mon, May 21, 2018 at 11:16 AM Lukasz Cwik <lc...@google.com> wrote:
> The archetype projects are coupled to their parents but don't have any
> meaningful dependencies so a significantly simpler archetype could be used.
> The dependency that exists right now in the archetype is to provide
> specific build ordering which can easily be moved to Gradle directly.
>
> An alternative would be to migrate the Maven archetype build to use
> Gradle. Assembling the Maven archetype jar is easy as no compilation is
> required, the issue was about running/validating that the archetype can be
> built.
>
> On Fri, May 18, 2018 at 12:20 PM Scott Wegner <sw...@google.com> wrote:
>
>> +1 to the Lukasz's proposed solution. Depending on artifacts published
>> from a previous build it's fragile and will add flakiness to our test runs.
>> We should make pre-commits as hermetic as possible.
>>
>> Depending on the transitive set of publishToMavenLocal tasks seems
>> cumbersome, but also necessary.
>>
>> On a related note: The archetype projects are shelling out to mvn for the
>> build, which uses the existing pom.xml files. This places a build
>> dependency on the pom.xml files down to the project root due to parent
>> relationships. Has there been any investigation on whether we can decouple
>> archetype generation from our Maven pom.xml files?
>>
>> On Fri, May 18, 2018 at 10:47 AM Lukasz Cwik <lc...@google.com> wrote:
>>
>>> We would need the archetype task to depend on all the dependencies
>>> publishToMavenLocal tasks transitively and then be configured to use
>>> whatever that maven local is on Jenkins / dev machine. It would be best if
>>> it was an ephemeral folder because it would be annoying to have stuff
>>> installed underneath a devs .m2/ directory that would need cleaning up.
>>>
>>> On Fri, May 18, 2018 at 10:41 AM Kenneth Knowles <kl...@google.com> wrote:
>>>
>>>> Is this just a build tweak, or are there costly steps that we'd have to
>>>> add that would slow down presubmit? (with mvn I know that `test` and
>>>> `install` did very different amounts of work - because mvn test didn't test
>>>> the right artifacts, but maybe with Gradle not so much?)
>>>>
>>>> On Fri, May 18, 2018 at 9:14 AM Lukasz Cwik <lc...@google.com> wrote:
>>>>
>>>>> The problem with the way that the archetypes tests are run (now with
>>>>> Gradle and in the past with Maven) is that they run against the nightly
>>>>> snapshot and not against artifacts from the current build. To get them to
>>>>> work, we would need to publish the dependent Maven modules to a temporary
>>>>> repo and instruct the archetype project to use it for building/testing
>>>>> purposes.
>>>>>
>>>>> On Fri, May 18, 2018 at 5:38 AM Kenneth Knowles <kl...@google.com>
>>>>> wrote:
>>>>>
>>>>>> Maybe something has changed, but the snapshots used to pull from the
>>>>>> public snapshot repo. We got failures for a while every time we cut a
>>>>>> release branch, but once there was a nightly snapshot they cleared up.
>>>>>>
>>>>>> Kenn
>>>>>>
>>>>>> On Thu, May 17, 2018 at 9:50 PM Scott Wegner <sw...@google.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I noticed that tests tests simply run "mvn clean install" on the
>>>>>>> archetype project. But I don't see any dependent task which installs built
>>>>>>> artifacts into the local Maven repo. Is that an oversight?
>>>>>>>
>>>>>>> If that's the case, perhaps the tests are failing sporadically when
>>>>>>> there are no previously installed snapshot artifacts cached on the machine.
>>>>>>>
>>>>>>> On Thu, May 17, 2018, 2:45 PM Pablo Estrada <pa...@google.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I'm seeing failures on Maven Archetype-related tests.
>>>>>>>>
>>>>>>>> Build Scan of a sample run:
>>>>>>>> https://scans.gradle.com/s/kr23q43mh6fmk
>>>>>>>>
>>>>>>>> And the failure is here specifically:
>>>>>>>> https://scans.gradle.com/s/kr23q43mh6fmk/console-log?task=:beam-sdks-java-maven-archetypes-examples:generateAndBuildArchetypeTest#L116
>>>>>>>>
>>>>>>>>
>>>>>>>> Does anyone know why this might be happening?
>>>>>>>> Best
>>>>>>>> -P.
>>>>>>>> --
>>>>>>>> Got feedback? go/pabloem-feedback
>>>>>>>> <https://goto.google.com/pabloem-feedback>
>>>>>>>>
>>>>>>>
Re: Java PreCommit seems broken
Posted by Lukasz Cwik <lc...@google.com>.
The archetype projects are coupled to their parents but don't have any
meaningful dependencies so a significantly simpler archetype could be used.
The dependency that exists right now in the archetype is to provide
specific build ordering which can easily be moved to Gradle directly.
An alternative would be to migrate the Maven archetype build to use Gradle.
Assembling the Maven archetype jar is easy as no compilation is required,
the issue was about running/validating that the archetype can be built.
On Fri, May 18, 2018 at 12:20 PM Scott Wegner <sw...@google.com> wrote:
> +1 to the Lukasz's proposed solution. Depending on artifacts published
> from a previous build it's fragile and will add flakiness to our test runs.
> We should make pre-commits as hermetic as possible.
>
> Depending on the transitive set of publishToMavenLocal tasks seems
> cumbersome, but also necessary.
>
> On a related note: The archetype projects are shelling out to mvn for the
> build, which uses the existing pom.xml files. This places a build
> dependency on the pom.xml files down to the project root due to parent
> relationships. Has there been any investigation on whether we can decouple
> archetype generation from our Maven pom.xml files?
>
> On Fri, May 18, 2018 at 10:47 AM Lukasz Cwik <lc...@google.com> wrote:
>
>> We would need the archetype task to depend on all the dependencies
>> publishToMavenLocal tasks transitively and then be configured to use
>> whatever that maven local is on Jenkins / dev machine. It would be best if
>> it was an ephemeral folder because it would be annoying to have stuff
>> installed underneath a devs .m2/ directory that would need cleaning up.
>>
>> On Fri, May 18, 2018 at 10:41 AM Kenneth Knowles <kl...@google.com> wrote:
>>
>>> Is this just a build tweak, or are there costly steps that we'd have to
>>> add that would slow down presubmit? (with mvn I know that `test` and
>>> `install` did very different amounts of work - because mvn test didn't test
>>> the right artifacts, but maybe with Gradle not so much?)
>>>
>>> On Fri, May 18, 2018 at 9:14 AM Lukasz Cwik <lc...@google.com> wrote:
>>>
>>>> The problem with the way that the archetypes tests are run (now with
>>>> Gradle and in the past with Maven) is that they run against the nightly
>>>> snapshot and not against artifacts from the current build. To get them to
>>>> work, we would need to publish the dependent Maven modules to a temporary
>>>> repo and instruct the archetype project to use it for building/testing
>>>> purposes.
>>>>
>>>> On Fri, May 18, 2018 at 5:38 AM Kenneth Knowles <kl...@google.com> wrote:
>>>>
>>>>> Maybe something has changed, but the snapshots used to pull from the
>>>>> public snapshot repo. We got failures for a while every time we cut a
>>>>> release branch, but once there was a nightly snapshot they cleared up.
>>>>>
>>>>> Kenn
>>>>>
>>>>> On Thu, May 17, 2018 at 9:50 PM Scott Wegner <sw...@google.com>
>>>>> wrote:
>>>>>
>>>>>> I noticed that tests tests simply run "mvn clean install" on the
>>>>>> archetype project. But I don't see any dependent task which installs built
>>>>>> artifacts into the local Maven repo. Is that an oversight?
>>>>>>
>>>>>> If that's the case, perhaps the tests are failing sporadically when
>>>>>> there are no previously installed snapshot artifacts cached on the machine.
>>>>>>
>>>>>> On Thu, May 17, 2018, 2:45 PM Pablo Estrada <pa...@google.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I'm seeing failures on Maven Archetype-related tests.
>>>>>>>
>>>>>>> Build Scan of a sample run: https://scans.gradle.com/s/kr23q43mh6fmk
>>>>>>>
>>>>>>> And the failure is here specifically:
>>>>>>> https://scans.gradle.com/s/kr23q43mh6fmk/console-log?task=:beam-sdks-java-maven-archetypes-examples:generateAndBuildArchetypeTest#L116
>>>>>>>
>>>>>>>
>>>>>>> Does anyone know why this might be happening?
>>>>>>> Best
>>>>>>> -P.
>>>>>>> --
>>>>>>> Got feedback? go/pabloem-feedback
>>>>>>> <https://goto.google.com/pabloem-feedback>
>>>>>>>
>>>>>>
Re: Java PreCommit seems broken
Posted by Scott Wegner <sw...@google.com>.
+1 to the Lukasz's proposed solution. Depending on artifacts published from
a previous build it's fragile and will add flakiness to our test runs. We
should make pre-commits as hermetic as possible.
Depending on the transitive set of publishToMavenLocal tasks seems
cumbersome, but also necessary.
On a related note: The archetype projects are shelling out to mvn for the
build, which uses the existing pom.xml files. This places a build
dependency on the pom.xml files down to the project root due to parent
relationships. Has there been any investigation on whether we can decouple
archetype generation from our Maven pom.xml files?
On Fri, May 18, 2018 at 10:47 AM Lukasz Cwik <lc...@google.com> wrote:
> We would need the archetype task to depend on all the dependencies
> publishToMavenLocal tasks transitively and then be configured to use
> whatever that maven local is on Jenkins / dev machine. It would be best if
> it was an ephemeral folder because it would be annoying to have stuff
> installed underneath a devs .m2/ directory that would need cleaning up.
>
> On Fri, May 18, 2018 at 10:41 AM Kenneth Knowles <kl...@google.com> wrote:
>
>> Is this just a build tweak, or are there costly steps that we'd have to
>> add that would slow down presubmit? (with mvn I know that `test` and
>> `install` did very different amounts of work - because mvn test didn't test
>> the right artifacts, but maybe with Gradle not so much?)
>>
>> On Fri, May 18, 2018 at 9:14 AM Lukasz Cwik <lc...@google.com> wrote:
>>
>>> The problem with the way that the archetypes tests are run (now with
>>> Gradle and in the past with Maven) is that they run against the nightly
>>> snapshot and not against artifacts from the current build. To get them to
>>> work, we would need to publish the dependent Maven modules to a temporary
>>> repo and instruct the archetype project to use it for building/testing
>>> purposes.
>>>
>>> On Fri, May 18, 2018 at 5:38 AM Kenneth Knowles <kl...@google.com> wrote:
>>>
>>>> Maybe something has changed, but the snapshots used to pull from the
>>>> public snapshot repo. We got failures for a while every time we cut a
>>>> release branch, but once there was a nightly snapshot they cleared up.
>>>>
>>>> Kenn
>>>>
>>>> On Thu, May 17, 2018 at 9:50 PM Scott Wegner <sw...@google.com>
>>>> wrote:
>>>>
>>>>> I noticed that tests tests simply run "mvn clean install" on the
>>>>> archetype project. But I don't see any dependent task which installs built
>>>>> artifacts into the local Maven repo. Is that an oversight?
>>>>>
>>>>> If that's the case, perhaps the tests are failing sporadically when
>>>>> there are no previously installed snapshot artifacts cached on the machine.
>>>>>
>>>>> On Thu, May 17, 2018, 2:45 PM Pablo Estrada <pa...@google.com>
>>>>> wrote:
>>>>>
>>>>>> I'm seeing failures on Maven Archetype-related tests.
>>>>>>
>>>>>> Build Scan of a sample run: https://scans.gradle.com/s/kr23q43mh6fmk
>>>>>>
>>>>>> And the failure is here specifically:
>>>>>> https://scans.gradle.com/s/kr23q43mh6fmk/console-log?task=:beam-sdks-java-maven-archetypes-examples:generateAndBuildArchetypeTest#L116
>>>>>>
>>>>>>
>>>>>> Does anyone know why this might be happening?
>>>>>> Best
>>>>>> -P.
>>>>>> --
>>>>>> Got feedback? go/pabloem-feedback
>>>>>> <https://goto.google.com/pabloem-feedback>
>>>>>>
>>>>>
Re: Java PreCommit seems broken
Posted by Lukasz Cwik <lc...@google.com>.
We would need the archetype task to depend on all the dependencies
publishToMavenLocal tasks transitively and then be configured to use
whatever that maven local is on Jenkins / dev machine. It would be best if
it was an ephemeral folder because it would be annoying to have stuff
installed underneath a devs .m2/ directory that would need cleaning up.
On Fri, May 18, 2018 at 10:41 AM Kenneth Knowles <kl...@google.com> wrote:
> Is this just a build tweak, or are there costly steps that we'd have to
> add that would slow down presubmit? (with mvn I know that `test` and
> `install` did very different amounts of work - because mvn test didn't test
> the right artifacts, but maybe with Gradle not so much?)
>
> On Fri, May 18, 2018 at 9:14 AM Lukasz Cwik <lc...@google.com> wrote:
>
>> The problem with the way that the archetypes tests are run (now with
>> Gradle and in the past with Maven) is that they run against the nightly
>> snapshot and not against artifacts from the current build. To get them to
>> work, we would need to publish the dependent Maven modules to a temporary
>> repo and instruct the archetype project to use it for building/testing
>> purposes.
>>
>> On Fri, May 18, 2018 at 5:38 AM Kenneth Knowles <kl...@google.com> wrote:
>>
>>> Maybe something has changed, but the snapshots used to pull from the
>>> public snapshot repo. We got failures for a while every time we cut a
>>> release branch, but once there was a nightly snapshot they cleared up.
>>>
>>> Kenn
>>>
>>> On Thu, May 17, 2018 at 9:50 PM Scott Wegner <sw...@google.com> wrote:
>>>
>>>> I noticed that tests tests simply run "mvn clean install" on the
>>>> archetype project. But I don't see any dependent task which installs built
>>>> artifacts into the local Maven repo. Is that an oversight?
>>>>
>>>> If that's the case, perhaps the tests are failing sporadically when
>>>> there are no previously installed snapshot artifacts cached on the machine.
>>>>
>>>> On Thu, May 17, 2018, 2:45 PM Pablo Estrada <pa...@google.com> wrote:
>>>>
>>>>> I'm seeing failures on Maven Archetype-related tests.
>>>>>
>>>>> Build Scan of a sample run: https://scans.gradle.com/s/kr23q43mh6fmk
>>>>>
>>>>> And the failure is here specifically:
>>>>> https://scans.gradle.com/s/kr23q43mh6fmk/console-log?task=:beam-sdks-java-maven-archetypes-examples:generateAndBuildArchetypeTest#L116
>>>>>
>>>>>
>>>>> Does anyone know why this might be happening?
>>>>> Best
>>>>> -P.
>>>>> --
>>>>> Got feedback? go/pabloem-feedback
>>>>> <https://goto.google.com/pabloem-feedback>
>>>>>
>>>>
Re: Java PreCommit seems broken
Posted by Kenneth Knowles <kl...@google.com>.
Is this just a build tweak, or are there costly steps that we'd have to add
that would slow down presubmit? (with mvn I know that `test` and `install`
did very different amounts of work - because mvn test didn't test the right
artifacts, but maybe with Gradle not so much?)
On Fri, May 18, 2018 at 9:14 AM Lukasz Cwik <lc...@google.com> wrote:
> The problem with the way that the archetypes tests are run (now with
> Gradle and in the past with Maven) is that they run against the nightly
> snapshot and not against artifacts from the current build. To get them to
> work, we would need to publish the dependent Maven modules to a temporary
> repo and instruct the archetype project to use it for building/testing
> purposes.
>
> On Fri, May 18, 2018 at 5:38 AM Kenneth Knowles <kl...@google.com> wrote:
>
>> Maybe something has changed, but the snapshots used to pull from the
>> public snapshot repo. We got failures for a while every time we cut a
>> release branch, but once there was a nightly snapshot they cleared up.
>>
>> Kenn
>>
>> On Thu, May 17, 2018 at 9:50 PM Scott Wegner <sw...@google.com> wrote:
>>
>>> I noticed that tests tests simply run "mvn clean install" on the
>>> archetype project. But I don't see any dependent task which installs built
>>> artifacts into the local Maven repo. Is that an oversight?
>>>
>>> If that's the case, perhaps the tests are failing sporadically when
>>> there are no previously installed snapshot artifacts cached on the machine.
>>>
>>> On Thu, May 17, 2018, 2:45 PM Pablo Estrada <pa...@google.com> wrote:
>>>
>>>> I'm seeing failures on Maven Archetype-related tests.
>>>>
>>>> Build Scan of a sample run: https://scans.gradle.com/s/kr23q43mh6fmk
>>>>
>>>> And the failure is here specifically:
>>>> https://scans.gradle.com/s/kr23q43mh6fmk/console-log?task=:beam-sdks-java-maven-archetypes-examples:generateAndBuildArchetypeTest#L116
>>>>
>>>>
>>>> Does anyone know why this might be happening?
>>>> Best
>>>> -P.
>>>> --
>>>> Got feedback? go/pabloem-feedback
>>>> <https://goto.google.com/pabloem-feedback>
>>>>
>>>
Re: Java PreCommit seems broken
Posted by Lukasz Cwik <lc...@google.com>.
The problem with the way that the archetypes tests are run (now with Gradle
and in the past with Maven) is that they run against the nightly snapshot
and not against artifacts from the current build. To get them to work, we
would need to publish the dependent Maven modules to a temporary repo and
instruct the archetype project to use it for building/testing purposes.
On Fri, May 18, 2018 at 5:38 AM Kenneth Knowles <kl...@google.com> wrote:
> Maybe something has changed, but the snapshots used to pull from the
> public snapshot repo. We got failures for a while every time we cut a
> release branch, but once there was a nightly snapshot they cleared up.
>
> Kenn
>
> On Thu, May 17, 2018 at 9:50 PM Scott Wegner <sw...@google.com> wrote:
>
>> I noticed that tests tests simply run "mvn clean install" on the
>> archetype project. But I don't see any dependent task which installs built
>> artifacts into the local Maven repo. Is that an oversight?
>>
>> If that's the case, perhaps the tests are failing sporadically when there
>> are no previously installed snapshot artifacts cached on the machine.
>>
>> On Thu, May 17, 2018, 2:45 PM Pablo Estrada <pa...@google.com> wrote:
>>
>>> I'm seeing failures on Maven Archetype-related tests.
>>>
>>> Build Scan of a sample run: https://scans.gradle.com/s/kr23q43mh6fmk
>>>
>>> And the failure is here specifically:
>>> https://scans.gradle.com/s/kr23q43mh6fmk/console-log?task=:beam-sdks-java-maven-archetypes-examples:generateAndBuildArchetypeTest#L116
>>>
>>>
>>> Does anyone know why this might be happening?
>>> Best
>>> -P.
>>> --
>>> Got feedback? go/pabloem-feedback
>>> <https://goto.google.com/pabloem-feedback>
>>>
>>
Re: Java PreCommit seems broken
Posted by Kenneth Knowles <kl...@google.com>.
Maybe something has changed, but the snapshots used to pull from the public
snapshot repo. We got failures for a while every time we cut a release
branch, but once there was a nightly snapshot they cleared up.
Kenn
On Thu, May 17, 2018 at 9:50 PM Scott Wegner <sw...@google.com> wrote:
> I noticed that tests tests simply run "mvn clean install" on the archetype
> project. But I don't see any dependent task which installs built artifacts
> into the local Maven repo. Is that an oversight?
>
> If that's the case, perhaps the tests are failing sporadically when there
> are no previously installed snapshot artifacts cached on the machine.
>
> On Thu, May 17, 2018, 2:45 PM Pablo Estrada <pa...@google.com> wrote:
>
>> I'm seeing failures on Maven Archetype-related tests.
>>
>> Build Scan of a sample run: https://scans.gradle.com/s/kr23q43mh6fmk
>>
>> And the failure is here specifically:
>> https://scans.gradle.com/s/kr23q43mh6fmk/console-log?task=:beam-sdks-java-maven-archetypes-examples:generateAndBuildArchetypeTest#L116
>>
>>
>> Does anyone know why this might be happening?
>> Best
>> -P.
>> --
>> Got feedback? go/pabloem-feedback
>> <https://goto.google.com/pabloem-feedback>
>>
>
Re: Java PreCommit seems broken
Posted by Scott Wegner <sw...@google.com>.
I noticed that tests tests simply run "mvn clean install" on the archetype
project. But I don't see any dependent task which installs built artifacts
into the local Maven repo. Is that an oversight?
If that's the case, perhaps the tests are failing sporadically when there
are no previously installed snapshot artifacts cached on the machine.
On Thu, May 17, 2018, 2:45 PM Pablo Estrada <pa...@google.com> wrote:
> I'm seeing failures on Maven Archetype-related tests.
>
> Build Scan of a sample run: https://scans.gradle.com/s/kr23q43mh6fmk
>
> And the failure is here specifically:
> https://scans.gradle.com/s/kr23q43mh6fmk/console-log?task=:beam-sdks-java-maven-archetypes-examples:generateAndBuildArchetypeTest#L116
>
>
> Does anyone know why this might be happening?
> Best
> -P.
> --
> Got feedback? go/pabloem-feedback
> <https://goto.google.com/pabloem-feedback>
>