You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Andrew Pilloud <ap...@google.com> on 2018/08/23 21:44:20 UTC
Gradle Races in beam-examples-java, beam-runners-apex
There seems to be a misconfiguration of gradle that is causing a high rate
of failure for the last several weeks in building beam-examples-java and
beam-runners-apex. It appears to be some sort of race condition in building
dependencies. Given that no one has made progress on fixing the root cause,
is this something we could mitigate by running jobs with `--no-parallel`
flag?
https://issues.apache.org/jira/browse/BEAM-5035
https://issues.apache.org/jira/browse/BEAM-5207
Andrew
Re: Gradle Races in beam-examples-java, beam-runners-apex
Posted by Maximilian Michels <mx...@apache.org>.
Do we have inotifywait available on Travis and could set it up to log
concurrent access to the relevant Jar files?
On 10.09.18 22:41, Lukasz Cwik wrote:
> I had originally suggested to use some Linux kernel tooling such as
> inotifywait[1] to watch what is happening.
>
> It is likely that we have some Gradle task which is running something in
> parallel to a different Gradle task when it shouldn't which means that
> the jar file is being changed/corrupted. I believe fixing our Gradle
> task dependency tree wrt to this would solve the problem. This crash
> does not reproduce on my desktop after 20 runs which makes it hard for
> me to test for.
>
> 1: https://www.linuxjournal.com/content/linux-filesystem-events-inotify
>
> On Mon, Sep 10, 2018 at 1:15 PM Ryan Williams <ryan@runsascoded.com
> <ma...@runsascoded.com>> wrote:
>
> this continues to be an issue locally (cf. some discussion in #beam
> slack)
>
> commands like `./gradlew javaPreCommit` or `./gradlew build`
> reliably fail with a range of different
> <https://gist.github.com/ryan-williams/d9d3f7bd5f67c7c715e68ae4107aa4a0#file-1-javaprecommit-fails-L881-L903>
> JVM crashes
> <https://gist.github.com/ryan-williams/d9d3f7bd5f67c7c715e68ae4107aa4a0#file-3-compile-failures-L328-L329>
> in a few different tasks, with messages that suggest filing a bug
> against the Java compiler
>
> what do we know about the actual race condition that is allowing one
> task to attempt to read from a JAR that is being overwritten by
> another task? presumably this is just a bug in our Gradle configs?
>
> On Mon, Aug 27, 2018 at 2:28 PM Andrew Pilloud <apilloud@google.com
> <ma...@google.com>> wrote:
>
> It appears that there is no one working on a fix for the flakes,
> so I've merged the change to disable parallel tasks on precommit.
>
> Andrew
>
> On Fri, Aug 24, 2018 at 1:30 PM Andrew Pilloud
> <apilloud@google.com <ma...@google.com>> wrote:
>
> I'm seeing failures due to this on 12 of the last 16
> PostCommits. Precommits take about 22 minutes run in
> parallel, so at a 25% pass rate that puts the expected time
> to a good test run at 264 minutes assuming you immediately
> restart on each failure. We are looking at 56 minutes for a
> precommit that isn't run in parallel:
> https://builds.apache.org/job/beam_PreCommit_Java_Phrase/266/ I'd
> rather have tests take a little longer then have to monitor
> them for several hours.
>
> I've opened a PR: https://github.com/apache/beam/pull/6274
>
> Andrew
>
> On Fri, Aug 24, 2018 at 10:47 AM Lukasz Cwik
> <lcwik@google.com <ma...@google.com>> wrote:
>
> I believe it would mitigate the issue but also make the
> jobs take much longer to complete.
>
> On Thu, Aug 23, 2018 at 2:44 PM Andrew Pilloud
> <apilloud@google.com <ma...@google.com>> wrote:
>
> There seems to be a misconfiguration of gradle that
> is causing a high rate of failure for the last
> several weeks in building beam-examples-java and
> beam-runners-apex. It appears to be some sort of
> race condition in building dependencies. Given that
> no one has made progress on fixing the root cause,
> is this something we could mitigate by running jobs
> with `--no-parallel` flag?
>
> https://issues.apache.org/jira/browse/BEAM-5035
> https://issues.apache.org/jira/browse/BEAM-5207
>
> Andrew
>
Re: Gradle Races in beam-examples-java, beam-runners-apex
Posted by Lukasz Cwik <lc...@google.com>.
I had originally suggested to use some Linux kernel tooling such as
inotifywait[1] to watch what is happening.
It is likely that we have some Gradle task which is running something in
parallel to a different Gradle task when it shouldn't which means that the
jar file is being changed/corrupted. I believe fixing our Gradle task
dependency tree wrt to this would solve the problem. This crash does not
reproduce on my desktop after 20 runs which makes it hard for me to test
for.
1: https://www.linuxjournal.com/content/linux-filesystem-events-inotify
On Mon, Sep 10, 2018 at 1:15 PM Ryan Williams <ry...@runsascoded.com> wrote:
> this continues to be an issue locally (cf. some discussion in #beam slack)
>
> commands like `./gradlew javaPreCommit` or `./gradlew build` reliably fail
> with a range of different
> <https://gist.github.com/ryan-williams/d9d3f7bd5f67c7c715e68ae4107aa4a0#file-1-javaprecommit-fails-L881-L903> JVM
> crashes
> <https://gist.github.com/ryan-williams/d9d3f7bd5f67c7c715e68ae4107aa4a0#file-3-compile-failures-L328-L329>
> in a few different tasks, with messages that suggest filing a bug against
> the Java compiler
>
> what do we know about the actual race condition that is allowing one task
> to attempt to read from a JAR that is being overwritten by another task?
> presumably this is just a bug in our Gradle configs?
>
> On Mon, Aug 27, 2018 at 2:28 PM Andrew Pilloud <ap...@google.com>
> wrote:
>
>> It appears that there is no one working on a fix for the flakes, so I've
>> merged the change to disable parallel tasks on precommit.
>>
>> Andrew
>>
>> On Fri, Aug 24, 2018 at 1:30 PM Andrew Pilloud <ap...@google.com>
>> wrote:
>>
>>> I'm seeing failures due to this on 12 of the last 16 PostCommits.
>>> Precommits take about 22 minutes run in parallel, so at a 25% pass rate
>>> that puts the expected time to a good test run at 264 minutes assuming you
>>> immediately restart on each failure. We are looking at 56 minutes for a
>>> precommit that isn't run in parallel:
>>> https://builds.apache.org/job/beam_PreCommit_Java_Phrase/266/ I'd
>>> rather have tests take a little longer then have to monitor them for
>>> several hours.
>>>
>>> I've opened a PR: https://github.com/apache/beam/pull/6274
>>>
>>> Andrew
>>>
>>> On Fri, Aug 24, 2018 at 10:47 AM Lukasz Cwik <lc...@google.com> wrote:
>>>
>>>> I believe it would mitigate the issue but also make the jobs take much
>>>> longer to complete.
>>>>
>>>> On Thu, Aug 23, 2018 at 2:44 PM Andrew Pilloud <ap...@google.com>
>>>> wrote:
>>>>
>>>>> There seems to be a misconfiguration of gradle that is causing a high
>>>>> rate of failure for the last several weeks in building beam-examples-java
>>>>> and beam-runners-apex. It appears to be some sort of race condition in
>>>>> building dependencies. Given that no one has made progress on fixing the
>>>>> root cause, is this something we could mitigate by running jobs with
>>>>> `--no-parallel` flag?
>>>>>
>>>>> https://issues.apache.org/jira/browse/BEAM-5035
>>>>> https://issues.apache.org/jira/browse/BEAM-5207
>>>>>
>>>>> Andrew
>>>>>
>>>>
Re: Gradle Races in beam-examples-java, beam-runners-apex
Posted by Ryan Williams <ry...@runsascoded.com>.
this continues to be an issue locally (cf. some discussion in #beam slack)
commands like `./gradlew javaPreCommit` or `./gradlew build` reliably fail
with a range of different
<https://gist.github.com/ryan-williams/d9d3f7bd5f67c7c715e68ae4107aa4a0#file-1-javaprecommit-fails-L881-L903>
JVM
crashes
<https://gist.github.com/ryan-williams/d9d3f7bd5f67c7c715e68ae4107aa4a0#file-3-compile-failures-L328-L329>
in a few different tasks, with messages that suggest filing a bug against
the Java compiler
what do we know about the actual race condition that is allowing one task
to attempt to read from a JAR that is being overwritten by another task?
presumably this is just a bug in our Gradle configs?
On Mon, Aug 27, 2018 at 2:28 PM Andrew Pilloud <ap...@google.com> wrote:
> It appears that there is no one working on a fix for the flakes, so I've
> merged the change to disable parallel tasks on precommit.
>
> Andrew
>
> On Fri, Aug 24, 2018 at 1:30 PM Andrew Pilloud <ap...@google.com>
> wrote:
>
>> I'm seeing failures due to this on 12 of the last 16 PostCommits.
>> Precommits take about 22 minutes run in parallel, so at a 25% pass rate
>> that puts the expected time to a good test run at 264 minutes assuming you
>> immediately restart on each failure. We are looking at 56 minutes for a
>> precommit that isn't run in parallel:
>> https://builds.apache.org/job/beam_PreCommit_Java_Phrase/266/ I'd rather
>> have tests take a little longer then have to monitor them for several hours.
>>
>> I've opened a PR: https://github.com/apache/beam/pull/6274
>>
>> Andrew
>>
>> On Fri, Aug 24, 2018 at 10:47 AM Lukasz Cwik <lc...@google.com> wrote:
>>
>>> I believe it would mitigate the issue but also make the jobs take much
>>> longer to complete.
>>>
>>> On Thu, Aug 23, 2018 at 2:44 PM Andrew Pilloud <ap...@google.com>
>>> wrote:
>>>
>>>> There seems to be a misconfiguration of gradle that is causing a high
>>>> rate of failure for the last several weeks in building beam-examples-java
>>>> and beam-runners-apex. It appears to be some sort of race condition in
>>>> building dependencies. Given that no one has made progress on fixing the
>>>> root cause, is this something we could mitigate by running jobs with
>>>> `--no-parallel` flag?
>>>>
>>>> https://issues.apache.org/jira/browse/BEAM-5035
>>>> https://issues.apache.org/jira/browse/BEAM-5207
>>>>
>>>> Andrew
>>>>
>>>
Re: Gradle Races in beam-examples-java, beam-runners-apex
Posted by Andrew Pilloud <ap...@google.com>.
It appears that there is no one working on a fix for the flakes, so I've
merged the change to disable parallel tasks on precommit.
Andrew
On Fri, Aug 24, 2018 at 1:30 PM Andrew Pilloud <ap...@google.com> wrote:
> I'm seeing failures due to this on 12 of the last 16 PostCommits.
> Precommits take about 22 minutes run in parallel, so at a 25% pass rate
> that puts the expected time to a good test run at 264 minutes assuming you
> immediately restart on each failure. We are looking at 56 minutes for a
> precommit that isn't run in parallel:
> https://builds.apache.org/job/beam_PreCommit_Java_Phrase/266/ I'd rather
> have tests take a little longer then have to monitor them for several hours.
>
> I've opened a PR: https://github.com/apache/beam/pull/6274
>
> Andrew
>
> On Fri, Aug 24, 2018 at 10:47 AM Lukasz Cwik <lc...@google.com> wrote:
>
>> I believe it would mitigate the issue but also make the jobs take much
>> longer to complete.
>>
>> On Thu, Aug 23, 2018 at 2:44 PM Andrew Pilloud <ap...@google.com>
>> wrote:
>>
>>> There seems to be a misconfiguration of gradle that is causing a high
>>> rate of failure for the last several weeks in building beam-examples-java
>>> and beam-runners-apex. It appears to be some sort of race condition in
>>> building dependencies. Given that no one has made progress on fixing the
>>> root cause, is this something we could mitigate by running jobs with
>>> `--no-parallel` flag?
>>>
>>> https://issues.apache.org/jira/browse/BEAM-5035
>>> https://issues.apache.org/jira/browse/BEAM-5207
>>>
>>> Andrew
>>>
>>
Re: Gradle Races in beam-examples-java, beam-runners-apex
Posted by Andrew Pilloud <ap...@google.com>.
I'm seeing failures due to this on 12 of the last 16 PostCommits.
Precommits take about 22 minutes run in parallel, so at a 25% pass rate
that puts the expected time to a good test run at 264 minutes assuming you
immediately restart on each failure. We are looking at 56 minutes for a
precommit that isn't run in parallel:
https://builds.apache.org/job/beam_PreCommit_Java_Phrase/266/ I'd rather
have tests take a little longer then have to monitor them for several hours.
I've opened a PR: https://github.com/apache/beam/pull/6274
Andrew
On Fri, Aug 24, 2018 at 10:47 AM Lukasz Cwik <lc...@google.com> wrote:
> I believe it would mitigate the issue but also make the jobs take much
> longer to complete.
>
> On Thu, Aug 23, 2018 at 2:44 PM Andrew Pilloud <ap...@google.com>
> wrote:
>
>> There seems to be a misconfiguration of gradle that is causing a high
>> rate of failure for the last several weeks in building beam-examples-java
>> and beam-runners-apex. It appears to be some sort of race condition in
>> building dependencies. Given that no one has made progress on fixing the
>> root cause, is this something we could mitigate by running jobs with
>> `--no-parallel` flag?
>>
>> https://issues.apache.org/jira/browse/BEAM-5035
>> https://issues.apache.org/jira/browse/BEAM-5207
>>
>> Andrew
>>
>
Re: Gradle Races in beam-examples-java, beam-runners-apex
Posted by Lukasz Cwik <lc...@google.com>.
I believe it would mitigate the issue but also make the jobs take much
longer to complete.
On Thu, Aug 23, 2018 at 2:44 PM Andrew Pilloud <ap...@google.com> wrote:
> There seems to be a misconfiguration of gradle that is causing a high rate
> of failure for the last several weeks in building beam-examples-java and
> beam-runners-apex. It appears to be some sort of race condition in building
> dependencies. Given that no one has made progress on fixing the root cause,
> is this something we could mitigate by running jobs with `--no-parallel`
> flag?
>
> https://issues.apache.org/jira/browse/BEAM-5035
> https://issues.apache.org/jira/browse/BEAM-5207
>
> Andrew
>