You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Jean-Baptiste Onofré <jb...@nanthrax.net> on 2018/02/09 10:35:04 UTC
[INFO] Gradle build is flaky on Jenkins
Hi guys,
I noticed that the Gradle build on Jenkins is flaky: it almost always fails for
different reason (I can't download pentaho artifact sometime, interrupted other
times).
Jenkins is doing:
Jenkins: ./gradlew --continue --rerun-tasks :javaPreCommit
I gonna investigate how to improve the situation.
Regards
JB
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: [INFO] Gradle build is flaky on Jenkins
Posted by Kenneth Knowles <kl...@google.com>.
Separately, we have many tests that seem to have been passing by luck and
Gradle exposes some of their weaknesses by its increased parallelism. They
are filed under the "flake" and later the "sickbay" tags. I've put together
a saved search: https://issues.apache.org/jira/issues/?filter=12343195
These may be good starter tasks that don't require deep Beam knowledge.
Kenn
On Fri, Feb 9, 2018 at 7:04 AM, Kenneth Knowles <kl...@google.com> wrote:
> I have been hitting these issues and investigating them.
>
> - HCatalog pentaho dependency: https://issues.apache.org/
> jira/browse/BEAM-3621
> - We rerun way way more than we need to: https://issues.apache.org/
> jira/browse/BEAM-3253 but dependency stuff occurs during config, not
> tasks, I think
>
> The biggest source of flakes with Maven is dependency download. We only
> got any green with maven once we saved .m2 across builds. It seems like
> gradle is in that state, and is not caching them. I'm not sure why - by
> default it should cache to $HOME/.gradle. I think some versions of the
> Jenkins plugin caches to $WORKSPACE/.gradle but that should also work just
> fine unless we are wiping out the workspace, which we shouldn't do as we
> have isolated the build to the $WORKSPACE/src directory. When I look in a
> workspace I don't see a cache anyhow.
>
> There's an issue with the Jenkins workers so I can't ssh to them at the
> moment to investigate. I am hoping that https://github.com/apache/
> beam/pull/4644 will also make it easier to see what is happening.
>
> Kenn
>
> On Fri, Feb 9, 2018 at 5:29 AM, Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
>> Hi Romain,
>>
>> that's actually part of my investigations ;)
>>
>> Regards
>> JB
>>
>> On 02/09/2018 02:26 PM, Romain Manni-Bucau wrote:
>> > did you check the concurrency? For me it is plain wrong since it is
>> hardcoded in
>> > the build file and doesnt let me or the tool customize it. This means
>> it just
>> > blocks my computer in general and makes some timeout related tests fail
>> easily.
>> > (this kind of config should always be customized on the CI and never
>> hardcoded IMHO)
>> >
>> >
>> > Romain Manni-Bucau
>> > @rmannibucau <https://twitter.com/rmannibucau> | Blog
>> > <https://rmannibucau.metawerx.net/> | Old Blog
>> > <http://rmannibucau.wordpress.com> | Github <
>> https://github.com/rmannibucau> |
>> > LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
>> > <https://www.packtpub.com/application-development/java-ee-8-
>> high-performance>
>> >
>> > 2018-02-09 12:00 GMT+01:00 Aljoscha Krettek <aljoscha@apache.org
>> > <ma...@apache.org>>:
>> >
>> > Yes, I was just about to write about this as well. In my recent PRs
>> this
>> > always failed for different reasons.
>> >
>> > Thanks for looking into this!
>> >
>> > > On 9. Feb 2018, at 11:35, Jean-Baptiste Onofré <jb@nanthrax.net
>> > <ma...@nanthrax.net>> wrote:
>> > >
>> > > Hi guys,
>> > >
>> > > I noticed that the Gradle build on Jenkins is flaky: it almost
>> always
>> > fails for
>> > > different reason (I can't download pentaho artifact sometime,
>> interrupted
>> > other
>> > > times).
>> > >
>> > > Jenkins is doing:
>> > >
>> > > Jenkins: ./gradlew --continue --rerun-tasks :javaPreCommit
>> > >
>> > > I gonna investigate how to improve the situation.
>> > >
>> > > Regards
>> > > JB
>> > > --
>> > > Jean-Baptiste Onofré
>> > > jbonofre@apache.org <ma...@apache.org>
>> > > http://blog.nanthrax.net
>> > > Talend - http://www.talend.com
>> >
>> >
>>
>> --
>> Jean-Baptiste Onofré
>> jbonofre@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>>
>
>
Re: [INFO] Gradle build is flaky on Jenkins
Posted by Kenneth Knowles <kl...@google.com>.
I have been hitting these issues and investigating them.
- HCatalog pentaho dependency:
https://issues.apache.org/jira/browse/BEAM-3621
- We rerun way way more than we need to:
https://issues.apache.org/jira/browse/BEAM-3253 but dependency stuff occurs
during config, not tasks, I think
The biggest source of flakes with Maven is dependency download. We only got
any green with maven once we saved .m2 across builds. It seems like gradle
is in that state, and is not caching them. I'm not sure why - by default it
should cache to $HOME/.gradle. I think some versions of the Jenkins plugin
caches to $WORKSPACE/.gradle but that should also work just fine unless we
are wiping out the workspace, which we shouldn't do as we have isolated the
build to the $WORKSPACE/src directory. When I look in a workspace I don't
see a cache anyhow.
There's an issue with the Jenkins workers so I can't ssh to them at the
moment to investigate. I am hoping that
https://github.com/apache/beam/pull/4644 will also make it easier to see
what is happening.
Kenn
On Fri, Feb 9, 2018 at 5:29 AM, Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> Hi Romain,
>
> that's actually part of my investigations ;)
>
> Regards
> JB
>
> On 02/09/2018 02:26 PM, Romain Manni-Bucau wrote:
> > did you check the concurrency? For me it is plain wrong since it is
> hardcoded in
> > the build file and doesnt let me or the tool customize it. This means it
> just
> > blocks my computer in general and makes some timeout related tests fail
> easily.
> > (this kind of config should always be customized on the CI and never
> hardcoded IMHO)
> >
> >
> > Romain Manni-Bucau
> > @rmannibucau <https://twitter.com/rmannibucau> | Blog
> > <https://rmannibucau.metawerx.net/> | Old Blog
> > <http://rmannibucau.wordpress.com> | Github <https://github.com/
> rmannibucau> |
> > LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
> > <https://www.packtpub.com/application-development/java-
> ee-8-high-performance>
> >
> > 2018-02-09 12:00 GMT+01:00 Aljoscha Krettek <aljoscha@apache.org
> > <ma...@apache.org>>:
> >
> > Yes, I was just about to write about this as well. In my recent PRs
> this
> > always failed for different reasons.
> >
> > Thanks for looking into this!
> >
> > > On 9. Feb 2018, at 11:35, Jean-Baptiste Onofré <jb@nanthrax.net
> > <ma...@nanthrax.net>> wrote:
> > >
> > > Hi guys,
> > >
> > > I noticed that the Gradle build on Jenkins is flaky: it almost
> always
> > fails for
> > > different reason (I can't download pentaho artifact sometime,
> interrupted
> > other
> > > times).
> > >
> > > Jenkins is doing:
> > >
> > > Jenkins: ./gradlew --continue --rerun-tasks :javaPreCommit
> > >
> > > I gonna investigate how to improve the situation.
> > >
> > > Regards
> > > JB
> > > --
> > > Jean-Baptiste Onofré
> > > jbonofre@apache.org <ma...@apache.org>
> > > http://blog.nanthrax.net
> > > Talend - http://www.talend.com
> >
> >
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
Re: [INFO] Gradle build is flaky on Jenkins
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi Romain,
that's actually part of my investigations ;)
Regards
JB
On 02/09/2018 02:26 PM, Romain Manni-Bucau wrote:
> did you check the concurrency? For me it is plain wrong since it is hardcoded in
> the build file and doesnt let me or the tool customize it. This means it just
> blocks my computer in general and makes some timeout related tests fail easily.
> (this kind of config should always be customized on the CI and never hardcoded IMHO)
>
>
> Romain Manni-Bucau
> @rmannibucau <https://twitter.com/rmannibucau> | Blog
> <https://rmannibucau.metawerx.net/> | Old Blog
> <http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
> LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>
> 2018-02-09 12:00 GMT+01:00 Aljoscha Krettek <aljoscha@apache.org
> <ma...@apache.org>>:
>
> Yes, I was just about to write about this as well. In my recent PRs this
> always failed for different reasons.
>
> Thanks for looking into this!
>
> > On 9. Feb 2018, at 11:35, Jean-Baptiste Onofré <jb@nanthrax.net
> <ma...@nanthrax.net>> wrote:
> >
> > Hi guys,
> >
> > I noticed that the Gradle build on Jenkins is flaky: it almost always
> fails for
> > different reason (I can't download pentaho artifact sometime, interrupted
> other
> > times).
> >
> > Jenkins is doing:
> >
> > Jenkins: ./gradlew --continue --rerun-tasks :javaPreCommit
> >
> > I gonna investigate how to improve the situation.
> >
> > Regards
> > JB
> > --
> > Jean-Baptiste Onofré
> > jbonofre@apache.org <ma...@apache.org>
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
>
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: [INFO] Gradle build is flaky on Jenkins
Posted by Romain Manni-Bucau <rm...@gmail.com>.
did you check the concurrency? For me it is plain wrong since it is
hardcoded in the build file and doesnt let me or the tool customize it.
This means it just blocks my computer in general and makes some timeout
related tests fail easily. (this kind of config should always be customized
on the CI and never hardcoded IMHO)
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<https://rmannibucau.metawerx.net/> | Old Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
<https://www.packtpub.com/application-development/java-ee-8-high-performance>
2018-02-09 12:00 GMT+01:00 Aljoscha Krettek <al...@apache.org>:
> Yes, I was just about to write about this as well. In my recent PRs this
> always failed for different reasons.
>
> Thanks for looking into this!
>
> > On 9. Feb 2018, at 11:35, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
> >
> > Hi guys,
> >
> > I noticed that the Gradle build on Jenkins is flaky: it almost always
> fails for
> > different reason (I can't download pentaho artifact sometime,
> interrupted other
> > times).
> >
> > Jenkins is doing:
> >
> > Jenkins: ./gradlew --continue --rerun-tasks :javaPreCommit
> >
> > I gonna investigate how to improve the situation.
> >
> > Regards
> > JB
> > --
> > Jean-Baptiste Onofré
> > jbonofre@apache.org
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
>
>
Re: [INFO] Gradle build is flaky on Jenkins
Posted by Aljoscha Krettek <al...@apache.org>.
Yes, I was just about to write about this as well. In my recent PRs this always failed for different reasons.
Thanks for looking into this!
> On 9. Feb 2018, at 11:35, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>
> Hi guys,
>
> I noticed that the Gradle build on Jenkins is flaky: it almost always fails for
> different reason (I can't download pentaho artifact sometime, interrupted other
> times).
>
> Jenkins is doing:
>
> Jenkins: ./gradlew --continue --rerun-tasks :javaPreCommit
>
> I gonna investigate how to improve the situation.
>
> Regards
> JB
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com