You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Jungtaek Lim <ka...@gmail.com> on 2019/09/17 02:13:13 UTC
Weird build failures in PR builder
Hi devs,
I'm seeing some build failures regarding generating documentation - they
all logged below error message:
[error] /home/jenkins/workspace/SparkPullRequestBuilder@6/core/src/main/scala/org/apache/spark/serializer/GenericAvroSerializer.scala:123:
> value createDatumWriter is not a member of
> org.apache.avro.generic.GenericData
> [error] writerCache.getOrElseUpdate(schema,
> GenericData.get.createDatumWriter(schema))
> [error] ^
> [info] No documentation generated with unsuccessful compiler run
> [error] one error found
But GenericData.createDatumWriter exists (and also compilation doesn't
fail), as well as I succeeded to run "sbt unidoc" with same option.
Note that they're different workers (amp-jenkins-worker-0x) with same
workspace /home/jenkins/workspace/SparkPullRequestBuilder@6.
Below links are the build links failed due to above reason:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110713/consoleFull
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110712/consoleFull
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110710/consoleFull
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110706/consoleFull
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110697/consoleFull
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110696/consoleFull
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110688/consoleFull
Any guess/suspect?
Thanks,
Jungtaek Lim (HeartSaVioR)
Re: Weird build failures in PR builder
Posted by Shane Knapp <sk...@berkeley.edu>.
yeah, it could be completely unrelated to the build env, but it's late
and i'm not going to dive in to this tonite. :)
On Mon, Sep 16, 2019 at 9:13 PM Jungtaek Lim <ka...@gmail.com> wrote:
>
> Thanks for following up and providing the info! I'll also keep an eye after clean-up to see whether it helps.
>
> On Tue, Sep 17, 2019 at 1:08 PM Shane Knapp <sk...@berkeley.edu> wrote:
>>
>> > Note that they're different workers (amp-jenkins-worker-0x) with same workspace /home/jenkins/workspace/SparkPullRequestBuilder@6.
>> >
>> the '@6' refers to the executor number that the build ran on (look at
>> the jenkins ui and the numbers next to each slot for each worker).
>>
>> oddly, at the beginning of the build when we overload HOME to point to
>> the .ivy dir, the executor doesn't seem to match up with what
>> per-executor dir is being set to. example:
>>
>> https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110688/consoleFull
>> executor number is 8 (from
>> https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110688/injectedEnvVars)
>> HOME is set properly: + HOME=/home/sparkivy/per-executor-caches/8
>> but the dir the build runs from is indeed
>> /home/jenkins/workspace/SparkPullRequestBuilder@6/
>>
>> > Any guess/suspect?
>> >
>> nope. no clue whatsoever. ¯\_(ツ)_/¯
>>
>> what i CAN do first thing tomorrow is pause builds, wipe out the
>> ivy/sbt caches and SparkPullRequestBuilder* dirs on all workers and
>> see if that helps.
>>
>> shane
>> --
>> Shane Knapp
>> UC Berkeley EECS Research / RISELab Staff Technical Lead
>> https://rise.cs.berkeley.edu
>
>
>
> --
> Name : Jungtaek Lim
> Blog : http://medium.com/@heartsavior
> Twitter : http://twitter.com/heartsavior
> LinkedIn : http://www.linkedin.com/in/heartsavior
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Weird build failures in PR builder
Posted by Jungtaek Lim <ka...@gmail.com>.
Thanks for following up and providing the info! I'll also keep an eye after
clean-up to see whether it helps.
On Tue, Sep 17, 2019 at 1:08 PM Shane Knapp <sk...@berkeley.edu> wrote:
> > Note that they're different workers (amp-jenkins-worker-0x) with same
> workspace /home/jenkins/workspace/SparkPullRequestBuilder@6.
> >
> the '@6' refers to the executor number that the build ran on (look at
> the jenkins ui and the numbers next to each slot for each worker).
>
> oddly, at the beginning of the build when we overload HOME to point to
> the .ivy dir, the executor doesn't seem to match up with what
> per-executor dir is being set to. example:
>
>
> https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110688/consoleFull
> executor number is 8 (from
>
> https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110688/injectedEnvVars
> )
> HOME is set properly: + HOME=/home/sparkivy/per-executor-caches/8
> but the dir the build runs from is indeed
> /home/jenkins/workspace/SparkPullRequestBuilder@6/
>
> > Any guess/suspect?
> >
> nope. no clue whatsoever. ¯\_(ツ)_/¯
>
> what i CAN do first thing tomorrow is pause builds, wipe out the
> ivy/sbt caches and SparkPullRequestBuilder* dirs on all workers and
> see if that helps.
>
> shane
> --
> Shane Knapp
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu
>
--
Name : Jungtaek Lim
Blog : http://medium.com/@heartsavior
Twitter : http://twitter.com/heartsavior
LinkedIn : http://www.linkedin.com/in/heartsavior
Re: Weird build failures in PR builder
Posted by Shane Knapp <sk...@berkeley.edu>.
> Note that they're different workers (amp-jenkins-worker-0x) with same workspace /home/jenkins/workspace/SparkPullRequestBuilder@6.
>
the '@6' refers to the executor number that the build ran on (look at
the jenkins ui and the numbers next to each slot for each worker).
oddly, at the beginning of the build when we overload HOME to point to
the .ivy dir, the executor doesn't seem to match up with what
per-executor dir is being set to. example:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110688/consoleFull
executor number is 8 (from
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/110688/injectedEnvVars)
HOME is set properly: + HOME=/home/sparkivy/per-executor-caches/8
but the dir the build runs from is indeed
/home/jenkins/workspace/SparkPullRequestBuilder@6/
> Any guess/suspect?
>
nope. no clue whatsoever. ¯\_(ツ)_/¯
what i CAN do first thing tomorrow is pause builds, wipe out the
ivy/sbt caches and SparkPullRequestBuilder* dirs on all workers and
see if that helps.
shane
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org