Updating Spark PR builder and 2.x test jobs to use Java 8 JDK

[Josh Rosen <jo...@databricks.com>]

In order to be able to run Java 8 API compatibility tests, I'm going to
push a new set of Jenkins configurations for Spark's test and PR builders
so that those jobs use a Java 8 JDK. I tried this once in the past and it
seemed to introduce some rare, transient flakiness in certain tests, so if
anyone observes new test failures please email me and I'll investigate
right away.

Note that this change has no impact on Spark's supported JDK versions and
our build will still target Java 7 and emit Java 7 bytecode; the purpose of
this change is simply to allow the Java 8 lambda tests to be run as part of
PR builder runs.

- Josh

Re: Updating Spark PR builder and 2.x test jobs to use Java 8 JDK

[Josh Rosen <jo...@databricks.com>]

I finally figured out the problem: it seems that my *export
JAVA_HOME=/path/to/java8/home* was somehow not affecting the javac
executable that Zinc's SBT incremental compiler uses when it forks out to
javac to handle Java source files. As a result, we were passing a -source
1.8 flag to the platform's default javac, which happens to be Java 7.

To fix this, I'm going to modify the build to just prepend $JAVA_HOME/bin
to $PATH while setting up the test environment

On Tue, Apr 5, 2016 at 5:09 PM Josh Rosen <jo...@databricks.com> wrote:

> I've reverted the bulk of the conf changes while I investigate. I think
> that Zinc might be handling JAVA_HOME in a weird way and am SSH'ing to
> Jenkins to try to reproduce the problem in isolation.
>
> On Tue, Apr 5, 2016 at 4:14 PM Ted Yu <yu...@gmail.com> wrote:
>
>> Josh:
>> You may have noticed the following error (
>> https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-maven-hadoop-2.7/566/console
>> ):
>>
>> [error] javac: invalid source release: 1.8
>> [error] Usage: javac <options> <source files>
>> [error] use -help for a list of possible options
>>
>>
>> On Tue, Apr 5, 2016 at 2:14 PM, Josh Rosen <jo...@databricks.com>
>> wrote:
>>
>>> In order to be able to run Java 8 API compatibility tests, I'm going to
>>> push a new set of Jenkins configurations for Spark's test and PR builders
>>> so that those jobs use a Java 8 JDK. I tried this once in the past and it
>>> seemed to introduce some rare, transient flakiness in certain tests, so if
>>> anyone observes new test failures please email me and I'll investigate
>>> right away.
>>>
>>> Note that this change has no impact on Spark's supported JDK versions
>>> and our build will still target Java 7 and emit Java 7 bytecode; the
>>> purpose of this change is simply to allow the Java 8 lambda tests to be run
>>> as part of PR builder runs.
>>>
>>> - Josh
>>>
>>
>>

Re: Updating Spark PR builder and 2.x test jobs to use Java 8 JDK

[Josh Rosen <jo...@databricks.com>]

I've reverted the bulk of the conf changes while I investigate. I think
that Zinc might be handling JAVA_HOME in a weird way and am SSH'ing to
Jenkins to try to reproduce the problem in isolation.

On Tue, Apr 5, 2016 at 4:14 PM Ted Yu <yu...@gmail.com> wrote:

> Josh:
> You may have noticed the following error (
> https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-maven-hadoop-2.7/566/console
> ):
>
> [error] javac: invalid source release: 1.8
> [error] Usage: javac <options> <source files>
> [error] use -help for a list of possible options
>
>
> On Tue, Apr 5, 2016 at 2:14 PM, Josh Rosen <jo...@databricks.com>
> wrote:
>
>> In order to be able to run Java 8 API compatibility tests, I'm going to
>> push a new set of Jenkins configurations for Spark's test and PR builders
>> so that those jobs use a Java 8 JDK. I tried this once in the past and it
>> seemed to introduce some rare, transient flakiness in certain tests, so if
>> anyone observes new test failures please email me and I'll investigate
>> right away.
>>
>> Note that this change has no impact on Spark's supported JDK versions and
>> our build will still target Java 7 and emit Java 7 bytecode; the purpose of
>> this change is simply to allow the Java 8 lambda tests to be run as part of
>> PR builder runs.
>>
>> - Josh
>>
>
>

Re: Updating Spark PR builder and 2.x test jobs to use Java 8 JDK

[Ted Yu <yu...@gmail.com>]

Josh:
You may have noticed the following error (
https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-maven-hadoop-2.7/566/console
):

[error] javac: invalid source release: 1.8
[error] Usage: javac <options> <source files>
[error] use -help for a list of possible options


On Tue, Apr 5, 2016 at 2:14 PM, Josh Rosen <jo...@databricks.com> wrote:

> In order to be able to run Java 8 API compatibility tests, I'm going to
> push a new set of Jenkins configurations for Spark's test and PR builders
> so that those jobs use a Java 8 JDK. I tried this once in the past and it
> seemed to introduce some rare, transient flakiness in certain tests, so if
> anyone observes new test failures please email me and I'll investigate
> right away.
>
> Note that this change has no impact on Spark's supported JDK versions and
> our build will still target Java 7 and emit Java 7 bytecode; the purpose of
> this change is simply to allow the Java 8 lambda tests to be run as part of
> PR builder runs.
>
> - Josh
>