You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Szymon Kuryło <sz...@gmail.com> on 2022/07/15 21:24:34 UTC

[Building] Building with JDK11

Hello,

I'm trying to build a Java 11 Spark distro using the
dev/make-distribution.sh script.
I have set JAVA_HOME to point to JDK11 location, I've also set the
java.version property in pom.xml to 11, effectively also setting
`maven.compile.source` and `maven.compile.target`.
When inspecting classes from the `dist` directory with `javap -v`, I find
that the class major version is 52, which is specific to JDK8. Am I missing
something? Is there a reliable way to set the JDK used in the build process?

Thanks,
Szymon K.

Re: [Building] Building with JDK11

Posted by Sean Owen <sr...@gmail.com>.

Why do you need Java 11 bytecode though?
Java 8 bytecode runs fine on Java 11. The settings in the build are really
there for testing, not because it's required to use Java 11.

On Mon, Jul 18, 2022 at 10:29 PM Gera Shegalov <ge...@apache.org> wrote:

> Bytecode version is controlled by javac "-target" option for Java, and by
> scalac "-target:" for Scala
> JDK can cross-compile between known versions.
>
> Spark uses 1.8 as source and target by default controlled by the Maven
> property java.version
> <https://github.com/apache/spark/blob/00d7094dc3024ae594605b311dcc55e95d277d5f/pom.xml#L109>.
> But it's also hard-coded with -target:jvm-1.8
> <https://github.com/apache/spark/blob/00d7094dc3024ae594605b311dcc55e95d277d5f/pom.xml#L2891>
> for Scala. Higher JDK versions can run lower version bytecode.
>
> if you want to try 11, replace occurences -target:jvm-1.8 by
> --target:jvm-${java.version} in pom.xml you should be able to produce 11
> bytecode by adding -Djava.version=11 to your Maven build command.
>
> ./build/mvn -Djava.version=11  ...
>
> However, I did not try beyond a quick compile on core and cannot say
> anything about fallout implications at run time.
>

Re: [Building] Building with JDK11

Posted by Gera Shegalov <ge...@apache.org>.

Bytecode version is controlled by javac "-target" option for Java, and by
scalac "-target:" for Scala
JDK can cross-compile between known versions.

Spark uses 1.8 as source and target by default controlled by the Maven
property java.version
<https://github.com/apache/spark/blob/00d7094dc3024ae594605b311dcc55e95d277d5f/pom.xml#L109>.
But it's also hard-coded with -target:jvm-1.8
<https://github.com/apache/spark/blob/00d7094dc3024ae594605b311dcc55e95d277d5f/pom.xml#L2891>
for Scala. Higher JDK versions can run lower version bytecode.

if you want to try 11, replace occurences -target:jvm-1.8 by
--target:jvm-${java.version} in pom.xml you should be able to produce 11
bytecode by adding -Djava.version=11 to your Maven build command.

./build/mvn -Djava.version=11  ...

However, I did not try beyond a quick compile on core and cannot say
anything about fallout implications at run time.

On Mon, Jul 18, 2022 at 3:49 PM Szymon Kuryło <sz...@gmail.com>
wrote:

> Ok, so I used a docker image `adoptopenjdk/openjdk11:latest` as a builder,
> and still got version 52 classes.
>
> pon., 18 lip 2022 o 09:51 Stephen Coy <sc...@infomedia.com.au> napisał(a):
>
>> Hi Sergey,
>>
>> I’m willing to be corrected, but I think the use of a JAVA_HOME
>> environment variable was something that was started by and continues to be
>> perpetuated by Apache Tomcat.
>>
>> … or maybe Apache Ant, but modern versions of Ant do not need it either.
>>
>> It is not needed for modern releases of Apache Maven.
>>
>> Cheers,
>>
>> Steve C
>>
>> On 18 Jul 2022, at 4:12 pm, Sergey B. <se...@gmail.com> wrote:
>>
>> Hi Steve,
>>
>> Can you shed some light why do they need $JAVA_HOME at all if
>> everything is already in place?
>>
>> Regards,
>> - Sergey
>>
>> On Mon, Jul 18, 2022 at 4:31 AM Stephen Coy <
>> scoy@infomedia.com.au.invalid> wrote:
>>
>>> Hi Szymon,
>>>
>>> There seems to be a common misconception that setting JAVA_HOME will set
>>> the version of Java that is used.
>>>
>>> This is not true, because in most environments you need to have a PATH
>>> environment variable set up that points at the version of Java that you
>>> want to use.
>>>
>>> You can set JAVA_HOME to anything at all and `java -version` will always
>>> return the same result.
>>>
>>> The way that you configure PATH varies from OS to OS:
>>>
>>>
>>>    - MacOS use `/usr/libexec/java_home -v11`
>>>    - On linux use `sudo alternatives --config java`
>>>    - On Windows I have no idea
>>>
>>>
>>> When you do this the `mvn` command will compute the value of JAVA_HOME
>>> for its own use; there is no need to explicitly set it yourself (unless
>>> other Java applications that you use insist u[on it).
>>>
>>>
>>> Cheers,
>>>
>>> Steve C
>>>
>>> On 16 Jul 2022, at 7:24 am, Szymon Kuryło <sz...@gmail.com>
>>> wrote:
>>>
>>> Hello,
>>>
>>> I'm trying to build a Java 11 Spark distro using the
>>> dev/make-distribution.sh script.
>>> I have set JAVA_HOME to point to JDK11 location, I've also set the
>>> java.version property in pom.xml to 11, effectively also setting
>>> `maven.compile.source` and `maven.compile.target`.
>>> When inspecting classes from the `dist` directory with `javap -v`, I
>>> find that the class major version is 52, which is specific to JDK8. Am
>>> I missing something? Is there a reliable way to set the JDK used in the
>>> build process?
>>>
>>> Thanks,
>>> Szymon K.
>>>
>>>
>>> This email contains confidential information of and is the copyright of
>>> Infomedia. It must not be forwarded, amended or disclosed without consent
>>> of the sender. If you received this message by mistake, please advise the
>>> sender and delete all copies. Security of transmission on the internet
>>> cannot be guaranteed, could be infected, intercepted, or corrupted and you
>>> should ensure you have suitable antivirus protection in place. By sending
>>> us your or any third party personal details, you consent to (or confirm you
>>> have obtained consent from such third parties) to Infomedia’s privacy
>>> policy. http://www.infomedia.com.au/privacy-policy/
>>>
>>
>>

Re: [Building] Building with JDK11

Posted by Szymon Kuryło <sz...@gmail.com>.

Ok, so I used a docker image `adoptopenjdk/openjdk11:latest` as a builder,
and still got version 52 classes.

pon., 18 lip 2022 o 09:51 Stephen Coy <sc...@infomedia.com.au> napisał(a):

> Hi Sergey,
>
> I’m willing to be corrected, but I think the use of a JAVA_HOME
> environment variable was something that was started by and continues to be
> perpetuated by Apache Tomcat.
>
> … or maybe Apache Ant, but modern versions of Ant do not need it either.
>
> It is not needed for modern releases of Apache Maven.
>
> Cheers,
>
> Steve C
>
> On 18 Jul 2022, at 4:12 pm, Sergey B. <se...@gmail.com> wrote:
>
> Hi Steve,
>
> Can you shed some light why do they need $JAVA_HOME at all if
> everything is already in place?
>
> Regards,
> - Sergey
>
> On Mon, Jul 18, 2022 at 4:31 AM Stephen Coy <sc...@infomedia.com.au.invalid>
> wrote:
>
>> Hi Szymon,
>>
>> There seems to be a common misconception that setting JAVA_HOME will set
>> the version of Java that is used.
>>
>> This is not true, because in most environments you need to have a PATH
>> environment variable set up that points at the version of Java that you
>> want to use.
>>
>> You can set JAVA_HOME to anything at all and `java -version` will always
>> return the same result.
>>
>> The way that you configure PATH varies from OS to OS:
>>
>>
>>    - MacOS use `/usr/libexec/java_home -v11`
>>    - On linux use `sudo alternatives --config java`
>>    - On Windows I have no idea
>>
>>
>> When you do this the `mvn` command will compute the value of JAVA_HOME
>> for its own use; there is no need to explicitly set it yourself (unless
>> other Java applications that you use insist u[on it).
>>
>>
>> Cheers,
>>
>> Steve C
>>
>> On 16 Jul 2022, at 7:24 am, Szymon Kuryło <sz...@gmail.com> wrote:
>>
>> Hello,
>>
>> I'm trying to build a Java 11 Spark distro using the
>> dev/make-distribution.sh script.
>> I have set JAVA_HOME to point to JDK11 location, I've also set the
>> java.version property in pom.xml to 11, effectively also setting
>> `maven.compile.source` and `maven.compile.target`.
>> When inspecting classes from the `dist` directory with `javap -v`, I find
>> that the class major version is 52, which is specific to JDK8. Am I missing
>> something? Is there a reliable way to set the JDK used in the build process?
>>
>> Thanks,
>> Szymon K.
>>
>>
>> This email contains confidential information of and is the copyright of
>> Infomedia. It must not be forwarded, amended or disclosed without consent
>> of the sender. If you received this message by mistake, please advise the
>> sender and delete all copies. Security of transmission on the internet
>> cannot be guaranteed, could be infected, intercepted, or corrupted and you
>> should ensure you have suitable antivirus protection in place. By sending
>> us your or any third party personal details, you consent to (or confirm you
>> have obtained consent from such third parties) to Infomedia’s privacy
>> policy. http://www.infomedia.com.au/privacy-policy/
>>
>
>

Re: [Building] Building with JDK11

Posted by Stephen Coy <sc...@infomedia.com.au.INVALID>.

Hi Sergey,

I’m willing to be corrected, but I think the use of a JAVA_HOME environment variable was something that was started by and continues to be perpetuated by Apache Tomcat.

… or maybe Apache Ant, but modern versions of Ant do not need it either.

It is not needed for modern releases of Apache Maven.

Cheers,

Steve C

On 18 Jul 2022, at 4:12 pm, Sergey B. <se...@gmail.com>> wrote:

Hi Steve,

Can you shed some light why do they need $JAVA_HOME at all if everything is already in place?

Regards,
- Sergey

On Mon, Jul 18, 2022 at 4:31 AM Stephen Coy <sc...@infomedia.com.au.invalid>> wrote:
Hi Szymon,

There seems to be a common misconception that setting JAVA_HOME will set the version of Java that is used.

This is not true, because in most environments you need to have a PATH environment variable set up that points at the version of Java that you want to use.

You can set JAVA_HOME to anything at all and `java -version` will always return the same result.

The way that you configure PATH varies from OS to OS:

* MacOS use `/usr/libexec/java_home -v11`
* On linux use `sudo alternatives --config java`
* On Windows I have no idea

When you do this the `mvn` command will compute the value of JAVA_HOME for its own use; there is no need to explicitly set it yourself (unless other Java applications that you use insist u[on it).

Cheers,

Steve C

On 16 Jul 2022, at 7:24 am, Szymon Kuryło <sz...@gmail.com>> wrote:

Hello,

I'm trying to build a Java 11 Spark distro using the dev/make-distribution.sh script.
I have set JAVA_HOME to point to JDK11 location, I've also set the java.version property in pom.xml to 11, effectively also setting `maven.compile.source` and `maven.compile.target`.
When inspecting classes from the `dist` directory with `javap -v`, I find that the class major version is 52, which is specific to JDK8. Am I missing something? Is there a reliable way to set the JDK used in the build process?

Thanks,
Szymon K.

This email contains confidential information of and is the copyright of Infomedia. It must not be forwarded, amended or disclosed without consent of the sender. If you received this message by mistake, please advise the sender and delete all copies. Security of transmission on the internet cannot be guaranteed, could be infected, intercepted, or corrupted and you should ensure you have suitable antivirus protection in place. By sending us your or any third party personal details, you consent to (or confirm you have obtained consent from such third parties) to Infomedia’s privacy policy. http://www.infomedia.com.au/privacy-policy/

Re: [Building] Building with JDK11

Posted by "Sergey B." <se...@gmail.com>.

Hi Steve,

Can you shed some light why do they need $JAVA_HOME at all if everything is
already in place?

Regards,
- Sergey

On Mon, Jul 18, 2022 at 4:31 AM Stephen Coy <sc...@infomedia.com.au.invalid>
wrote:

> Hi Szymon,
>
> There seems to be a common misconception that setting JAVA_HOME will set
> the version of Java that is used.
>
> This is not true, because in most environments you need to have a PATH
> environment variable set up that points at the version of Java that you
> want to use.
>
> You can set JAVA_HOME to anything at all and `java -version` will always
> return the same result.
>
> The way that you configure PATH varies from OS to OS:
>
>
>    - MacOS use `/usr/libexec/java_home -v11`
>    - On linux use `sudo alternatives --config java`
>    - On Windows I have no idea
>
>
> When you do this the `mvn` command will compute the value of JAVA_HOME for
> its own use; there is no need to explicitly set it yourself (unless other
> Java applications that you use insist u[on it).
>
>
> Cheers,
>
> Steve C
>
> On 16 Jul 2022, at 7:24 am, Szymon Kuryło <sz...@gmail.com> wrote:
>
> Hello,
>
> I'm trying to build a Java 11 Spark distro using the
> dev/make-distribution.sh script.
> I have set JAVA_HOME to point to JDK11 location, I've also set the
> java.version property in pom.xml to 11, effectively also setting
> `maven.compile.source` and `maven.compile.target`.
> When inspecting classes from the `dist` directory with `javap -v`, I find
> that the class major version is 52, which is specific to JDK8. Am I missing
> something? Is there a reliable way to set the JDK used in the build process?
>
> Thanks,
> Szymon K.
>
>
> This email contains confidential information of and is the copyright of
> Infomedia. It must not be forwarded, amended or disclosed without consent
> of the sender. If you received this message by mistake, please advise the
> sender and delete all copies. Security of transmission on the internet
> cannot be guaranteed, could be infected, intercepted, or corrupted and you
> should ensure you have suitable antivirus protection in place. By sending
> us your or any third party personal details, you consent to (or confirm you
> have obtained consent from such third parties) to Infomedia’s privacy
> policy. http://www.infomedia.com.au/privacy-policy/
>

Re: [Building] Building with JDK11

Posted by Stephen Coy <sc...@infomedia.com.au.INVALID>.

Hi Szymon,

There seems to be a common misconception that setting JAVA_HOME will set the version of Java that is used.

This is not true, because in most environments you need to have a PATH environment variable set up that points at the version of Java that you want to use.

You can set JAVA_HOME to anything at all and `java -version` will always return the same result.

The way that you configure PATH varies from OS to OS:

* MacOS use `/usr/libexec/java_home -v11`
* On linux use `sudo alternatives --config java`
* On Windows I have no idea

When you do this the `mvn` command will compute the value of JAVA_HOME for its own use; there is no need to explicitly set it yourself (unless other Java applications that you use insist u[on it).

Cheers,

Steve C

On 16 Jul 2022, at 7:24 am, Szymon Kuryło <sz...@gmail.com>> wrote:

Hello,

Thanks,
Szymon K.

Re: [Building] Building with JDK11

Posted by Tufan Rakshit <tu...@gmail.com>.

maybe try intellij or some other IDE with SBT . Maven has been always
magical for me

Best
Tufan

On Sat, 16 Jul 2022 at 00:11, Sean Owen <sr...@gmail.com> wrote:

> Java 8 binaries are probably on your PATH
>
> On Fri, Jul 15, 2022, 5:01 PM Szymon Kuryło <sz...@gmail.com>
> wrote:
>
>> Hello,
>>
>> I'm trying to build a Java 11 Spark distro using the
>> dev/make-distribution.sh script.
>> I have set JAVA_HOME to point to JDK11 location, I've also set the
>> java.version property in pom.xml to 11, effectively also setting
>> `maven.compile.source` and `maven.compile.target`.
>> When inspecting classes from the `dist` directory with `javap -v`, I find
>> that the class major version is 52, which is specific to JDK8. Am I missing
>> something? Is there a reliable way to set the JDK used in the build process?
>>
>> Thanks,
>> Szymon K.
>>
>

Re: [Building] Building with JDK11

Posted by Sean Owen <sr...@gmail.com>.

Java 8 binaries are probably on your PATH

On Fri, Jul 15, 2022, 5:01 PM Szymon Kuryło <sz...@gmail.com> wrote:

> Hello,
>
> I'm trying to build a Java 11 Spark distro using the
> dev/make-distribution.sh script.
> I have set JAVA_HOME to point to JDK11 location, I've also set the
> java.version property in pom.xml to 11, effectively also setting
> `maven.compile.source` and `maven.compile.target`.
> When inspecting classes from the `dist` directory with `javap -v`, I find
> that the class major version is 52, which is specific to JDK8. Am I missing
> something? Is there a reliable way to set the JDK used in the build process?
>
> Thanks,
> Szymon K.
>