You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2023/03/06 19:06:42 UTC
[spark] branch branch-3.3 updated: [SPARK-42673][BUILD] Make `build/mvn` build Spark only with the verified maven version
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.3 by this push:
new 210ed7d6d48 [SPARK-42673][BUILD] Make `build/mvn` build Spark only with the verified maven version
210ed7d6d48 is described below
commit 210ed7d6d4808ba903d0ab6d441528f63ac7a41d
Author: yangjie01 <ya...@baidu.com>
AuthorDate: Mon Mar 6 11:05:42 2023 -0800
[SPARK-42673][BUILD] Make `build/mvn` build Spark only with the verified maven version
### What changes were proposed in this pull request?
`build/mvn` tends to use the new maven version to build Spark now, and GA starts to use 3.9.0 as the default maven version.
But there may be some uncertain factors when building Spark with unverified version.
For example, `java-11-17` GA build task build with maven 3.9.0 has many error logs in master like follow:
```
Error: [ERROR] An error occurred attempting to read POM
org.codehaus.plexus.util.xml.pull.XmlPullParserException: UTF-8 BOM plus xml decl of ISO-8859-1 is incompatible (position: START_DOCUMENT seen <?xml version="1.0" encoding="ISO-8859-1"... 1:42)
at org.codehaus.plexus.util.xml.pull.MXParser.parseXmlDeclWithVersion (MXParser.java:3423)
at org.codehaus.plexus.util.xml.pull.MXParser.parseXmlDecl (MXParser.java:3345)
at org.codehaus.plexus.util.xml.pull.MXParser.parsePI (MXParser.java:3197)
at org.codehaus.plexus.util.xml.pull.MXParser.parseProlog (MXParser.java:1828)
at org.codehaus.plexus.util.xml.pull.MXParser.nextImpl (MXParser.java:1757)
at org.codehaus.plexus.util.xml.pull.MXParser.next (MXParser.java:1375)
at org.apache.maven.model.io.xpp3.MavenXpp3Reader.read (MavenXpp3Reader.java:3940)
at org.apache.maven.model.io.xpp3.MavenXpp3Reader.read (MavenXpp3Reader.java:612)
at org.apache.maven.model.io.xpp3.MavenXpp3Reader.read (MavenXpp3Reader.java:627)
at org.cyclonedx.maven.BaseCycloneDxMojo.readPom (BaseCycloneDxMojo.java:759)
at org.cyclonedx.maven.BaseCycloneDxMojo.readPom (BaseCycloneDxMojo.java:746)
at org.cyclonedx.maven.BaseCycloneDxMojo.retrieveParentProject (BaseCycloneDxMojo.java:694)
at org.cyclonedx.maven.BaseCycloneDxMojo.getClosestMetadata (BaseCycloneDxMojo.java:524)
at org.cyclonedx.maven.BaseCycloneDxMojo.convert (BaseCycloneDxMojo.java:481)
at org.cyclonedx.maven.CycloneDxMojo.execute (CycloneDxMojo.java:70)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126)
at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:342)
at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:330)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:213)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:175)
at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:76)
at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:163)
at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:160)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:260)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:172)
at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:100)
at org.apache.maven.cli.MavenCli.execute (MavenCli.java:821)
at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:270)
at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:77)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:568)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
```
So this pr change the version check condition of `build/mvn` to make it build Spark only with the verified maven version.
### Why are the changes needed?
Make `build/mvn` build Spark only with the verified maven version
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
- `java-11-17` GA build task pass and no error message as above
- Manual test:
1. Make the system use maven 3.9.0( > 3.8.7 ) by default:
run `mvn -version`
```
Apache Maven 3.9.0 (9b58d2bad23a66be161c4664ef21ce219c2c8584)
Maven home: /Users/${userName}/Tools/maven
Java version: 1.8.0_362, vendor: Azul Systems, Inc., runtime: /Users/${userName}/Tools/zulu8/zulu-8.jdk/Contents/Home/jre
Default locale: zh_CN, platform encoding: UTF-8
OS name: "mac os x", version: "13.2.1", arch: "aarch64", family: "mac"
```
and run `build/mvn -version`
```
Using `mvn` from path: /${basedir}/spark/build/apache-maven-3.8.7/bin/mvn
Using SPARK_LOCAL_IP=localhost
Apache Maven 3.8.7 (b89d5959fcde851dcb1c8946a785a163f14e1e29)
Maven home: /${basedir}/spark/build/apache-maven-3.8.7
Java version: 1.8.0_362, vendor: Azul Systems, Inc., runtime: /Users/${userName}/Tools/zulu8/zulu-8.jdk/Contents/Home/jre
Default locale: zh_CN, platform encoding: UTF-8
OS name: "mac os x", version: "13.2.1", arch: "aarch64", family: "mac"
```
We can see Spark use 3.8.7 in build directory when the system default maven > 3.8.7
2. Make the system use maven 3.8.7 by default:
run `mvn -version`
```
mvn -version
Apache Maven 3.8.7 (b89d5959fcde851dcb1c8946a785a163f14e1e29)
Maven home: /Users/${userName}/Tools/maven
Java version: 1.8.0_362, vendor: Azul Systems, Inc., runtime: /Users/${userName}/Tools/zulu8/zulu-8.jdk/Contents/Home/jre
Default locale: zh_CN, platform encoding: UTF-8
OS name: "mac os x", version: "13.2.1", arch: "aarch64", family: "mac"
```
and run `build/mvn -version`
```
Using `mvn` from path: /Users/${userName}/Tools/maven/bin/mvn
Using SPARK_LOCAL_IP=localhost
Apache Maven 3.8.7 (b89d5959fcde851dcb1c8946a785a163f14e1e29)
Maven home: /Users/${userName}/Tools/maven
Java version: 1.8.0_362, vendor: Azul Systems, Inc., runtime: /Users/${userName}/Tools/zulu8/zulu-8.jdk/Contents/Home/jre
Default locale: zh_CN, platform encoding: UTF-8
OS name: "mac os x", version: "13.2.1", arch: "aarch64", family: "mac"
```
We can see Spark use system default maven 3.8.7 when the system default maven is 3.8.7.
3. Make the system use maven 3.8.6( < 3.8.7 ) by default:
run `mvn -version`
```
mvn -version
Apache Maven 3.8.6 (84538c9988a25aec085021c365c560670ad80f63)
Maven home: /Users/${userName}/Tools/maven
Java version: 1.8.0_362, vendor: Azul Systems, Inc., runtime: /Users/${userName}/Tools/zulu8/zulu-8.jdk/Contents/Home/jre
Default locale: zh_CN, platform encoding: UTF-8
OS name: "mac os x", version: "13.2.1", arch: "aarch64", family: "mac"
```
and run `build/mvn -version`
```
Using `mvn` from path: /Users/${userName}/Tools/maven/bin/mvn
Using SPARK_LOCAL_IP=localhost
Apache Maven 3.8.7 (b89d5959fcde851dcb1c8946a785a163f14e1e29)
Maven home: /Users/${userName}/Tools/maven
Java version: 1.8.0_362, vendor: Azul Systems, Inc., runtime: /Users/${userName}/Tools/zulu8/zulu-8.jdk/Contents/Home/jre
Default locale: zh_CN, platform encoding: UTF-8
OS name: "mac os x", version: "13.2.1", arch: "aarch64", family: "mac"
```
We can see Spark use 3.8.7 in build directory when the system default maven < 3.8.7.
Closes #40283 from LuciferYang/ban-maven-3.9.x.
Authored-by: yangjie01 <ya...@baidu.com>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
(cherry picked from commit f70b8cf1a00002b6c6b96ec4e6ad4d6c2f0ab392)
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
build/mvn | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/build/mvn b/build/mvn
index 4989c2d7efd..e91141d732e 100755
--- a/build/mvn
+++ b/build/mvn
@@ -119,7 +119,7 @@ install_mvn() {
if [ "$MVN_BIN" ]; then
local MVN_DETECTED_VERSION="$(mvn --version | head -n1 | awk '{print $3}')"
fi
- if [ $(version $MVN_DETECTED_VERSION) -lt $(version $MVN_VERSION) ]; then
+ if [ $(version $MVN_DETECTED_VERSION) -ne $(version $MVN_VERSION) ]; then
local MVN_TARBALL="apache-maven-${MVN_VERSION}-bin.tar.gz"
local FILE_PATH="maven/maven-3/${MVN_VERSION}/binaries/${MVN_TARBALL}"
local APACHE_MIRROR=${APACHE_MIRROR:-'https://www.apache.org/dyn/closer.lua'}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org