You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/04/12 17:02:39 UTC
[GitHub] [hudi] xushiyan commented on a diff in pull request #5297: [HUDI-3859] Fix spark profiles and utilities-slim dep
xushiyan commented on code in PR #5297:
URL: https://github.com/apache/hudi/pull/5297#discussion_r848666807
##########
README.md:
##########
@@ -72,32 +74,31 @@ mvn clean javadoc:aggregate -Pjavadocs
### Build with different Spark versions
-The default Spark version supported is 2.4.4. To build for different Spark versions and Scala 2.12, use the
-corresponding profile
+The default Spark version supported is 2.4.4. Refer to the table below for building with different Spark and Scala versions.
-| Label | Artifact Name for Spark Bundle | Maven Profile Option | Notes |
-|--|--|--|--|
-| Spark 2.4, Scala 2.11 | hudi-spark2.4-bundle_2.11 | `-Pspark2.4` | For Spark 2.4.4, which is the same as the default |
-| Spark 2.4, Scala 2.12 | hudi-spark2.4-bundle_2.12 | `-Pspark2.4,scala-2.12` | For Spark 2.4.4, which is the same as the default and Scala 2.12 |
-| Spark 3.1, Scala 2.12 | hudi-spark3.1-bundle_2.12 | `-Pspark3.1` | For Spark 3.1.x |
-| Spark 3.2, Scala 2.12 | hudi-spark3.2-bundle_2.12 | `-Pspark3.2` | For Spark 3.2.x |
-| Spark 3, Scala 2.12 | hudi-spark3-bundle_2.12 | `-Pspark3` | This is the same as `Spark 3.2, Scala 2.12` |
-| Spark, Scala 2.11 | hudi-spark-bundle_2.11 | Default | The default profile, supporting Spark 2.4.4 |
-| Spark, Scala 2.12 | hudi-spark-bundle_2.12 | `-Pscala-2.12` | The default profile (for Spark 2.4.4) with Scala 2.12 |
+| Maven build options | Expected Spark bundle jar name | Notes |
+|:--------------------------|:---------------------------------------------|:-------------------------------------------------|
+| (empty) | hudi-spark-bundle_2.11 (legacy bundle name) | For Spark 2.4.4 and Scala 2.11 (default options) |
+| `-Dspark2.4` | hudi-spark2.4-bundle_2.11 | For Spark 2.4.4 and Scala 2.11 (same as default) |
+| `-Dspark2.4 -Dscala-2.12` | hudi-spark2.4-bundle_2.12 | For Spark 2.4.4 and Scala 2.12 |
+| `-Dspark3.1 -Dscala-2.12` | hudi-spark3.1-bundle_2.12 | For Spark 3.1.x and Scala 2.12 |
+| `-Dspark3.2 -Dscala-2.12` | hudi-spark3.2-bundle_2.12 | For Spark 3.2.x and Scala 2.12 |
+| `-Dspark3` | hudi-spark3-bundle_2.12 (legacy bundle name) | For Spark 3.2.x and Scala 2.12 |
+| `-Dscala-2.12` | hudi-spark-bundle_2.12 (legacy bundle name) | For Spark 2.4.4 and Scala 2.12 |
For example,
```
-# Build against Spark 3.2.x (the default build shipped with the public Spark 3 bundle)
-mvn clean package -DskipTests -Pspark3.2
+# Build against Spark 3.2.x
+mvn clean package -DskipTests -Dspark3.2 -Dscala-2.12
Review Comment:
`spark3.2` sets the scala 12 dependencies but `scala-2.12` additionally runs the enforcer plugin to guard the artifacts' suffix. so i think it's better to activate both profiles to build
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org