You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/03/30 18:49:36 UTC
[GitHub] [hudi] alexeykudinkin commented on a change in pull request #5171: [HUDI-3681] Provision additional hudi-spark-bundle with different versions
alexeykudinkin commented on a change in pull request #5171:
URL: https://github.com/apache/hudi/pull/5171#discussion_r838859902
##########
File path: pom.xml
##########
@@ -1587,40 +1589,69 @@
</build>
</profile>
+ <!-- hudi-spark2-bundle is an alias for hudi-spark-bundle -->
Review comment:
Should we rather affiliate the profiles, rather than artifacts here? It's not obvious and therefore confusing how spark profile trickle down into the artifacts
##########
File path: pom.xml
##########
@@ -1587,40 +1589,69 @@
</build>
</profile>
+ <!-- hudi-spark2-bundle is an alias for hudi-spark-bundle -->
<profile>
<id>spark2</id>
+ <properties>
+ <sparkbundle.version>2</sparkbundle.version>
+ </properties>
+ <activation>
+ <property>
+ <name>spark2</name>
+ </property>
+ </activation>
+ </profile>
+
+ <profile>
+ <id>spark2.x</id>
Review comment:
What's the diff b/w spark2 and spark2.x?
##########
File path: .github/workflows/bot.yml
##########
@@ -14,39 +14,57 @@ jobs:
build:
runs-on: ubuntu-latest
strategy:
- max-parallel: 7
+ max-parallel: 11
matrix:
include:
- # Spark 2.4.4
+ # Spark 2.4.4, scala 2.11
- scalaProfile: "scala-2.11"
- sparkProfile: "spark2"
+ sparkProfile: "spark2.4"
+ sparkVersion: "2.4.4"
+
+ # Spark 2.4.4, scala 2.12
+ - scalaProfile: "scala-2.12"
+ sparkProfile: "spark2.4"
sparkVersion: "2.4.4"
# Spark 3.1.x
- scalaProfile: "scala-2.12"
- sparkProfile: "spark3.1.x"
+ sparkProfile: "spark3.1"
sparkVersion: "3.1.0"
- scalaProfile: "scala-2.12"
- sparkProfile: "spark3.1.x"
+ sparkProfile: "spark3.1"
sparkVersion: "3.1.1"
- scalaProfile: "scala-2.12"
- sparkProfile: "spark3.1.x"
+ sparkProfile: "spark3.1"
sparkVersion: "3.1.2"
- scalaProfile: "scala-2.12"
- sparkProfile: "spark3.1.x"
+ sparkProfile: "spark3.1"
sparkVersion: "3.1.3"
# Spark 3.2.x
- scalaProfile: "scala-2.12"
- sparkProfile: "spark3.2.0"
+ sparkProfile: "spark3.2"
sparkVersion: "3.2.0"
+ - scalaProfile: "scala-2.12"
+ sparkProfile: "spark3.2"
+ sparkVersion: "3.2.1"
+
+ # Alias
Review comment:
AFAIK we have limited budget of CI minutes on GH, so i'd much rather burn it on running more tests than testing profile aliases
##########
File path: pom.xml
##########
@@ -1635,6 +1666,71 @@
</activation>
</profile>
+ <profile>
+ <id>spark3.1</id>
+ <properties>
+ <spark3.version>3.1.3</spark3.version>
+ <spark.version>${spark3.version}</spark.version>
+ <sparkbundle.version>3.1</sparkbundle.version>
+ <scala.version>${scala12.version}</scala.version>
+ <scala.binary.version>2.12</scala.binary.version>
+ <hudi.spark.module>hudi-spark3.1.x</hudi.spark.module>
+ <hudi.spark.common.module>hudi-spark3-common</hudi.spark.common.module>
+ <scalatest.version>${scalatest.spark3.version}</scalatest.version>
+ <kafka.version>${kafka.spark3.version}</kafka.version>
+ <fasterxml.version>${fasterxml.spark3.version}</fasterxml.version>
+ <fasterxml.jackson.databind.version>${fasterxml.spark3.version}</fasterxml.jackson.databind.version>
+ <fasterxml.jackson.module.scala.version>${fasterxml.spark3.version}</fasterxml.jackson.module.scala.version>
+ <fasterxml.jackson.dataformat.yaml.version>${fasterxml.spark3.version}
+ </fasterxml.jackson.dataformat.yaml.version>
+ <skip.hudi-spark2.unit.tests>true</skip.hudi-spark2.unit.tests>
+ <skipITs>true</skipITs>
+ </properties>
+ <modules>
+ <module>hudi-spark-datasource/hudi-spark3.1.x</module>
+ <module>hudi-spark-datasource/hudi-spark3-common</module>
+ </modules>
+ <activation>
+ <property>
+ <name>spark3.1</name>
+ </property>
+ </activation>
+ </profile>
+
+ <profile>
+ <id>spark3.2</id>
+ <properties>
+ <spark3.version>3.2.1</spark3.version>
+ <spark.version>${spark3.version}</spark.version>
+ <sparkbundle.version>3.2</sparkbundle.version>
+ <scala.version>${scala12.version}</scala.version>
+ <scala.binary.version>2.12</scala.binary.version>
+ <hudi.spark.module>hudi-spark3</hudi.spark.module>
+ <hudi.spark.common.module>hudi-spark3-common</hudi.spark.common.module>
+ <scalatest.version>${scalatest.spark3.version}</scalatest.version>
+ <kafka.version>${kafka.spark3.version}</kafka.version>
+ <parquet.version>1.12.2</parquet.version>
+ <avro.version>1.10.2</avro.version>
+ <orc.version>1.6.12</orc.version>
+ <fasterxml.version>${fasterxml.spark3.version}</fasterxml.version>
+ <fasterxml.jackson.databind.version>${fasterxml.spark3.version}</fasterxml.jackson.databind.version>
+ <fasterxml.jackson.module.scala.version>${fasterxml.spark3.version}</fasterxml.jackson.module.scala.version>
+ <fasterxml.jackson.dataformat.yaml.version>${fasterxml.spark3.version}
+ </fasterxml.jackson.dataformat.yaml.version>
+ <skip.hudi-spark2.unit.tests>true</skip.hudi-spark2.unit.tests>
+ <skipITs>true</skipITs>
+ </properties>
+ <modules>
+ <module>hudi-spark-datasource/hudi-spark3</module>
+ <module>hudi-spark-datasource/hudi-spark3-common</module>
+ </modules>
+ <activation>
+ <property>
+ <name>spark3.2</name>
+ </property>
+ </activation>
+ </profile>
+
<profile>
<id>spark3.2.0</id>
Review comment:
Let's kill this one, given that we've got the new profiles
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org