You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kyuubi.apache.org by ul...@apache.org on 2021/10/22 06:51:56 UTC
[incubator-kyuubi] branch master updated: [KYUUBI #1257] [BUILD]
Reduce unnecessary maven profile
This is an automated email from the ASF dual-hosted git repository.
ulyssesyou pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-kyuubi.git
The following commit(s) were added to refs/heads/master by this push:
new 589b23f [KYUUBI #1257] [BUILD] Reduce unnecessary maven profile
589b23f is described below
commit 589b23f578735a9508b2d2dd7ceb80582a366803
Author: Cheng Pan <ch...@apache.org>
AuthorDate: Fri Oct 22 14:51:42 2021 +0800
[KYUUBI #1257] [BUILD] Reduce unnecessary maven profile
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
The changes based on https://github.com/apache/incubator-kyuubi/pull/1226#issuecomment-942979767
In this PR, I'm going to remove profiles `kyuubi-extension-spark-3-1` and `kyuubi-extension-spark-3-2`, and keep `spark-3.0`, `spark-3.1`, `spark-3.2`.
After changes, when `spark-3.1` is active, set `spark.version` to 3.1.2 and enable `kyuubi-extension-spark-common` module and `kyuubi-extension-spark-3-1` module, `spark-3.2` does the same thing
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes #1257 from pan3793/pom.
Closes #1257
1d871b6c [Cheng Pan] nit
874f30c1 [Cheng Pan] Address comments
00a32b25 [Cheng Pan] nit
445666ee [Cheng Pan] typo
bd96097e [Cheng Pan] nit
e85116cd [Cheng Pan] nit
98347c8c [Cheng Pan] Update doc
6615d7cc [Cheng Pan] Exclude HudiTest in cross test
087d3fde [Cheng Pan] Add spark 3.2 binary test
ccff63c1 [Cheng Pan] update GA
23b81dcd [Cheng Pan] Address comments
3055c2a8 [Cheng Pan] nit
49a41e86 [Cheng Pan] nit
8d400798 [Cheng Pan] Reduce unnecessary maven profile
ced8d987 [Cheng Pan] Reduce unnecessary maven profile
Lead-authored-by: Cheng Pan <ch...@apache.org>
Co-authored-by: Cheng Pan <ch...@ChengdeMac-mini.local>
Signed-off-by: ulysses-you <ul...@apache.org>
---
.github/workflows/master.yml | 8 +++++---
.github/workflows/nightly.yml | 2 +-
.github/workflows/publish-snapshot.yml | 4 +++-
build/release/create-package.sh | 2 +-
build/release/release.sh | 6 ++++++
dev/kyuubi-codecov/pom.xml | 8 ++++++++
dev/kyuubi-extension-spark-3-1/pom.xml | 9 --------
dev/kyuubi-extension-spark-3-2/pom.xml | 10 ---------
dev/kyuubi-extension-spark-common/pom.xml | 34 +++++++++----------------------
docs/community/release.md | 2 +-
docs/sql/rules.md | 18 ++++++++++------
docs/sql/z-order-benchmark.md | 2 --
externals/kyuubi-download/pom.xml | 8 ++++++++
pom.xml | 24 ++++++++--------------
tools/spark-block-cleaner/pom.xml | 5 -----
15 files changed, 63 insertions(+), 79 deletions(-)
diff --git a/.github/workflows/master.yml b/.github/workflows/master.yml
index 983c4ab..86d1395 100644
--- a/.github/workflows/master.yml
+++ b/.github/workflows/master.yml
@@ -39,16 +39,18 @@ jobs:
- java: 8
spark: '3.1'
spark-hadoop: '2.7'
- profiles: '-Dspark.archive.mirror=https://archive.apache.org/dist/spark/spark-3.0.3 -Dspark.archive.name=spark-3.0.3-bin-hadoop2.7.tgz -Dmaven.plugin.scalatest.exclude.tags=org.apache.kyuubi.tags.ExtendedSQLTest,org.apache.kyuubi.tags.DeltaTest,org.apache.kyuubi.tags.IcebergTest'
+ profiles: '-Dspark.archive.mirror=https://archive.apache.org/dist/spark/spark-3.0.3 -Dspark.archive.name=spark-3.0.3-bin-hadoop2.7.tgz -Dmaven.plugin.scalatest.exclude.tags=org.apache.kyuubi.tags.ExtendedSQLTest,org.apache.kyuubi.tags.DeltaTest,org.apache.kyuubi.tags.HudiTest,org.apache.kyuubi.tags.IcebergTest'
+ - java: 8
+ spark: '3.1'
+ spark-hadoop: '3.2'
+ profiles: '-Dspark.archive.mirror=https://archive.apache.org/dist/spark/spark-3.2.0 -Dspark.archive.name=spark-3.2.0-bin-hadoop3.2.tgz -Dmaven.plugin.scalatest.exclude.tags=org.apache.kyuubi.tags.ExtendedSQLTest,org.apache.kyuubi.tags.DeltaTest,org.apache.kyuubi.tags.HudiTest,org.apache.kyuubi.tags.IcebergTest'
- java: 8
spark: '3.1'
spark-hadoop: '3.2'
- profiles: '-Pkyuubi-extension-spark-3-1'
codecov: 'true'
- java: 8
spark: '3.2'
spark-hadoop: '3.2'
- profiles: '-Pkyuubi-extension-spark-3-2'
- java: 11
spark: '3.1'
spark-hadoop: '3.2'
diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml
index af6f2f4..b5a9583 100644
--- a/.github/workflows/nightly.yml
+++ b/.github/workflows/nightly.yml
@@ -11,7 +11,7 @@ jobs:
strategy:
matrix:
profiles:
- - '-Pspark-master -pl :kyuubi-spark-sql-engine_2.12,:kyuubi-common_2.12,:kyuubi-ha_2.12,:kyuubi-zookeeper_2.12,:kyuubi-spark-monitor_2.12'
+ - '-Pspark-master -pl externals/kyuubi-spark-sql-engine -am'
env:
SPARK_LOCAL_IP: localhost
steps:
diff --git a/.github/workflows/publish-snapshot.yml b/.github/workflows/publish-snapshot.yml
index 3855281..e1e4ff2 100644
--- a/.github/workflows/publish-snapshot.yml
+++ b/.github/workflows/publish-snapshot.yml
@@ -36,4 +36,6 @@ jobs:
ASF_PASSWORD: ${{ secrets.NEXUS_PW }}
run: |
rm -rf ~/.m2/repository/org/apache/kyuubi
- ./build/mvn clean deploy -DskipTests -Pkyuubi-extension-spark-3-1,spark-provided -s ./build/release/asf-settings.xml
+ ./build/mvn clean deploy -s ./build/release/asf-settings.xml -DskipTests -Pspark-provided
+ ./build/mvn clean deploy -s ./build/release/asf-settings.xml -DskipTests -Pspark-provided,spark-3.1 -pl dev/kyuubi-extension-spark-3-1 -am
+ ./build/mvn clean deploy -s ./build/release/asf-settings.xml -DskipTests -Pspark-provided,spark-3.2 -pl dev/kyuubi-extension-spark-3-2 -am
diff --git a/build/release/create-package.sh b/build/release/create-package.sh
index 9b0bb9f..f9cf52d 100755
--- a/build/release/create-package.sh
+++ b/build/release/create-package.sh
@@ -75,7 +75,7 @@ package_binary() {
echo "Creating binary release tarball ${BIN_TGZ_FILE}"
- ${KYUUBI_DIR}/build/dist --tgz --spark-provided -Pkyuubi-extension-spark-3-1
+ ${KYUUBI_DIR}/build/dist --tgz --spark-provided -Pspark-3.1
cp "${BIN_TGZ_FILE}" "${RELEASE_DIR}"
diff --git a/build/release/release.sh b/build/release/release.sh
index 0cd6919..00d7fa8 100755
--- a/build/release/release.sh
+++ b/build/release/release.sh
@@ -96,6 +96,12 @@ upload_svn_staging() {
upload_nexus_staging() {
${KYUUBI_DIR}/build/mvn clean deploy -DskipTests -Papache-release,spark-provided \
-s "${KYUUBI_DIR}/build/release/asf-settings.xml"
+ ${KYUUBI_DIR}/build/mvn clean deploy -DskipTests -Papache-release,spark-provided,spark-3.1 \
+ -s "${KYUUBI_DIR}/build/release/asf-settings.xml" \
+ -pl dev/kyuubi-extension-spark-3-1 -am
+ ${KYUUBI_DIR}/build/mvn clean deploy -DskipTests -Papache-release,spark-provided,spark-3.2 \
+ -s "${KYUUBI_DIR}/build/release/asf-settings.xml" \
+ -pl dev/kyuubi-extension-spark-3-2 -am
}
finalize_svn() {
diff --git a/dev/kyuubi-codecov/pom.xml b/dev/kyuubi-codecov/pom.xml
index 18ca239..3a3f0c0 100644
--- a/dev/kyuubi-codecov/pom.xml
+++ b/dev/kyuubi-codecov/pom.xml
@@ -111,6 +111,14 @@
<skip>true</skip>
</configuration>
</plugin>
+
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-deploy-plugin</artifactId>
+ <configuration>
+ <skip>true</skip>
+ </configuration>
+ </plugin>
</plugins>
</build>
diff --git a/dev/kyuubi-extension-spark-3-1/pom.xml b/dev/kyuubi-extension-spark-3-1/pom.xml
index 1a7d517..f43e933 100644
--- a/dev/kyuubi-extension-spark-3-1/pom.xml
+++ b/dev/kyuubi-extension-spark-3-1/pom.xml
@@ -155,13 +155,4 @@
</plugin>
</plugins>
</build>
-
- <profiles>
- <profile>
- <id>kyuubi-extension-spark-3-1</id>
- <properties>
- <spark.version>3.1.2</spark.version>
- </properties>
- </profile>
- </profiles>
</project>
diff --git a/dev/kyuubi-extension-spark-3-2/pom.xml b/dev/kyuubi-extension-spark-3-2/pom.xml
index 5cf2f9b..eccf9d6 100644
--- a/dev/kyuubi-extension-spark-3-2/pom.xml
+++ b/dev/kyuubi-extension-spark-3-2/pom.xml
@@ -161,14 +161,4 @@
</plugin>
</plugins>
</build>
-
- <profiles>
- <profile>
- <id>kyuubi-extension-spark-3-2</id>
- <properties>
- <spark.version>3.2.0</spark.version>
- </properties>
- </profile>
- </profiles>
-
</project>
diff --git a/dev/kyuubi-extension-spark-common/pom.xml b/dev/kyuubi-extension-spark-common/pom.xml
index aa981e5..b3fb584 100644
--- a/dev/kyuubi-extension-spark-common/pom.xml
+++ b/dev/kyuubi-extension-spark-common/pom.xml
@@ -135,13 +135,11 @@
</plugin>
<plugin>
- <groupId>net.alchim31.maven</groupId>
- <artifactId>scala-maven-plugin</artifactId>
- </plugin>
-
- <plugin>
- <groupId>org.scalatest</groupId>
- <artifactId>scalatest-maven-plugin</artifactId>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-dependency-plugin</artifactId>
+ <configuration>
+ <skip>true</skip>
+ </configuration>
</plugin>
<plugin>
@@ -159,24 +157,12 @@
</plugin>
<plugin>
- <groupId>org.scalastyle</groupId>
- <artifactId>scalastyle-maven-plugin</artifactId>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-deploy-plugin</artifactId>
+ <configuration>
+ <skip>true</skip>
+ </configuration>
</plugin>
</plugins>
</build>
-
- <profiles>
- <profile>
- <id>kyuubi-extension-spark-3-1</id>
- <properties>
- <spark.version>3.1.2</spark.version>
- </properties>
- </profile>
- <profile>
- <id>kyuubi-extension-spark-3-2</id>
- <properties>
- <spark.version>3.2.0</spark.version>
- </properties>
- </profile>
- </profiles>
</project>
diff --git a/docs/community/release.md b/docs/community/release.md
index d2d34bd..1a9afe7 100644
--- a/docs/community/release.md
+++ b/docs/community/release.md
@@ -159,7 +159,7 @@ export RELEASE_RC_NO=<RC number, e.g. 0>
```shell
build/mvn versions:set -DgenerateBackupPoms=false \
-DnewVersion="${RELEASE_VERSION}" \
- -Pkubernetes,kyuubi-extension-spark-3-1,spark-block-cleaner,tpcds
+ -Pspark-3.1,spark-block-cleaner
git commit -am "[RELEASE] Bump ${RELEASE_VERSION}"
```
diff --git a/docs/sql/rules.md b/docs/sql/rules.md
index a1d810b..0723ed4 100644
--- a/docs/sql/rules.md
+++ b/docs/sql/rules.md
@@ -15,8 +15,6 @@
- limitations under the License.
-->
-<!-- DO NOT MODIFY THIS FILE DIRECTLY, IT IS AUTO GENERATED BY [org.apache.kyuubi.engine.spark.udf.KyuubiUDFRegistrySuite] -->
-
<div align=center>
@@ -53,10 +51,18 @@ And don't worry, Kyuubi will support the new Apache Spark version in future. Tha
## How to use Kyuubi SQL extension
-1. you need to choose Apache Spark branch-3.1 or higher version with Kyuubi binary tgz.
-2. if you want to compile Kyuubi by yourself, the maven opt should add `-Pkyuubi-extension-spark-3-1`
-3. move the jar(`kyuubi-extension-spark-*.jar`) which is in `$KYUUBI_HOME/extension` into `$SPARK_HOME/jars`
-4. add a config into `spark-defaults.conf`, `spark.sql.extensions=org.apache.kyuubi.sql.KyuubiSparkSQLExtension`
+| Kyuubi Spark SQL extension | Supported Spark version(s) | Available since | EOL | Bundled in Binary release tarball | Maven profile
+| -------------------------- | -------------------------- | ---------------- | ---------------- | --------------------------------- | -------------
+| kyuubi-extension-spark-3-1 | 3.1.x | 1.3.0-incubating | N/A | 1.3.0-incubating | spark-3.1
+| kyuubi-extension-spark-3-2 | 3.2.x | 1.4.0-incubating | N/A | 1.4.0-incubating | spark-3.2
+
+1. Check the matrix that if you are using the supported Spark version, and find the corresponding Kyuubi Spark SQL Extension jar
+2. Get the Kyuubi Spark SQL Extension jar
+ 1. Each Kyuubi binary release tarball only contains one default version of Kyuubi Spark SQL Extension jar, if you are looking for such version, you can find it under `$KYUUBI_HOME/extension`
+ 2. All supported versions of Kyuubi Spark SQL Extension jar will be deployed to [Maven Central](https://search.maven.org/search?q=kyuubi-extension-spark)
+ 3. If you like, you can compile Kyuubi Spark SQL Extension jar by yourself, please activate the corresponding Maven's profile on you compile command, i.e. you can get Kyuubi Spark SQL Extension jar for Spark 3.1 under `dev/kyuubi-extension-spark-3-1/target` when compile with `-Pspark-3.1`
+3. Put the Kyuubi Spark SQL extension jar `kyuubi-extension-spark-*.jar` into `$SPARK_HOME/jars`
+4. Enable `KyuubiSparkSQLExtension`, i.e. add a config into `$SPARK_HOME/conf/spark-defaults.conf`, `spark.sql.extensions=org.apache.kyuubi.sql.KyuubiSparkSQLExtension`
Now, you can enjoy the Kyuubi SQL Extension, and also Kyuubi provides some configs to make these feature easy to use.
diff --git a/docs/sql/z-order-benchmark.md b/docs/sql/z-order-benchmark.md
index d60cb7c..5beb180 100644
--- a/docs/sql/z-order-benchmark.md
+++ b/docs/sql/z-order-benchmark.md
@@ -15,8 +15,6 @@
- limitations under the License.
-->
-<!-- DO NOT MODIFY THIS FILE DIRECTLY, IT IS AUTO GENERATED BY [org.apache.kyuubi.engine.spark.udf.KyuubiUDFRegistrySuite] -->
-
<div align=center>
diff --git a/externals/kyuubi-download/pom.xml b/externals/kyuubi-download/pom.xml
index 7954567..a5613c5 100644
--- a/externals/kyuubi-download/pom.xml
+++ b/externals/kyuubi-download/pom.xml
@@ -57,6 +57,14 @@
</execution>
</executions>
</plugin>
+
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-deploy-plugin</artifactId>
+ <configuration>
+ <skip>true</skip>
+ </configuration>
+ </plugin>
</plugins>
</build>
</project>
diff --git a/pom.xml b/pom.xml
index 88d8f1c..d8ffe7f 100644
--- a/pom.xml
+++ b/pom.xml
@@ -1775,6 +1775,10 @@
<delta.version>1.0.0</delta.version>
<maven.plugin.scalatest.exclude.tags>org.apache.kyuubi.tags.ExtendedSQLTest,org.apache.kyuubi.tags.HudiTest</maven.plugin.scalatest.exclude.tags>
</properties>
+ <modules>
+ <module>dev/kyuubi-extension-spark-common</module>
+ <module>dev/kyuubi-extension-spark-3-1</module>
+ </modules>
</profile>
<profile>
@@ -1783,6 +1787,10 @@
<spark.version>3.2.0</spark.version>
<maven.plugin.scalatest.exclude.tags>org.apache.kyuubi.tags.ExtendedSQLTest,org.apache.kyuubi.tags.DeltaTest,org.apache.kyuubi.tags.IcebergTest,org.apache.kyuubi.tags.HudiTest</maven.plugin.scalatest.exclude.tags>
</properties>
+ <modules>
+ <module>dev/kyuubi-extension-spark-common</module>
+ <module>dev/kyuubi-extension-spark-3-2</module>
+ </modules>
</profile>
<profile>
@@ -1827,22 +1835,6 @@
</profile>
<profile>
- <id>kyuubi-extension-spark-3-1</id>
- <modules>
- <module>dev/kyuubi-extension-spark-common</module>
- <module>dev/kyuubi-extension-spark-3-1</module>
- </modules>
- </profile>
-
- <profile>
- <id>kyuubi-extension-spark-3-2</id>
- <modules>
- <module>dev/kyuubi-extension-spark-common</module>
- <module>dev/kyuubi-extension-spark-3-2</module>
- </modules>
- </profile>
-
- <profile>
<id>spark-block-cleaner</id>
<modules>
<module>tools/spark-block-cleaner</module>
diff --git a/tools/spark-block-cleaner/pom.xml b/tools/spark-block-cleaner/pom.xml
index e186693..e09d6aa 100644
--- a/tools/spark-block-cleaner/pom.xml
+++ b/tools/spark-block-cleaner/pom.xml
@@ -55,10 +55,5 @@
<build>
<outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory>
<testOutputDirectory>target/scala-${scala.binary.version}/test-classes</testOutputDirectory>
- <resources>
- <resource>
- <directory>${project.basedir}/src/main/resources</directory>
- </resource>
- </resources>
</build>
</project>