You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/10/12 18:00:36 UTC
(datafusion-comet) 01/01: build: stop publishing comet-common, the parent pom and comet-spark-integration (#6888)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-6888-907c0e074c4e34812b3ae808736db28ecb173d6e
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
commit 1768216b8c3d730355894f56338da3104dea1eaf
Author: Andy Grove <ag...@apache.org>
AuthorDate: Mon Oct 12 18:00:12 2026 +0000
build: stop publishing comet-common, the parent pom and comet-spark-integration (#6888)
Only comet-spark is useful to a consumer. Its published pom is flattened
and shade-reduced, so it has no <parent> and no comet-common dependency,
and comet-common's six classes are inside the shaded jar. The parent pom
is installed as the raw root pom.xml with unresolved ${...} placeholders
in its artifactId, and comet-spark-integration only checks the jar.
The nightly snapshot workflow passed -Dmaven.deploy.skip=false, so it
uploaded all four modules for each of the five variants. Release staging
uploads whatever `install` leaves in its temporary repository, which
includes comet-common and the parent pom.
Skip deploying every module except comet-spark, drop the override from
the snapshot workflow, and have publish-to-maven.sh upload only
comet-spark-spark* directories. Installing is unchanged: builds run from
inside spark/ (make benchmark-*, the TPC-H and TPC-DS data generation in
pr_build_linux.yml) resolve comet-common from the local repository.
---
.github/workflows/publish_snapshot.yml | 9 +++++----
common/pom.xml | 9 ++++-----
dev/release/publish-to-maven.sh | 5 +++++
docs/source/contributor-guide/release_process.md | 5 +++++
pom.xml | 7 ++++++-
spark-integration/pom.xml | 6 ++----
spark/pom.xml | 2 +-
7 files changed, 28 insertions(+), 15 deletions(-)
diff --git a/.github/workflows/publish_snapshot.yml b/.github/workflows/publish_snapshot.yml
index ae98bc855e..e5d8eeea1d 100644
--- a/.github/workflows/publish_snapshot.yml
+++ b/.github/workflows/publish_snapshot.yml
@@ -279,11 +279,12 @@ jobs:
# target directory the check ran against, and the parent pom pins
# project.build.outputTimestamp, so it re-creates the same jar.
#
- # The root pom skips deploying the parent pom, but consumers need it
- # to resolve the child poms, and the release publishes it. The user
- # property overrides every module's setting.
+ # Only comet-spark opts in to deploy (see the root pom), so this
+ # publishes its shaded jar, sources jars and pom and nothing else.
+ # Leave -Dmaven.deploy.skip=false off: as a user property it would
+ # override every module's setting.
# shellcheck disable=SC2086
- ./mvnw -B "$GOAL" -DskipTests -Dmaven.deploy.skip=false $profiles
+ ./mvnw -B "$GOAL" -DskipTests $profiles
echo "::endgroup::"
done
ls -l jars
diff --git a/common/pom.xml b/common/pom.xml
index fa72c9c1d2..24e28838f5 100644
--- a/common/pom.xml
+++ b/common/pom.xml
@@ -39,12 +39,11 @@ under the License.
Kept small on purpose so `mvn compile -pl common` is a cheap prerequisite
for `cargo test` in the Rust workspace. All other JVM code lives in
comet-spark.
- -->
- <properties>
- <!-- Reverse default (skip installation), and then enable only for child modules -->
- <maven.deploy.skip>false</maven.deploy.skip>
- </properties>
+ Not deployed: comet-spark bundles these classes into its shaded jar, so the deploy skip
+ inherited from the parent pom stays in force. It is still installed locally, which is where
+ builds run from inside spark/ look for it.
+ -->
<build>
<plugins>
diff --git a/dev/release/publish-to-maven.sh b/dev/release/publish-to-maven.sh
index 314b0855cf..1e01fbeb3f 100755
--- a/dev/release/publish-to-maven.sh
+++ b/dev/release/publish-to-maven.sh
@@ -120,6 +120,11 @@ echo "Deploying artifacts from $LOCAL_REPO"
pushd $LOCAL_REPO/org/apache/datafusion
+# Only comet-spark is published. comet-common is shaded into the comet-spark jar and the comet-spark
+# pom is flattened, so nothing published refers to comet-common or the parent pom. The build installs
+# both locally, and comet-spark-integration only checks the jar.
+find . -mindepth 1 -maxdepth 1 ! -name 'comet-spark-spark*' -exec rm -rf {} +
+
# Remove any extra files generated during install
find . -type f |grep -v \.jar |grep -v \.pom | xargs rm
diff --git a/docs/source/contributor-guide/release_process.md b/docs/source/contributor-guide/release_process.md
index cd3ee4b6d8..ef6d26bcc5 100644
--- a/docs/source/contributor-guide/release_process.md
+++ b/docs/source/contributor-guide/release_process.md
@@ -499,6 +499,11 @@ Creating Nexus staging repository
In the Nexus repository UI (https://repository.apache.org/) locate and verify the artifacts in
staging (https://central.sonatype.org/publish/release/#locate-and-examine-your-staging-repository).
+The staging repository should hold only the `comet-spark-spark<spark version>_<scala version>` artifacts, one for
+each Spark and Scala combination that `build-release-comet.sh` builds. The build installs `comet-common` and the
+parent pom too, but `publish-to-maven.sh` leaves them out of the upload: `comet-common` is shaded into the
+`comet-spark` jar and the published `comet-spark` pom has no parent, so nothing refers to either of them.
+
The script closes the staging repository but does not release it. Releasing to Maven Central is a manual step
performed only after the vote passes (see [Publishing Maven Artifacts](#publishing-maven-artifacts) below).
diff --git a/pom.xml b/pom.xml
index 0704b7a86c..084c086e44 100644
--- a/pom.xml
+++ b/pom.xml
@@ -117,7 +117,12 @@ under the License.
<!-- Used by some tests inherited from Spark to get project root directory -->
<spark.test.home>${session.executionRootDirectory}</spark.test.home>
- <!-- Reverse default (skip installation), and then enable only for child modules -->
+ <!-- Deploy nothing by default and have comet-spark opt back in, so the snapshot repository gets
+ only the shaded comet-spark jar. comet-common is bundled into that jar, its published pom is
+ flattened and names neither a parent nor comet-common, and comet-spark-integration only
+ checks the jar's contents, so a consumer needs none of the others. Installing is left alone:
+ builds run from inside spark/ resolve comet-common from the local repository.
+ dev/release/publish-to-maven.sh applies the same rule to the release. -->
<maven.deploy.skip>true</maven.deploy.skip>
<!-- For testing with JDK 17 -->
diff --git a/spark-integration/pom.xml b/spark-integration/pom.xml
index f656bdd858..643447e2d7 100644
--- a/spark-integration/pom.xml
+++ b/spark-integration/pom.xml
@@ -34,10 +34,8 @@ under the License.
<name>comet-spark-integration</name>
<packaging>pom</packaging>
- <properties>
- <!-- Reverse default (skip installation), and then enable only for child modules -->
- <maven.deploy.skip>false</maven.deploy.skip>
- </properties>
+ <!-- Only checks the contents of the comet-spark jar. Neither installed (see the plugin below) nor
+ deployed (the deploy skip is inherited from the parent pom). -->
<dependencies>
<dependency>
diff --git a/spark/pom.xml b/spark/pom.xml
index 0dfd7bb259..6b71b199eb 100644
--- a/spark/pom.xml
+++ b/spark/pom.xml
@@ -34,7 +34,7 @@ under the License.
<name>comet-spark</name>
<properties>
- <!-- Reverse default (skip installation), and then enable only for child modules -->
+ <!-- The only published module: the parent pom skips deploying every module -->
<maven.deploy.skip>false</maven.deploy.skip>
</properties>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org