You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/09/12 16:48:21 UTC

[I] Decide how the Delta Lake contrib jar is built, versioned, and published in releases [datafusion-comet]

andygrove opened a new issue, #5882:
URL: https://github.com/apache/datafusion-comet/issues/5882

   ## Context
   
   #5365 adds the Delta Lake scan contrib as a separate Maven module (`contrib/delta-spark`) that only enters the reactor under the `delta` profile and produces its own jar, `comet-contrib-delta-spark<version>_<scala>`. The user guide and the module README tell users to put that jar on the classpath alongside `delta-spark`. This issue is for deciding how that jar reaches users, and more generally how table-format integrations are packaged and released, since Delta and Iceberg currently take different shapes.
   
   ## How the release tooling behaves today
   
   - `dev/release/build-release-comet.sh` runs `./mvnw install` into a temporary local repository for six profile combinations (Spark 3.4 and 3.5 with Scala 2.12 and 2.13, then Spark 4.0 and 4.1). None of those invocations pass `-Pdelta`, so the contrib module is never built during a release.
   - `dev/release/publish-to-maven.sh` does not run `mvn deploy`. It signs and uploads every `.jar` and `.pom` found under `org/apache/datafusion` in that local repository. Two things follow:
     - `maven.deploy.skip` has no effect on what gets published. Whether the contrib jar ships depends only on whether `-Pdelta` is on the `install` lines.
     - Anything `install` produces is published. #5365 adds an unconditional `test-jar` execution to `spark/pom.xml` (about 6.7 MB per profile) so that the contrib suites can reuse `CometTestBase`. As written it would ship six `-tests.jar` artifacts per release.
   - The native library needs no release change. The release containers build with default Cargo features, and the `delta` feature (deletion-vector decoding plus the planner arm) is in the default set, so the released `libcomet` already carries everything the contrib jar needs.
   
   ## Questions to settle
   
   1. **Should the contrib jar be published at all?** If yes, `-Pdelta` goes on the Spark 3.5, 4.0 and 4.1 `install` lines (the 3.4 pairing is `delta-core` 2.4 and does not compile). If no, the docs should say build-from-source is the only route and the `maven.deploy.skip=false` override in the contrib pom should go. An experimental feature that only exists if users build from source is unlikely to get real-workload testing.
   2. **What Delta versions does each published jar support?** Each jar compiles against one Delta release per Spark line (3.3.2, 4.0.1, 4.3.1). Nothing verifies binary compatibility across Delta minors, and `pom.xml` already records that Delta 4.1.0 and 4.2.0 are incompatible with Spark 4.1.3. Publishing implies a support statement in the user guide.
   3. **Should the spark test-jar be published?** Options: bind the `test-jar` execution inside the `delta` profile so it only exists when the contrib is built, or accept publishing it (there is partial precedent in the `-test-sources.jar`).
   4. **Why is Delta packaged differently from Iceberg, and should it stay that way?** Iceberg support lives inside `comet-spark`: `iceberg-spark-runtime` is a test-scope dependency only and `IcebergReflection` addresses every Iceberg class by name, so there is no profile, no separate jar, and no dependency on a specific Iceberg release. The Delta contrib compiles against `delta-spark` internals (`DeltaParquetFileFormat`, `DeletionVectorDescriptor`, `DeltaColumnMapping`), which forces a `provided` dependency, one artifact per Spark-and-Delta pairing, and rules out Spark 3.4. That structural difference explains the module, the profile, the separate CI workflow and the opt-in config. The two consistent end states are (a) fold Delta into `comet-spark` behind reflection like Iceberg, or (b) keep the contrib shape and finish it (publish the jar, document the Delta pairing, gate the test-jar). Deciding this on purpose also settles part of the core-versus-contrib boundary question raised in
  the #5365 review and in #5411.
   
   ## Related
   
   - #5365 (the contrib PR)
   - #174 (Delta Lake support)
   - #5411 (convergence of the two Delta read paths)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Decide how the Delta Lake contrib jar is built, versioned, and published in releases [datafusion-comet]

Posted by "dwsmith1983 (via GitHub) via github" <gi...@datafusion.apache.org>.
dwsmith1983 commented on issue #5882:
URL: https://github.com/apache/datafusion-comet/issues/5882#issuecomment-6547256976

   Some data from trying the contrib against 1.2.0 the way a user would, which may help whichever way this goes.
   
   I built `comet-contrib-delta-spark` from the `1.2.0-rc1` tag in a clean `maven:3.9-eclipse-temurin-17` container, with `comet-spark-spark4.1_2.13:1.2.0` resolved as an external artifact (built from the same commit, since staging wasn't reachable), and ran it on `apache/spark:4.1.3` with delta-spark 4.3.1. The native scan took a table with deletion vectors (`CometDeltaNativeScan` in the plan) and matched Spark's reader, so the released `libcomet` does carry everything the contrib needs, as the issue says.
   
   Two things a build-from-source user runs into today:
   
   1. `./mvnw -Pspark-4.1,delta -pl contrib/delta-spark package -Dmaven.test.skip=true` fails outside the reactor. Maven resolves the test-scoped `comet-spark` `tests` jar before it skips anything, and that jar isn't published. Calling the goals directly (`resources:resources scala:compile jar:jar`) works. This ties into question 3: if the test-jar stays unpublished, the README should give that command, or the contrib's dependency on it should move behind a profile so a plain `package` works against published artifacts.
   2. Loading Delta with `--packages` puts delta-spark in a child classloader. The contrib catches the resulting `NoClassDefFoundError: org/apache/spark/sql/delta/DeltaParquetFileFormat` and declines the scan, so the query quietly runs on Spark's reader, and the warning says the handler "was built against a different version of Comet's internals", which points at the wrong cause. Whatever the publishing answer, the user guide needs the one-classpath requirement (all jars on `extraClassPath`), and the warning could name the missing class instead.
   
   On question 1, I'd lean toward (b): publish the jar for Spark 3.5, 4.0 and 4.1 starting with 1.3.0, with the test-jar execution moved into the `delta` profile, so the experimental reader gets tested on real workloads without a source build.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org