[I] Decide and document the delta-spark version pin for the Delta contrib [datafusion-comet]

["schenksj (via GitHub)" <gi...@apache.org>]

schenksj opened a new issue, #5390:
URL: https://github.com/apache/datafusion-comet/issues/5390

   Follow-up from review of #4952 ([tracking comment](https://github.com/apache/datafusion-comet/pull/4366#issuecomment-5320446796), item 3).
   
   ### What is pinned today
   
   The `contrib-delta` Maven profile pins a `delta-spark` version per Spark profile, in `spark/pom.xml`:
   
   | Spark profile | `delta.version` |
   |---|---|
   | spark-3.5 | 3.3.2 |
   | spark-4.0 | 4.0.0 |
   | spark-4.1 | 4.1.0 |
   
   with a top-level default of `4.1.0` in the root `pom.xml`. `dev/verify-contrib-delta-gate.sh`
   asserts the mapping holds (spark-4.1 -> delta-spark 4.1.x, spark-3.5 -> 3.x, spark-4.0 -> 4.0.x)
   and fails the build gate otherwise.
   
   ### The gap
   
   The pin is a real compatibility decision — the contrib reflects into delta-spark internals
   (`TahoeLogFileIndex`, `PreparedDeltaFileIndex`, `TahoeBatchFileIndex`, DV plumbing), so the
   supported range is narrower than "any delta-spark". But that decision is currently only implicit
   in the poms plus an assertion in the gate script. There is no issue recording:
   
   - **which** delta-spark versions the contrib claims to support, and why those;
   - **what a user should expect** if they run against a different delta-spark than the pinned one;
   - **who updates the pin** when a new delta-spark ships, and what has to be re-verified (the
     reflection surface is the risky part, not the wire format);
   - whether the per-Spark mapping should ever become a **range** rather than an exact pin.
   
   ### Ask
   
   Settle the policy and write it down — in the contrib user guide and/or the profile comments — and
   keep the gate script assertion as the enforcement. This is a documentation/decision issue rather
   than a code change; the current pins work.
   
   ---
   🤖 Filed with [Claude Code](https://claude.com/claude-code).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org