You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "kerwin-zk (via GitHub)" <gi...@apache.org> on 2026/04/28 07:24:40 UTC
[PR] [spark] Reduce paimon-spark-4.0 shadows via SparkShim copy factories [paimon]
kerwin-zk opened a new pull request, #7721:
URL: https://github.com/apache/paimon/pull/7721
### Purpose
Follow-up to #7648 (Spark 4.1 module). After the reverse-shim layout landed, three of the files copied into `paimon-spark-4.0/src/main` only differed across versions because of `case class .copy(...)` calls on Spark types whose arity changed between 4.0.2 and 4.1.1:
- `DataSourceV2Relation` gained `Option[TimeTravelSpec]` (8 → 9 fields) — `relation.copy(table = ...)` compiled against 4.1.1 emits `copy\$default\$9`, which crashes on 4.0 with `NoSuchMethodError`.
- `TableSpec` gained `Seq[Constraint]` (8 → 9 fields) — same problem for `spec.copy(location = ...)` and `spec.copy(properties = ...)`.
Per-version scalac is the only thing that knows the right `copy\$default\$N` to emit, so we route those three calls through new `SparkShim` factories (one per call site). The implementations live in `Spark3Shim` / `Spark4Shim` (plus the 4.0 override), and the cross-version source files no longer need to be physically duplicated.
### Linked issue
Follow-up of #7648.
### Tests
- `paimon-spark-4.1` full UT: 36 suites / 728 succeeded / 0 failed
- `paimon-spark-4.0` full UT: 36 suites / 727 succeeded / 1 canceled / 0 failed
- Verified on top of latest master (post-#7648 merge).
### API and Format
No new public API. Three internal helper methods added to `org.apache.spark.sql.paimon.shims.SparkShim`:
- `copyDataSourceV2Relation(relation, newTable)`
- `copyTableSpecLocation(spec, location)`
- `copyTableSpecProperties(spec, properties)`
### Documentation
No user-facing changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Reduce paimon-spark-4.0 shadows via SparkShim copy factories [paimon]
Posted by "Zouxxyy (via GitHub)" <gi...@apache.org>.
Zouxxyy merged PR #7721:
URL: https://github.com/apache/paimon/pull/7721
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Reduce paimon-spark-4.0 shadows via SparkShim copy factories [paimon]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #7721:
URL: https://github.com/apache/paimon/pull/7721#discussion_r3192724349
##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/spark/sql/paimon/shims/SparkShim.scala:
##########
@@ -65,6 +66,27 @@ trait SparkShim {
partitions: Array[Transform],
properties: JMap[String, String]): Table
+ /**
+ * Returns a `DataSourceV2Relation` like `relation` but with `table` replaced. Spark 4.1 added
+ * `Option[TimeTravelSpec]` as the 6th field of `DataSourceV2Relation`, so a `relation.copy(table
+ * = ...)` call compiled against 4.1.1 emits a `copy$default$6` reference that crashes with
+ * `NoSuchMethodError` on Spark 4.0 runtime. Routing through this factory lets each per-version
+ * SparkShim implementation generate the matching copy bytecode.
Review Comment:
The Scaladoc for `copyDataSourceV2Relation` says `relation.copy(table = ...)` compiled against Spark 4.1.1 emits a `copy$default$6` reference. Given the same comment block later cites `copy$default$9` for `TableSpec`, and because `copy$default$N` depends on the total arity of the case class, the specific `$6` here is likely inaccurate/misleading. Please either correct the index (if known) or reword to avoid hard-coding a specific `copy$default$N` value and just describe the general arity-mismatch problem.
##########
paimon-spark/paimon-spark-common/src/main/scala/org/apache/spark/sql/paimon/shims/SparkShim.scala:
##########
@@ -65,6 +66,27 @@ trait SparkShim {
partitions: Array[Transform],
properties: JMap[String, String]): Table
+ /**
+ * Returns a `DataSourceV2Relation` like `relation` but with `table` replaced. Spark 4.1 added
+ * `Option[TimeTravelSpec]` as the 6th field of `DataSourceV2Relation`, so a `relation.copy(table
+ * = ...)` call compiled against 4.1.1 emits a `copy$default$6` reference that crashes with
+ * `NoSuchMethodError` on Spark 4.0 runtime. Routing through this factory lets each per-version
+ * SparkShim implementation generate the matching copy bytecode.
+ */
+ def copyDataSourceV2Relation(
+ relation: DataSourceV2Relation,
+ newTable: Table): DataSourceV2Relation
+
Review Comment:
The new shim factory only covers `relation.copy(table = ...)`, but there are still `DataSourceV2Relation.copy(...)` call sites in the shared `paimon-spark-common` code (compiled against Spark 4.1 under the spark4 profile). For example, `MergeIntoPaimonDataEvolutionTable` uses `touchedFileTargetRelation.copy(output = ...)` and `touchedFileTargetRelation.copy(targetRelation.table, ...)`; these will have the same Spark 4.1 ↔ 4.0 binary-incompatibility risk and can still trigger `NoSuchMethodError` on Spark 4.0 runtimes. Consider routing all `DataSourceV2Relation.copy(...)` usages in cross-version code through shim factories (one per call-site or a small set of generic helpers), or replace them with a stable constructor API if available (e.g., `DataSourceV2Relation.create(...)`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org