You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "kerwin-zk (via GitHub)" <gi...@apache.org> on 2026/04/15 02:58:35 UTC
[PR] [spark] Add paimon spark4.1 module [paimon]
kerwin-zk opened a new pull request, #7648:
URL: https://github.com/apache/paimon/pull/7648
### Purpose
Add paimon spark4.1 module
### Tests
CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Add paimon spark4.1 module [paimon]
Posted by "YannByron (via GitHub)" <gi...@apache.org>.
YannByron merged PR #7648:
URL: https://github.com/apache/paimon/pull/7648
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Add paimon spark4.1 module [paimon]
Posted by "YannByron (via GitHub)" <gi...@apache.org>.
YannByron commented on code in PR #7648:
URL: https://github.com/apache/paimon/pull/7648#discussion_r3099390721
##########
paimon-spark/paimon-spark-4.1/src/main/scala/org/apache/paimon/spark/catalyst/optimizer/MergePaimonScalarSubqueries.scala:
##########
Review Comment:
Maybe `hive-test-udfs.jar`、`hive-site.xml` and `log4j2-test.properties`, consider these as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Add paimon spark4.1 module [paimon]
Posted by "kerwin-zk (via GitHub)" <gi...@apache.org>.
kerwin-zk commented on PR #7648:
URL: https://github.com/apache/paimon/pull/7648#issuecomment-4331883529
> @kerwin-zk I have some thoughts which may be worth considering.
>
> 1. These classes (`PaimonFormatTable`, `ScanPlanHelper`, `PaimonBatchWrite`, `AbstractPaimonSparkSqlExtensionsParser`, `PaimonStrategyHelper`, `PaimonCreateTableAsSelectStrategy` and `PaimonRelation`) maybe don't have to copy to `paimon-spark-4.0` cause these use the stable api. Please check.
> 2. And For classes (`AssignmentAlignmentHelper`, `PaimonMergeIntoResolver`, `PaimonMergeIntoResolverBase`, `RewriteUpsertTable`, `MergeIntoPaimonTable`and `MergeIntoPaimonDataEvolutionTable`), can we identify specific incompatibilities and then draw them into 'SparkShim' to solve them?
@YannByron `PaimonRelation` has been removed, and the rest need some refactoring. After this PR is merged, I will submit follow-up PRs to refactor and optimize this part of the logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Add paimon spark4.1 module [paimon]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #7648:
URL: https://github.com/apache/paimon/pull/7648#discussion_r3085400703
##########
pom.xml:
##########
@@ -427,6 +427,7 @@ under the License.
<modules>
<module>paimon-spark/paimon-spark4-common</module>
<module>paimon-spark/paimon-spark-4.0</module>
+ <module>paimon-spark/paimon-spark-4.1</module>
</modules>
Review Comment:
In the `spark4` Maven profile, adding the `paimon-spark-4.1` module means the reactor build now pulls Spark 4.1.1 (from `paimon-spark-4.1/pom.xml`) while the profile properties still pin `paimon-spark-common.spark.version` / `test.spark.*` to 4.0.x. This mixes Spark minor versions within the same profile and can make it unclear which Spark version the shared `paimon-spark4-common` and any Spark-based tests/e2e tooling are targeting. Consider either updating the `spark4` profile properties to 4.1.x (if 4.1 is now the baseline) or introducing a separate `spark41` profile so the common module and test properties stay aligned with the included Spark modules.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Add paimon spark4.1 module [paimon]
Posted by "YannByron (via GitHub)" <gi...@apache.org>.
YannByron commented on PR #7648:
URL: https://github.com/apache/paimon/pull/7648#issuecomment-4332252592
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Add paimon spark4.1 module [paimon]
Posted by "YannByron (via GitHub)" <gi...@apache.org>.
YannByron commented on code in PR #7648:
URL: https://github.com/apache/paimon/pull/7648#discussion_r3099368691
##########
paimon-spark/paimon-spark-4.1/src/main/scala/org/apache/paimon/spark/catalyst/optimizer/MergePaimonScalarSubqueries.scala:
##########
Review Comment:
The `MergePaimonScalarSubqueries` in `paimon-spark-4.1` is identical to the one in `paimon-spark-4.0`. Would it be better to move these version-agnostic implementations into `paimon-spark4-common` instead of duplicating them in each version-specific module?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Add paimon spark4.1 module [paimon]
Posted by "kerwin-zk (via GitHub)" <gi...@apache.org>.
kerwin-zk commented on code in PR #7648:
URL: https://github.com/apache/paimon/pull/7648#discussion_r3101009095
##########
paimon-spark/paimon-spark-4.1/src/main/scala/org/apache/paimon/spark/catalyst/optimizer/MergePaimonScalarSubqueries.scala:
##########
Review Comment:
@YannByron `log4j2-test.properties` and `function/hive-test-udfs.jar` have been removed from `paimon-spark-4.1` — `paimon-spark-ut` already ships identical copies on the test classpath. `hive-site.xml` is kept per-module on purpose. `paimon-spark-3.2` through `paimon-spark-4.0` each carry their own copy with a distinct `hive.metastore.uris` port (3.2→9087, 3.3→9088, 3.4→9089, 3.5→9090, 4.0→9090) so the ITCases can run in parallel without port collisions; `paimon-spark-4.1` follows the same pattern with 9091.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Add paimon spark4.1 module [paimon]
Posted by "YannByron (via GitHub)" <gi...@apache.org>.
YannByron commented on code in PR #7648:
URL: https://github.com/apache/paimon/pull/7648#discussion_r3099349728
##########
pom.xml:
##########
@@ -427,6 +427,7 @@ under the License.
<modules>
<module>paimon-spark/paimon-spark4-common</module>
<module>paimon-spark/paimon-spark-4.0</module>
+ <module>paimon-spark/paimon-spark-4.1</module>
Review Comment:
Should we consider to update `<paimon-spark-common.spark.version>` to 4.1.1 so that `paimon-spark4-common` and `paimon-spark-ut` are compiled against the latest Spark version in the profile?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Add paimon spark4.1 module [paimon]
Posted by "YannByron (via GitHub)" <gi...@apache.org>.
YannByron commented on PR #7648:
URL: https://github.com/apache/paimon/pull/7648#issuecomment-4325310052
@kerwin-zk I have some thoughts which may be worth considering.
1. These classes (`PaimonRelation`, `ScanPlanHelper`, `PaimonBatchWrite`, `AbstractPaimonSparkSqlExtensionsParser`, `PaimonStrategyHelper`, `PaimonCreateTableAsSelectStrategy` and `PaimonRelation`) maybe don't have to copy to `paimon-spark-4.0` cause these use the stable api. Please check.
2. And For classes (`AssignmentAlignmentHelper`, `PaimonMergeIntoResolver`, `PaimonMergeIntoResolverBase`, `RewriteUpsertTable`, `MergeIntoPaimonTable`and `MergeIntoPaimonDataEvolutionTable`), can we identify specific incompatibilities and then draw them into 'SparkShim' to solve them?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org