You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "wayneli-vt (via GitHub)" <gi...@apache.org> on 2025/12/18 06:19:32 UTC
[PR] [wip] [Spark] Support computed column shortcut [paimon]
wayneli-vt opened a new pull request, #6827:
URL: https://github.com/apache/paimon/pull/6827
### Purpose
This PR optimizes `MERGE INTO` on Paimon row-tracking (data-evolution) append tables for the self-merge pattern:
```SQL
MERGE INTO target
USING target AS source
ON target._ROW_ID = source._ROW_ID
WHEN MATCHED THEN UPDATE SET col2 = col1 + 1
```
When `source` and `target` are the same table and the merge condition is `_ROW_ID` equality, the joined result is identical to scanning the target table itself. Therefore, we can avoid all `join/shuffle/sort` and only write the updated columns.
Changes included:
1. `targetRelatedSplits`: for self-merge on `_ROW_ID`, directly scan all splits instead of computing touched splits via source scan/join.
2. **Join Free**`updateActionInvoke`: for self-merge on `_ROW_ID`, remove the target-source join. Read the required columns from the target scan once, and rewrite all source-side attribute references in merge actions/conditions to target attributes so `MergeRows` can be evaluated on the target scan output.
3. **Reduce write-side shuffle/sort**: let the scan report its natural partitioning and ordering:
- **Partitioned by** `_FIRST_ROW_ID`
- **Ordered by** `(_FIRST_ROW_ID, _ROW_ID)`
With this, Spark can eliminate the explicit shuffle/sort previously required before writing partial column updates:
```scala
repartitionByRange(firstRowIdColumn)
.sortWithinPartitions(FIRST_ROW_ID_NAME, ROW_ID_NAME)
### Tests
A new test case has been added to `RowTrackingTestBase` to specifically verify this pr:
* `org.apache.paimon.spark.sql.RowTrackingTestBase#Data Evolution: merge into table with data-evolution with _ROW_ID shortcut`
### API and Format
No
### Documentation
TODO
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Optimize MERGE INTO self-merge updates on dataEvolution table [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #6827:
URL: https://github.com/apache/paimon/pull/6827#issuecomment-3669025482
Thanks @wayneli-vt for the contribution! Can you share more about the specific scenario?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Optimize MERGE INTO self-merge updates on dataEvolution table [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi closed pull request #6827: [spark] Optimize MERGE INTO self-merge updates on dataEvolution table
URL: https://github.com/apache/paimon/pull/6827
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Optimize MERGE INTO self-merge updates on dataEvolution table [paimon]
Posted by "wayneli-vt (via GitHub)" <gi...@apache.org>.
wayneli-vt closed pull request #6827: [spark] Optimize MERGE INTO self-merge updates on dataEvolution table
URL: https://github.com/apache/paimon/pull/6827
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Optimize MERGE INTO self-merge updates on dataEvolution table [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #6827:
URL: https://github.com/apache/paimon/pull/6827
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Optimize MERGE INTO self-merge updates on dataEvolution table [paimon]
Posted by "wayneli-vt (via GitHub)" <gi...@apache.org>.
wayneli-vt closed pull request #6827: [spark] Optimize MERGE INTO self-merge updates on dataEvolution table
URL: https://github.com/apache/paimon/pull/6827
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Optimize MERGE INTO self-merge updates on dataEvolution table [paimon]
Posted by "wayneli-vt (via GitHub)" <gi...@apache.org>.
wayneli-vt commented on PR #6827:
URL: https://github.com/apache/paimon/pull/6827#issuecomment-3674366344
Thank you for your review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Optimize MERGE INTO self-merge updates on dataEvolution table [paimon]
Posted by "wayneli-vt (via GitHub)" <gi...@apache.org>.
wayneli-vt commented on PR #6827:
URL: https://github.com/apache/paimon/pull/6827#issuecomment-3669649637
Thank you for the reply! @JingsongLi
These optimizations are mainly for `computed-column backfill` scenarios. For example, users may have already ingested multi-modal data into a Paimon table and they want to use Spark to process the data, derive some features, and write them back to the same table.
In this case, it should be possible to do `Scan -> Calculate -> Write` directly, since the scan output already satisfies the required layout, so no additional shuffle/sort/join is needed. This PR is focused on optimizing this scenario.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Optimize MERGE INTO self-merge updates on dataEvolution table [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #6827:
URL: https://github.com/apache/paimon/pull/6827#issuecomment-3670395666
@wayneli-vt Thanks for the explanation. I see. Please fix the cases in Spark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org