You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "Aitozi (via GitHub)" <gi...@apache.org> on 2025/02/22 08:49:07 UTC

[I] [Feature] Improve the spark merge into performance for append table [paimon]

Aitozi opened a new issue, #5136:
URL: https://github.com/apache/paimon/issues/5136

   ### Search before asking
   
   - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
   
   
   ### Motivation
   
   Currently, the merge into will generate the `union` node to union the touched and untouched files from the target table, this can result in longer processing times due to the expensive union shuffle operation.
   
   <img width="1181" alt="Image" src="https://github.com/user-attachments/assets/ce9a8034-7c53-48c9-9bf5-8565feccf50d" />
   
   If we do not have the `notMatchedBySource` action, we do not have to consume the untouched files, so we can eliminate the union stage.
   
   BTW, i found the task parallelism is 42, I don't know why the parallelism is so small, please let me know if you guys have some information.
   
   ### Solution
   
   _No response_
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [x] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Feature] Improve the spark merge into performance for append table [paimon]

Posted by "YannByron (via GitHub)" <gi...@apache.org>.
YannByron closed issue #5136: [Feature] Improve the spark merge into performance for append table
URL: https://github.com/apache/paimon/issues/5136


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org