You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "arunb2w (via GitHub)" <gi...@apache.org> on 2023/02/24 11:36:25 UTC
[GitHub] [iceberg] arunb2w opened a new issue, #6928: Merge-on-read vs copy-on-write behavior during merge into
arunb2w opened a new issue, #6928:
URL: https://github.com/apache/iceberg/issues/6928
### Apache Iceberg version
1.1.0 (latest release)
### Query engine
Spark
### Please describe the bug 🐞
I am running a merge-into query and wanted to see how it behaves with write.merge.mode'='copy-on-write' and 'merge-on-read' for the same input on the same cluster config."
From the spark UI, the merge sql took only **2.5mins in copy-on-write** and for the same load **merge-on-read took 11mins.**
Attaching the images to show stage level behavior in which we can see huge shuffle write with merge-on-read whereas it is very minimal with copy-on-write.
When further analysing the SQL tab of the spark UI, were able to find out dynamic-pruning is not happened with MoR whereas we could see that filter in CoW.
My understanding is that, for faster writes we should prefer MoR but in this case MoR is actually performing worst than CoW.
Questions:
Why dynamic pruning is not happening with MoR when running merge into query?
Why shuffle write is huge when using MoR for the same input batch?
How to optimize the MoR performance?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] BsoBird commented on issue #6928: Merge-on-read vs copy-on-write behavior during merge into
Posted by "BsoBird (via GitHub)" <gi...@apache.org>.
BsoBird commented on issue #6928:
URL: https://github.com/apache/iceberg/issues/6928#issuecomment-1520525134
+1
The write performance of MOR tables is worse than COW tables.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Merge-on-read vs copy-on-write behavior during merge into [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6928:
URL: https://github.com/apache/iceberg/issues/6928#issuecomment-1890802421
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Merge-on-read vs copy-on-write behavior during merge into [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #6928:
URL: https://github.com/apache/iceberg/issues/6928#issuecomment-1872632203
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Merge-on-read vs copy-on-write behavior during merge into [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #6928: Merge-on-read vs copy-on-write behavior during merge into
URL: https://github.com/apache/iceberg/issues/6928
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org