You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "xuchang-66 (via GitHub)" <gi...@apache.org> on 2023/12/18 08:44:10 UTC
[I] How to improve write speed for data in the same partition? [iceberg]
xuchang-66 opened a new issue, #9330:
URL: https://github.com/apache/iceberg/issues/9330
### Query engine
spark v3.3
iceberg v1.2.2
### Question
When using Spark SQL to write to Iceberg table, only one task is used to write to each partition. However, when dealing with significantly large partition data (e.g., 20GB in a single partition), the write speed becomes very slow. What settings can be configured to increase concurrency and improve write speed in such scenarios?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] How to improve write speed for data in the same partition? [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #9330: How to improve write speed for data in the same partition?
URL: https://github.com/apache/iceberg/issues/9330
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] How to improve write speed for data in the same partition? [iceberg]
Posted by "atifiu (via GitHub)" <gi...@apache.org>.
atifiu commented on issue #9330:
URL: https://github.com/apache/iceberg/issues/9330#issuecomment-1867510715
I am also facing the same issue when writing data to single partition of iceberg table using dataframe when I write using spark.sql with insert into select * then performance is pretty good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] How to improve write speed for data in the same partition? [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #9330:
URL: https://github.com/apache/iceberg/issues/9330#issuecomment-2599401729
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] How to improve write speed for data in the same partition? [iceberg]
Posted by "TechTinkerer42 (via GitHub)" <gi...@apache.org>.
TechTinkerer42 commented on issue #9330:
URL: https://github.com/apache/iceberg/issues/9330#issuecomment-1868957150
Each partition should be written by only one task to prevent multiple tasks from writing to the same partition, which can lead to the creation of small files.
Here are a few options you can try:
Enable AQE (Adaptive Query Execution) and allow partition splitting for large partitions by setting the following:
spark.sql.adaptive.enabled
spark.sql.adaptive.optimizeSkewsInRebalancePartitions.enabled
spark.sql.adaptive.advisoryPartitionSizeInBytes
Another method is to set the write.distribution-mode to none on the table.
However, be aware that this could result in the creation of small files and potentially degrade query performance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] How to improve write speed for data in the same partition? [iceberg]
Posted by "atifiu (via GitHub)" <gi...@apache.org>.
atifiu commented on issue #9330:
URL: https://github.com/apache/iceberg/issues/9330#issuecomment-1869464036
@xuchang-66 @TechTinkerer42 I have tried to test this with `write.distribution-mode = none` and it can indeed help to improve the performance but can introduce the problem of small files. But we can control number of file created by repartitioning the input data on some column which can also distribute the data in each of the spark partition evenly and number of repartition should be a factor of number of cores. Each spark partition will create one file if we are writing to a single partition.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] How to improve write speed for data in the same partition? [iceberg]
Posted by "coolderli (via GitHub)" <gi...@apache.org>.
coolderli commented on issue #9330:
URL: https://github.com/apache/iceberg/issues/9330#issuecomment-1868649138
try to set `write.distribution-mode` to `none`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] How to improve write speed for data in the same partition? [iceberg]
Posted by "xuchang-66 (via GitHub)" <gi...@apache.org>.
xuchang-66 commented on issue #9330:
URL: https://github.com/apache/iceberg/issues/9330#issuecomment-1869222882
Thanks all, I will have a try.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org