You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "pp-eyushin (via GitHub)" <gi...@apache.org> on 2025/10/20 14:20:14 UTC
[I] Respect write.parquet.row-group-size-bytes table property for Spark integration [iceberg]
pp-eyushin opened a new issue, #14377:
URL: https://github.com/apache/iceberg/issues/14377
### Feature Request / Improvement
Current Spark integration doesn't seem to support `write.parquet.row-group-size-bytes` table property. As a workaround, we use extra call to [rewrite_data_files](https://iceberg.apache.org/docs/latest/spark-procedures/#rewrite_data_files) procedure to align actual underlying parquet files layout after Spark complete its writes, which is sub-optimal.
This is crucial when we want to limit parallelism and reduce amount of data to be fed to single Spark task reading the table to trade-off OutOfMemory errors.
### Query engine
Spark
### Willingness to contribute
- [x] I can contribute this improvement/feature independently
- [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
- [ ] I cannot contribute this improvement/feature at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Respect write.parquet.row-group-size-bytes table property for Spark integration [iceberg]
Posted by "pp-eyushin (via GitHub)" <gi...@apache.org>.
pp-eyushin commented on issue #14377:
URL: https://github.com/apache/iceberg/issues/14377#issuecomment-3431907251
> [@pp-eyushin](https://github.com/pp-eyushin) SparkWrite delegates to Parquet writer which should read all parquet configs in the table properties.
Indeed, thanks for clarification! I was confused we have subset of metrics to override for Spark plugin.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Respect write.parquet.row-group-size-bytes table property for Spark integration [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on issue #14377:
URL: https://github.com/apache/iceberg/issues/14377#issuecomment-3427385787
SparkWrite delegates to Parquet writer which should read all parquet configs in the table properties.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Respect write.parquet.row-group-size-bytes table property for Spark integration [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on issue #14377:
URL: https://github.com/apache/iceberg/issues/14377#issuecomment-3426477839
Both are respected with write options taking precedence.
https://github.com/apache/iceberg/blob/fa4890e0f923ae1f9fbf0065d2430e21e77174f3/spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/SparkWriteConf.java#L189-L196
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Respect write.parquet.row-group-size-bytes table property for Spark integration [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on issue #14377:
URL: https://github.com/apache/iceberg/issues/14377#issuecomment-3424344713
@pp-eyushin Isn't it already supported as per https://iceberg.apache.org/docs/nightly/configuration/#write-properties?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Respect write.parquet.row-group-size-bytes table property for Spark integration [iceberg]
Posted by "pp-eyushin (via GitHub)" <gi...@apache.org>.
pp-eyushin closed issue #14377: Respect write.parquet.row-group-size-bytes table property for Spark integration
URL: https://github.com/apache/iceberg/issues/14377
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Respect write.parquet.row-group-size-bytes table property for Spark integration [iceberg]
Posted by "pp-eyushin (via GitHub)" <gi...@apache.org>.
pp-eyushin commented on issue #14377:
URL: https://github.com/apache/iceberg/issues/14377#issuecomment-3425710734
> [@pp-eyushin](https://github.com/pp-eyushin) Isn't it already supported as per https://iceberg.apache.org/docs/nightly/configuration/#write-properties?
It's [table properties](https://iceberg.apache.org/docs/latest/configuration/#write-properties), Spark has its own [counterparts](https://iceberg.apache.org/docs/latest/spark-configuration/#write-options).
For example, `write.target-file-size-bytes` table property is mapped to `target-file-size-bytes` in Spark:
https://github.com/apache/iceberg/blob/71cbe17cea5cac567920779de924eba62432934b/spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/SparkWriteOptions.java#L30
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Respect write.parquet.row-group-size-bytes table property for Spark integration [iceberg]
Posted by "pp-eyushin (via GitHub)" <gi...@apache.org>.
pp-eyushin commented on issue #14377:
URL: https://github.com/apache/iceberg/issues/14377#issuecomment-3422298672
@aokolnychyi I see you contributed quite a lot for `SparkWriteOptions`, do you think this feature would be a good addition to the current functionality?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Respect write.parquet.row-group-size-bytes table property for Spark integration [iceberg]
Posted by "pp-eyushin (via GitHub)" <gi...@apache.org>.
pp-eyushin commented on issue #14377:
URL: https://github.com/apache/iceberg/issues/14377#issuecomment-3426879908
@manuzhang that's true for `TARGET_FILE_SIZE_BYTES`, but what's for `write.parquet.row-group-size-bytes` which is the topic of this feature request?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org