You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "cloud-fan (via GitHub)" <gi...@apache.org> on 2026/04/09 13:51:01 UTC
[PR] [SPARK-xxxx][SQL] Per-write options should take precedence over session config in Parquet and Avro [spark]
cloud-fan opened a new pull request, #55280:
URL: https://github.com/apache/spark/pull/55280
### What changes were proposed in this pull request?
In `ParquetUtils.prepareWrite` and `AvroUtils.prepareWrite`, several Hadoop configuration keys are unconditionally set from the session-level `SQLConf`. Since write options are already merged into the Hadoop conf upstream via `SessionState.newHadoopConfWithOptions`, these unconditional sets silently overwrite any per-write options the user provided.
This PR introduces a shared `DataSourceUtils.setConfIfAbsent` utility that only sets a key from `SQLConf` if it is not already present in the conf, allowing per-write options to take precedence.
**Parquet keys fixed:**
- `spark.sql.parquet.writeLegacyFormat`
- `spark.sql.parquet.outputTimestampType`
- `spark.sql.parquet.fieldId.write.enabled`
- `spark.sql.legacy.parquet.nanosAsLong`
- `spark.sql.parquet.annotateVariantLogicalType`
**Avro keys fixed:**
- Zstandard buffer pool (`avro.output.codec.zstd.bufferpool`)
- Compression levels (`avro.mapred.<codec>.level`)
### Why are the changes needed?
Per-write options (passed via `DataFrameWriter.option()`) should take precedence over session-level SQLConf defaults. This is already the case for compression codecs in both Parquet and Avro, but other write configuration keys had their per-write values silently overwritten. For example, setting `spark.sql.parquet.outputTimestampType` as a write option had no effect because `prepareWrite` always replaced it with the session config value.
### Does this PR introduce _any_ user-facing change?
Yes. Per-write options for the listed keys now take effect instead of being silently ignored. Previously, only the session-level SQLConf value was used regardless of what was passed as a write option.
### How was this patch tested?
Added an integration test in `ParquetEncodingSuite` that verifies per-write options override session config for `outputTimestampType` (checks the physical Parquet type in the file footer) and `writerVersion` (checks delta encoding is used with PARQUET_2_0).
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (claude-opus-4-6)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56414][SQL] Per-write options should take precedence over session config in file source writes [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan closed pull request #55280: [SPARK-56414][SQL] Per-write options should take precedence over session config in file source writes
URL: https://github.com/apache/spark/pull/55280
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56414][SQL] Per-write options should take precedence over session config in file source writes [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #55280:
URL: https://github.com/apache/spark/pull/55280#issuecomment-4223728310
the docker job failure is unrelated, thanks for review, merging to master!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org