You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/10/20 07:09:20 UTC
(datafusion) branch gh-readonly-queue/main/pr-26420-a7c7c08365a37f0b46bce81b1fd45515f3f18045 created (now d89a85d3e3)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a change to branch gh-readonly-queue/main/pr-26420-a7c7c08365a37f0b46bce81b1fd45515f3f18045
in repository https://gitbox.apache.org/repos/asf/datafusion.git
at d89a85d3e3 fix: validate parquet.coerce_int96_tz when it is set. (#26420)
This branch includes the following new commits:
new d89a85d3e3 fix: validate parquet.coerce_int96_tz when it is set. (#26420)
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org
(datafusion) 01/01: fix: validate parquet.coerce_int96_tz when it is set. (#26420)
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-26420-a7c7c08365a37f0b46bce81b1fd45515f3f18045
in repository https://gitbox.apache.org/repos/asf/datafusion.git
commit d89a85d3e340062a9efad3d8c4b32f10a7b139c5
Author: akashjainn <80...@users.noreply.github.com>
AuthorDate: Tue Oct 20 07:08:55 2026 +0000
fix: validate parquet.coerce_int96_tz when it is set. (#26420)
## Which issue does this PR close?
- Closes #26362.
## Rationale for this change
`SET datafusion.execution.parquet.coerce_int96_tz = 'Not/A_Timezone'`
succeeds today, and the error only shows up on the next Parquet read.
After #25628 the unit option next to it is checked when it is set, so
this does the same for the timezone.
## What changes are included in this PR?
`ParquetOptions::coerce_int96_tz` is now `Option<ConfigTimeZone>`, the
type `execution.time_zone` already uses, so it accepts exactly what it
accepted before. `parse_coerce_int96_tz_string` is removed, and both
protobuf decoders parse the value.
An invalid value is rejected even when `coerce_int96` is unset, as the
issue asked.
## What is the testing strategy for this PR?
- `set_variable.slt`: SET/SHOW keeps the spelling (`+08`), a bad value
is rejected and leaves the old one, RESET goes back to NULL, and bad
table options fail at CREATE with an inferred and an explicit schema.
These fail on main.
- `test_parquet_coerce_int96_tz_validation` in `config.rs`.
- A rejection test for each protobuf decoder, and the existing
round-trip tests use the typed value.
- The existing timezone read tests in `parquet.slt` pass unchanged.
## Are there any user-facing changes?
Yes. Invalid timezones are rejected when set. The public field type
changes, so this needs `api change`. I added a note to the 56.0.0
upgrade guide.
---
datafusion/common/src/config.rs | 46 ++++++-
datafusion/datasource-parquet/src/file_format.rs | 7 +-
datafusion/datasource-parquet/src/source.rs | 18 +--
datafusion/ffi/src/session/mod.rs | 1 +
datafusion/proto-common/src/from_proto/mod.rs | 45 ++++++-
datafusion/proto-common/src/to_proto/mod.rs | 2 +-
datafusion/proto-models/src/from_proto.rs | 41 +++++-
.../sqllogictest/test_files/information_schema.slt | 2 +-
datafusion/sqllogictest/test_files/parquet.slt | 142 +++++++++++++++++++++
.../sqllogictest/test_files/set_variable.slt | 52 ++++++++
docs/source/library-user-guide/upgrading/56.0.0.md | 25 ++++
docs/source/user-guide/configs.md | 2 +-
12 files changed, 346 insertions(+), 37 deletions(-)
diff --git a/datafusion/common/src/config.rs b/datafusion/common/src/config.rs
index fc7202a6c5..dec26ec8f9 100644
--- a/datafusion/common/src/config.rs
+++ b/datafusion/common/src/config.rs
@@ -1395,8 +1395,8 @@ config_namespace! {
/// values as UTC-adjusted instants, so callers that need the resulting
/// Arrow type to be timezone-aware (e.g. for Spark `TimestampType`
/// semantics) should set this to `"UTC"`. No effect when `coerce_int96`
- /// is `None`.
- pub coerce_int96_tz: Option<String>, default = None
+ /// is `None`, but the value is still validated when it is set.
+ pub coerce_int96_tz: Option<ConfigTimeZone>, default = None
/// (reading) Use any available bloom filters when reading parquet files
pub bloom_filter_on_read: bool, default = true
@@ -5061,6 +5061,48 @@ mod tests {
assert_eq!(scalar, DFTimeUnit::Microsecond);
}
+ #[test]
+ fn test_parquet_coerce_int96_tz_validation() {
+ use crate::config::ConfigOptions;
+
+ const KEY: &str = "datafusion.execution.parquet.coerce_int96_tz";
+ let mut config = ConfigOptions::default();
+ let tz = |config: &ConfigOptions| {
+ config
+ .execution
+ .parquet
+ .coerce_int96_tz
+ .as_ref()
+ .map(|tz| tz.to_string())
+ };
+ assert_eq!(tz(&config), None);
+
+ // Valid values are kept exactly as written, with or without coerce_int96.
+ for value in ["UTC", "America/Los_Angeles", "+08:00", "-0530", "+08"] {
+ config.set(KEY, value).unwrap();
+ assert_eq!(tz(&config).as_deref(), Some(value));
+ }
+
+ // A rejected value leaves the previous one in place.
+ for value in ["Not/A_Timezone", "utc", "08:00", ""] {
+ let err = config.set(KEY, value).unwrap_err();
+ assert_contains!(err.to_string(), format!("Invalid time zone: {value}."));
+ assert_eq!(tz(&config).as_deref(), Some("+08"));
+ }
+
+ // Reset restores None, and an invalid update leaves it unset.
+ config.reset(KEY).unwrap();
+ assert_eq!(tz(&config), None);
+ assert!(config.set(KEY, "Not/A_Timezone").is_err());
+ assert_eq!(tz(&config), None);
+
+ // The option has no nested fields.
+ config.set(KEY, "UTC").unwrap();
+ assert!(config.set(&format!("{KEY}.typo"), "UTC").is_err());
+ assert!(config.reset(&format!("{KEY}.typo")).is_err());
+ assert_eq!(tz(&config).as_deref(), Some("UTC"));
+ }
+
#[cfg(feature = "parquet")]
#[test]
fn set_cdc_enabled_flag() {
diff --git a/datafusion/datasource-parquet/src/file_format.rs b/datafusion/datasource-parquet/src/file_format.rs
index 50e76d2d21..76d0e83134 100644
--- a/datafusion/datasource-parquet/src/file_format.rs
+++ b/datafusion/datasource-parquet/src/file_format.rs
@@ -62,7 +62,7 @@ use datafusion_session::Session;
use crate::metadata::{DFParquetMetadata, lex_ordering_to_sorting_columns};
use crate::reader::CachedParquetFileReaderFactory;
-use crate::source::{ParquetSource, parse_coerce_int96_tz_string};
+use crate::source::ParquetSource;
use async_trait::async_trait;
use bytes::Bytes;
use datafusion_datasource::source::DataSourceExec;
@@ -366,8 +366,7 @@ impl FileFormat for ParquetFormat {
.global
.coerce_int96_tz
.as_ref()
- .map(|tz| parse_coerce_int96_tz_string(tz))
- .transpose()?;
+ .map(|tz| Arc::<str>::from(tz.as_str()));
let file_metadata_cache =
state.runtime_env().cache_manager.get_file_metadata_cache();
@@ -762,7 +761,7 @@ impl From<&ParquetFormatFactory> for protobuf::TableParquetOptions {
parquet_options::CoerceInt96Opt::CoerceInt96(time_unit.to_string())
}),
coerce_int96_tz_opt: global_options.global.coerce_int96_tz.map(|tz| {
- parquet_options::CoerceInt96TzOpt::CoerceInt96Tz(tz)
+ parquet_options::CoerceInt96TzOpt::CoerceInt96Tz(tz.to_string())
}),
max_predicate_cache_size_opt: global_options.global.max_predicate_cache_size.map(|size| {
parquet_options::MaxPredicateCacheSizeOpt::MaxPredicateCacheSize(size as u64)
diff --git a/datafusion/datasource-parquet/src/source.rs b/datafusion/datasource-parquet/src/source.rs
index 0c7c1ee280..fbb6265847 100644
--- a/datafusion/datasource-parquet/src/source.rs
+++ b/datafusion/datasource-parquet/src/source.rs
@@ -37,9 +37,7 @@ use datafusion_datasource::as_file_source;
use datafusion_datasource::file_stream::FileOpener;
use datafusion_datasource::morsel::Morselizer;
-use arrow::array::timezone::Tz;
use arrow::datatypes::TimeUnit;
-use datafusion_common::DataFusionError;
use datafusion_common::config::TableParquetOptions;
use datafusion_common::tree_node::TreeNodeRecursion;
#[cfg(feature = "proto")]
@@ -537,19 +535,6 @@ impl ParquetSource {
}
}
-/// Validates that `tz` is a parseable IANA timezone and returns it as an
-/// `Arc<str>` for use in `Timestamp(_, Some(tz))` types.
-pub(crate) fn parse_coerce_int96_tz_string(
- tz: &str,
-) -> datafusion_common::Result<Arc<str>> {
- tz.parse::<Tz>().map_err(|e| {
- DataFusionError::Configuration(format!(
- "Invalid parquet coerce_int96_tz {tz:?}: {e}"
- ))
- })?;
- Ok(Arc::<str>::from(tz))
-}
-
/// Allows easy conversion from ParquetSource to Arc<dyn FileSource>
impl From<ParquetSource> for Arc<dyn FileSource> {
fn from(source: ParquetSource) -> Self {
@@ -605,8 +590,7 @@ impl FileSource for ParquetSource {
.global
.coerce_int96_tz
.as_ref()
- .map(|tz| parse_coerce_int96_tz_string(tz))
- .transpose()?;
+ .map(|tz| Arc::<str>::from(tz.as_str()));
if coerce_int96_tz.is_some() && coerce_int96.is_none() {
warn!(
"coerce_int96_tz is set but coerce_int96 is not; the timezone will be ignored"
diff --git a/datafusion/ffi/src/session/mod.rs b/datafusion/ffi/src/session/mod.rs
index fcf01d4264..9bbe3e4abd 100644
--- a/datafusion/ffi/src/session/mod.rs
+++ b/datafusion/ffi/src/session/mod.rs
@@ -928,6 +928,7 @@ mod tests {
{
table_options.parquet.global.coerce_int96 =
Some(datafusion_common::parquet_config::DFTimeUnit::Microsecond);
+ table_options.parquet.global.coerce_int96_tz = Some("UTC".parse()?);
}
table_options.current_format = Some(ConfigFileType::JSON);
diff --git a/datafusion/proto-common/src/from_proto/mod.rs b/datafusion/proto-common/src/from_proto/mod.rs
index 24429f2fd1..bcb5de374c 100644
--- a/datafusion/proto-common/src/from_proto/mod.rs
+++ b/datafusion/proto-common/src/from_proto/mod.rs
@@ -1287,9 +1287,9 @@ impl TryFrom<&protobuf::ParquetOptions> for ParquetOptions {
coerce_int96: value.coerce_int96_opt.as_ref().map(|opt| match opt {
protobuf::parquet_options::CoerceInt96Opt::CoerceInt96(v) => v.parse(),
}).transpose()?,
- coerce_int96_tz: value.coerce_int96_tz_opt.clone().map(|opt| match opt {
- protobuf::parquet_options::CoerceInt96TzOpt::CoerceInt96Tz(v) => Some(v),
- }).unwrap_or(None),
+ coerce_int96_tz: value.coerce_int96_tz_opt.as_ref().map(|opt| match opt {
+ protobuf::parquet_options::CoerceInt96TzOpt::CoerceInt96Tz(v) => v.parse(),
+ }).transpose()?,
skip_arrow_metadata: value.skip_arrow_metadata,
write_row_group_number_distinct_values: value.write_row_group_number_distinct_values,
max_predicate_cache_size: value.max_predicate_cache_size_opt.map(|opt| match opt {
@@ -1670,16 +1670,27 @@ mod tests {
assert_eq!(recovered.coerce_int96_tz, None);
}
+ #[test]
+ fn test_parquet_options_coerce_int96_tz_without_unit_round_trip() {
+ let opts = ParquetOptions {
+ coerce_int96_tz: Some("+08".parse().unwrap()),
+ ..ParquetOptions::default()
+ };
+ let recovered = parquet_options_proto_round_trip(opts.clone());
+ assert_eq!(recovered.coerce_int96, None);
+ assert_eq!(recovered.coerce_int96_tz, opts.coerce_int96_tz);
+ }
+
#[test]
fn test_parquet_options_coerce_int96_tz_set_round_trip() {
let opts = ParquetOptions {
coerce_int96: Some(DFTimeUnit::Microsecond),
- coerce_int96_tz: Some("UTC".to_string()),
+ coerce_int96_tz: Some("UTC".parse().unwrap()),
..ParquetOptions::default()
};
let recovered = parquet_options_proto_round_trip(opts.clone());
assert_eq!(recovered.coerce_int96, Some(DFTimeUnit::Microsecond));
- assert_eq!(recovered.coerce_int96_tz, Some("UTC".to_string()));
+ assert_eq!(recovered.coerce_int96_tz, opts.coerce_int96_tz);
}
#[test]
@@ -1773,6 +1784,26 @@ mod tests {
);
}
+ #[test]
+ fn test_invalid_parquet_coerce_int96_tz_rejected_from_proto() {
+ // Rejected even when no coerce_int96 unit is set.
+ let opts = ParquetOptions::default();
+ let mut proto: crate::protobuf_common::ParquetOptions =
+ (&opts).try_into().expect("to_proto");
+ proto.coerce_int96_tz_opt = Some(
+ crate::protobuf_common::parquet_options::CoerceInt96TzOpt::CoerceInt96Tz(
+ "Not/A_Timezone".to_string(),
+ ),
+ );
+
+ let err = ParquetOptions::try_from(&proto).unwrap_err();
+ assert!(
+ err.to_string()
+ .contains("Invalid time zone: Not/A_Timezone."),
+ "unexpected error: {err}"
+ );
+ }
+
#[test]
fn test_invalid_parquet_coerce_int96_rejected_from_proto() {
let opts = ParquetOptions::default();
@@ -1832,12 +1863,12 @@ mod tests {
fn test_table_parquet_options_coerce_int96_tz_round_trip() {
let mut opts = TableParquetOptions::default();
opts.global.coerce_int96 = Some(DFTimeUnit::Microsecond);
- opts.global.coerce_int96_tz = Some("America/Los_Angeles".to_string());
+ opts.global.coerce_int96_tz = Some("America/Los_Angeles".parse().unwrap());
let recovered = table_parquet_options_proto_round_trip(opts.clone());
assert_eq!(recovered.global.coerce_int96, Some(DFTimeUnit::Microsecond));
assert_eq!(
recovered.global.coerce_int96_tz,
- Some("America/Los_Angeles".to_string())
+ opts.global.coerce_int96_tz
);
}
diff --git a/datafusion/proto-common/src/to_proto/mod.rs b/datafusion/proto-common/src/to_proto/mod.rs
index 22b291843d..447a2ac03a 100644
--- a/datafusion/proto-common/src/to_proto/mod.rs
+++ b/datafusion/proto-common/src/to_proto/mod.rs
@@ -968,7 +968,7 @@ impl TryFrom<&ParquetOptions> for protobuf::ParquetOptions {
skip_arrow_metadata: value.skip_arrow_metadata,
write_row_group_number_distinct_values: value.write_row_group_number_distinct_values,
coerce_int96_opt: value.coerce_int96.map(|v| protobuf::parquet_options::CoerceInt96Opt::CoerceInt96(v.to_string())),
- coerce_int96_tz_opt: value.coerce_int96_tz.clone().map(protobuf::parquet_options::CoerceInt96TzOpt::CoerceInt96Tz),
+ coerce_int96_tz_opt: value.coerce_int96_tz.as_ref().map(|v| protobuf::parquet_options::CoerceInt96TzOpt::CoerceInt96Tz(v.to_string())),
max_predicate_cache_size_opt: value.max_predicate_cache_size.map(|v| protobuf::parquet_options::MaxPredicateCacheSizeOpt::MaxPredicateCacheSize(v as u64)),
max_row_group_bytes_opt: value.max_row_group_bytes.map(|v| protobuf::parquet_options::MaxRowGroupBytesOpt::MaxRowGroupBytes(v.get() as u64)),
content_defined_chunking: Some((&value.content_defined_chunking).into()),
diff --git a/datafusion/proto-models/src/from_proto.rs b/datafusion/proto-models/src/from_proto.rs
index 6d3cadc577..e1b37ec046 100644
--- a/datafusion/proto-models/src/from_proto.rs
+++ b/datafusion/proto-models/src/from_proto.rs
@@ -487,10 +487,9 @@ impl TryFrom<&ParquetOptionsProto> for ParquetOptions {
.coerce_int96_tz_opt
.as_ref()
.map(|opt| match opt {
- parquet_options::CoerceInt96TzOpt::CoerceInt96Tz(tz) => {
- tz.clone()
- }
- }),
+ parquet_options::CoerceInt96TzOpt::CoerceInt96Tz(tz) => tz.parse(),
+ })
+ .transpose()?,
max_predicate_cache_size: proto
.max_predicate_cache_size_opt
.as_ref()
@@ -618,4 +617,38 @@ mod tests {
.contains("Invalid parquet statistics setting: invalid")
);
}
+
+ #[test]
+ fn accepts_parquet_coerce_int96_tz_without_unit() {
+ let proto = ParquetOptionsProto {
+ coerce_int96_tz_opt: Some(parquet_options::CoerceInt96TzOpt::CoerceInt96Tz(
+ "+08".to_string(),
+ )),
+ ..Default::default()
+ };
+
+ let options = ParquetOptions::try_from(&proto).unwrap();
+ assert_eq!(options.coerce_int96, None);
+ assert_eq!(
+ options.coerce_int96_tz.as_ref().map(|tz| tz.as_str()),
+ Some("+08")
+ );
+ }
+
+ #[test]
+ fn rejects_invalid_parquet_coerce_int96_tz() {
+ let proto = ParquetOptionsProto {
+ coerce_int96_tz_opt: Some(parquet_options::CoerceInt96TzOpt::CoerceInt96Tz(
+ "Not/A_Timezone".to_string(),
+ )),
+ ..Default::default()
+ };
+
+ let err = ParquetOptions::try_from(&proto).unwrap_err();
+ assert!(
+ err.to_string()
+ .contains("Invalid time zone: Not/A_Timezone."),
+ "unexpected error: {err}"
+ );
+ }
}
diff --git a/datafusion/sqllogictest/test_files/information_schema.slt b/datafusion/sqllogictest/test_files/information_schema.slt
index 25ada65f69..fb6d2d9dc7 100644
--- a/datafusion/sqllogictest/test_files/information_schema.slt
+++ b/datafusion/sqllogictest/test_files/information_schema.slt
@@ -403,7 +403,7 @@ datafusion.execution.parquet.bloom_filter_ndv NULL (writing) Sets bloom filter n
datafusion.execution.parquet.bloom_filter_on_read true (reading) Use any available bloom filters when reading parquet files
datafusion.execution.parquet.bloom_filter_on_write false (writing) Write bloom filters for all columns when creating parquet files
datafusion.execution.parquet.coerce_int96 NULL (reading) If set, parquet reader will read columns of physical type int96 as originating from this resolution instead of nanosecond. Valid values are: ns, us, ms, and s. This is useful for reading data from systems like Spark which store microsecond resolution timestamps in an int96, allowing them to write values with a larger date range than 64-bit timestamps with nanosecond resolution.
-datafusion.execution.parquet.coerce_int96_tz NULL (reading) Optional timezone applied to INT96 columns when `coerce_int96` is set. When `Some`, INT96 columns coerce to `Timestamp(<coerce_int96>, Some(<tz>))` instead of the default `Timestamp(<coerce_int96>, None)`. Spark and other systems write INT96 values as UTC-adjusted instants, so callers that need the resulting Arrow type to be timezone-aware (e.g. for Spark `TimestampType` semantics) should set this to `"UTC"`. No effect when `coe [...]
+datafusion.execution.parquet.coerce_int96_tz NULL (reading) Optional timezone applied to INT96 columns when `coerce_int96` is set. When `Some`, INT96 columns coerce to `Timestamp(<coerce_int96>, Some(<tz>))` instead of the default `Timestamp(<coerce_int96>, None)`. Spark and other systems write INT96 values as UTC-adjusted instants, so callers that need the resulting Arrow type to be timezone-aware (e.g. for Spark `TimestampType` semantics) should set this to `"UTC"`. No effect when `coe [...]
datafusion.execution.parquet.column_index_truncate_length 64 (writing) Sets column index truncate length
datafusion.execution.parquet.compression zstd(3) (writing) Sets default parquet compression codec. Valid values are: uncompressed, snappy, gzip(level), brotli(level), lz4, zstd(level), and lz4_raw. These values are not case sensitive. If NULL, uses default parquet writer setting Note that this default setting is not the same as the default parquet writer setting.
datafusion.execution.parquet.content_defined_chunking.enabled false (writing) EXPERIMENTAL: Enable content-defined chunking (CDC) when writing parquet files. When enabled, parallel writing is automatically disabled since the chunker state must persist across row groups.
diff --git a/datafusion/sqllogictest/test_files/parquet.slt b/datafusion/sqllogictest/test_files/parquet.slt
index 9750038fed..5620157dc9 100644
--- a/datafusion/sqllogictest/test_files/parquet.slt
+++ b/datafusion/sqllogictest/test_files/parquet.slt
@@ -784,9 +784,151 @@ ERROR: Cast error: Failed to convert -9357363680509551 to datetime for Timestamp
statement ok
drop table int96_from_spark;
+# Without coerce_int96 the timezone has no effect: same type and values as the
+# default read above.
+statement ok
+reset datafusion.execution.parquet.coerce_int96;
+
+statement ok
+CREATE EXTERNAL TABLE int96_from_spark
+STORED AS PARQUET
+LOCATION '../../parquet-testing/data/int96_from_spark.parquet';
+
+query TTT
+describe int96_from_spark;
+----
+a Timestamp(ns) YES
+
+query P
+select * from int96_from_spark
+----
+2024-01-01T20:34:56.123456
+2024-01-01T01:00:00
+1816-03-29T08:56:08.066277376
+2024-12-30T23:00:00
+NULL
+1815-11-08T16:01:01.191053312
+
+statement ok
+drop table int96_from_spark;
+
+# An explicit nanosecond unit applies the timezone.
+statement ok
+set datafusion.execution.parquet.coerce_int96 = ns;
+
+statement ok
+CREATE EXTERNAL TABLE int96_from_spark
+STORED AS PARQUET
+LOCATION '../../parquet-testing/data/int96_from_spark.parquet';
+
+query TTT
+describe int96_from_spark;
+----
+a Timestamp(ns, "UTC") YES
+
+query P
+select * from int96_from_spark
+----
+2024-01-01T20:34:56.123456Z
+2024-01-01T01:00:00Z
+1816-03-29T08:56:08.066277376Z
+2024-12-30T23:00:00Z
+NULL
+1815-11-08T16:01:01.191053312Z
+
+statement ok
+drop table int96_from_spark;
+
+# Microseconds with the timezone. The last value is out of range for display,
+# as in the `coerce_int96 = ms` blocks above.
+statement ok
+set datafusion.execution.parquet.coerce_int96 = us;
+
+statement ok
+CREATE EXTERNAL TABLE int96_from_spark
+STORED AS PARQUET
+LOCATION '../../parquet-testing/data/int96_from_spark.parquet';
+
+query TTT
+describe int96_from_spark;
+----
+a Timestamp(µs, "UTC") YES
+
+query P
+select * from int96_from_spark
+----
+2024-01-01T20:34:56.123456Z
+2024-01-01T01:00:00Z
+9999-12-31T03:00:00Z
+2024-12-30T23:00:00Z
+NULL
+ERROR: Cast error: Failed to convert 9089380393200000000 to datetime for Timestamp(µs, "UTC")
+
+statement ok
+drop table int96_from_spark;
+
+# An explicit schema with the same type reads the same values. `WITH TIME ZONE`
+# takes the session time zone, so set it to match.
+statement ok
+set datafusion.execution.time_zone = 'UTC';
+
+statement ok
+CREATE EXTERNAL TABLE int96_from_spark (a TIMESTAMP(6) WITH TIME ZONE)
+STORED AS PARQUET
+LOCATION '../../parquet-testing/data/int96_from_spark.parquet';
+
+query TTT
+describe int96_from_spark;
+----
+a Timestamp(µs, "UTC") YES
+
+query P
+select * from int96_from_spark
+----
+2024-01-01T20:34:56.123456Z
+2024-01-01T01:00:00Z
+9999-12-31T03:00:00Z
+2024-12-30T23:00:00Z
+NULL
+ERROR: Cast error: Failed to convert 9089380393200000000 to datetime for Timestamp(µs, "UTC")
+
+statement ok
+drop table int96_from_spark;
+
+statement ok
+reset datafusion.execution.time_zone;
+
+# A non-UTC offset keeps the stored instant and renders it at that offset,
+# eight hours ahead of the UTC values above.
statement ok
set datafusion.execution.parquet.coerce_int96 = ns;
+statement ok
+set datafusion.execution.parquet.coerce_int96_tz = '+08:00';
+
+statement ok
+CREATE EXTERNAL TABLE int96_from_spark
+STORED AS PARQUET
+LOCATION '../../parquet-testing/data/int96_from_spark.parquet';
+
+query TTT
+describe int96_from_spark;
+----
+a Timestamp(ns, "+08:00") YES
+
+query P
+select * from int96_from_spark
+----
+2024-01-02T04:34:56.123456+08:00
+2024-01-01T09:00:00+08:00
+1816-03-29T16:56:08.066277376+08:00
+2024-12-31T07:00:00+08:00
+NULL
+1815-11-09T00:01:01.191053312+08:00
+
+statement ok
+drop table int96_from_spark;
+
statement ok
reset datafusion.execution.parquet.coerce_int96_tz;
diff --git a/datafusion/sqllogictest/test_files/set_variable.slt b/datafusion/sqllogictest/test_files/set_variable.slt
index 307de8c24c..6e793b8661 100644
--- a/datafusion/sqllogictest/test_files/set_variable.slt
+++ b/datafusion/sqllogictest/test_files/set_variable.slt
@@ -862,6 +862,58 @@ OPTIONS ('format.coerce_int96' 'nanos');
DataFusion error: Invalid or Unsupported Configuration: Invalid parquet coerce_int96 setting: nanos. Expected one of: ns, us, ms, s
+# parquet.coerce_int96_tz is validated when it is set, even without
+# coerce_int96
+statement ok
+SET datafusion.execution.parquet.coerce_int96_tz = '+08'
+
+query TT
+SHOW datafusion.execution.parquet.coerce_int96_tz
+----
+datafusion.execution.parquet.coerce_int96_tz +08
+
+statement error
+SET datafusion.execution.parquet.coerce_int96_tz = 'Not/A_Timezone'
+----
+DataFusion error: Error setting config datafusion.execution.parquet.coerce_int96_tz
+caused by
+Invalid or Unsupported Configuration: Invalid time zone: Not/A_Timezone. Valid values are UTC offsets such as +08:00 or IANA time zone names such as Asia/Taipei
+
+
+# The rejected value did not change the setting
+query TT
+SHOW datafusion.execution.parquet.coerce_int96_tz
+----
+datafusion.execution.parquet.coerce_int96_tz +08
+
+statement ok
+RESET datafusion.execution.parquet.coerce_int96_tz
+
+query TT
+SHOW datafusion.execution.parquet.coerce_int96_tz
+----
+datafusion.execution.parquet.coerce_int96_tz NULL
+
+# Table options are validated at CREATE time, with an inferred or an explicit
+# schema
+statement error
+CREATE EXTERNAL TABLE int96_bad_tz
+STORED AS PARQUET
+LOCATION '../../parquet-testing/data/int96_from_spark.parquet'
+OPTIONS ('format.coerce_int96_tz' 'Not/A_Timezone');
+----
+DataFusion error: Invalid or Unsupported Configuration: Invalid time zone: Not/A_Timezone. Valid values are UTC offsets such as +08:00 or IANA time zone names such as Asia/Taipei
+
+
+statement error
+CREATE EXTERNAL TABLE int96_bad_tz (a TIMESTAMP)
+STORED AS PARQUET
+LOCATION '../../parquet-testing/data/int96_from_spark.parquet'
+OPTIONS ('format.coerce_int96' 'us', 'format.coerce_int96_tz' 'Not/A_Timezone');
+----
+DataFusion error: Invalid or Unsupported Configuration: Invalid time zone: Not/A_Timezone. Valid values are UTC offsets such as +08:00 or IANA time zone names such as Asia/Taipei
+
+
statement error DataFusion error: Error setting config datafusion\.execution\.batch_size\ncaused by\nInvalid or Unsupported Configuration: value must be greater than 0
SET datafusion.execution.batch_size = 0
diff --git a/docs/source/library-user-guide/upgrading/56.0.0.md b/docs/source/library-user-guide/upgrading/56.0.0.md
index 648071a251..8f5604a5cd 100644
--- a/docs/source/library-user-guide/upgrading/56.0.0.md
+++ b/docs/source/library-user-guide/upgrading/56.0.0.md
@@ -403,6 +403,31 @@ options.execution.time_zone = Some("America/Denver".parse()?);
let tz: Option<&str> = options.execution.time_zone.as_ref().map(|tz| tz.as_str());
```
+### `ParquetOptions::coerce_int96_tz` is now a validated `ConfigTimeZone`
+
+`ParquetOptions::coerce_int96_tz` changed from `Option<String>` to
+`Option<ConfigTimeZone>`, the same type as `ExecutionOptions::time_zone`. An
+invalid time zone is now rejected by `SET`, by table options, and by protobuf
+decoding instead of on the first Parquet read. This applies even when
+`coerce_int96` is not set, where the time zone has no effect. Valid values and
+the protobuf wire format are unchanged.
+
+**Who is affected:**
+
+- Code that reads or assigns `ParquetOptions::coerce_int96_tz` directly.
+- Code that sets the value through SQL or `ConfigOptions::set` needs no code
+ changes, but invalid time zones are now rejected immediately.
+
+**Migration guide:**
+
+```rust,ignore
+// Before
+options.execution.parquet.coerce_int96_tz = Some("UTC".to_string());
+
+// After
+options.execution.parquet.coerce_int96_tz = Some("UTC".parse()?);
+```
+
### `ExecutionOptions` has a new `soft_max_bytes_per_output_file` field
`ExecutionOptions` gained a public `soft_max_bytes_per_output_file: ConfigNonZeroUsize`
diff --git a/docs/source/user-guide/configs.md b/docs/source/user-guide/configs.md
index 44868ae2c9..27356ab8c1 100644
--- a/docs/source/user-guide/configs.md
+++ b/docs/source/user-guide/configs.md
@@ -90,7 +90,7 @@ The following configuration settings are available:
| datafusion.execution.parquet.schema_force_view_types | true | (reading) If true, parquet reader will read columns of `Utf8/Utf8Large` with `Utf8View`, and `Binary/BinaryLarge` with `BinaryView`. The parquet reader is optimized for reading `Utf8View` and `BinaryView`, so such queries are significantly faster than reading `Utf8`/`Binary` and then casting to the view types. [...]
| datafusion.execution.parquet.binary_as_string | false | (reading) If true, parquet reader will read columns of `Binary/LargeBinary` with `Utf8`, and `BinaryView` with `Utf8View`. Parquet files generated by some legacy writers do not correctly set the UTF8 flag for strings, causing string columns to be loaded as BLOB instead. The parquet reader has special optimizations for `Utf8` validation, so reading such columns as strings is significant [...]
| datafusion.execution.parquet.coerce_int96 | NULL | (reading) If set, parquet reader will read columns of physical type int96 as originating from this resolution instead of nanosecond. Valid values are: ns, us, ms, and s. This is useful for reading data from systems like Spark which store microsecond resolution timestamps in an int96, allowing them to write values with a larger date range than 64-bit timestamps with nanosecond resolutio [...]
-| datafusion.execution.parquet.coerce_int96_tz | NULL | (reading) Optional timezone applied to INT96 columns when `coerce_int96` is set. When `Some`, INT96 columns coerce to `Timestamp(<coerce_int96>, Some(<tz>))` instead of the default `Timestamp(<coerce_int96>, None)`. Spark and other systems write INT96 values as UTC-adjusted instants, so callers that need the resulting Arrow type to be timezone-aware (e.g. for Spark `TimestampType` sema [...]
+| datafusion.execution.parquet.coerce_int96_tz | NULL | (reading) Optional timezone applied to INT96 columns when `coerce_int96` is set. When `Some`, INT96 columns coerce to `Timestamp(<coerce_int96>, Some(<tz>))` instead of the default `Timestamp(<coerce_int96>, None)`. Spark and other systems write INT96 values as UTC-adjusted instants, so callers that need the resulting Arrow type to be timezone-aware (e.g. for Spark `TimestampType` sema [...]
| datafusion.execution.parquet.bloom_filter_on_read | true | (reading) Use any available bloom filters when reading parquet files [...]
| datafusion.execution.parquet.max_predicate_cache_size | NULL | (reading) The maximum predicate cache size, in bytes. When `pushdown_filters` is enabled, sets the maximum memory used to cache the results of predicate evaluation between filter evaluation and output generation. Decreasing this value will reduce memory usage, but may increase IO and CPU usage. None means use the default parquet reader setting. 0 means no caching. [...]
| datafusion.execution.parquet.max_in_list_size | 20 | Maximum number of input values in an `IN (...)` list eligible for min/max pruning. Lists above this cap, or a cap of 0, skip this rewrite; other predicates and Bloom-filter pruning remain available. Within the cap, nonempty lists of at most 20 values use the existing per-value rewrite. Larger literal lists use a compact representation when the column type is string, variable-length bin [...]
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org