You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "ryzhyk (via GitHub)" <gi...@apache.org> on 2026/08/30 06:44:20 UTC
[I] `Int96Coercer` drops the metadata of struct, list and map fields. [datafusion]
ryzhyk opened a new issue, #24786:
URL: https://github.com/apache/datafusion/issues/24786
Reading a Parquet file with `datafusion.execution.parquet.coerce_int96` set is supposed to only change the time unit that INT96 timestamp columns decode at. However it also empties the metadata of every struct, list and map field in the file.
## Cause
In `datafusion/datasource-parquet/src/schema_coercion.rs`, leaf fields are cloned:
```rust
// :423
fn field_with_new_type(field: &FieldRef, new_type: DataType) -> FieldRef {
Arc::new(field.as_ref().clone().with_data_type(new_type))
}
```
while container fields are constructed fresh, and `Field::new*` starts from empty metadata:
```rust
// :327
let processed_struct = Field::new_struct(
current_field.name(),
processed_children.as_slice(),
current_field.is_nullable(),
);
// :360
let processed_list = Field::new_list(
current_field.name(),
Arc::clone(&processed_children[0]),
current_field.is_nullable(),
);
// :392
DataType::Map(Arc::clone(&processed_children[0]), *sorted),
```
Name, type and nullability are carried across; `current_field.metadata()` is never read.
A test that demonstrates the issue: https://github.com/ryzhyk/coerce_int96_bug/tree/main
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `Int96Coercer` drops the metadata of struct, list and map fields. [datafusion]
Posted by "iSparshP (via GitHub)" <gi...@apache.org>.
iSparshP commented on issue #24786:
URL: https://github.com/apache/datafusion/issues/24786#issuecomment-5498454714
take
Confirmed the root cause: in `datasource-parquet/src/schema_coercion.rs`, leaf fields are cloned (preserving metadata via `field.clone().with_data_type(...)`), but struct/list/map containers are rebuilt with `Field::new_struct`/`new_list`/`Map`, which start from empty metadata — so `current_field.metadata()` is dropped. Fix is to carry it across at the three construction sites (`.with_metadata(current_field.metadata().clone())`). I'll add a regression test asserting container-field metadata survives `coerce_int96`. PR incoming.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `Int96Coercer` drops the metadata of struct, list and map fields. [datafusion]
Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew closed issue #24786: `Int96Coercer` drops the metadata of struct, list and map fields.
URL: https://github.com/apache/datafusion/issues/24786
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `Int96Coercer` drops the metadata of struct, list and map fields. [datafusion]
Posted by "iSparshP (via GitHub)" <gi...@apache.org>.
iSparshP commented on issue #24786:
URL: https://github.com/apache/datafusion/issues/24786#issuecomment-5504254694
Update: @ryzhyk already has a fix open in #24790 that predates my attempt, so I've closed mine (#24856) in favor of it. Un-claiming this from my side. #24790 looks good and should close this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org