You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "chenkovsky (via GitHub)" <gi...@apache.org> on 2025/04/17 14:39:02 UTC
[PR] fix: parquet coerce_int96 schema [datafusion]
chenkovsky opened a new pull request, #15750:
URL: https://github.com/apache/datafusion/pull/15750
## Which issue does this PR close?
- Closes #15721.
## Rationale for this change
coerce_int96 is ignored when infer schema.
## What changes are included in this PR?
call coerce_int96_to_resolution when fetching schema
## Are these changes tested?
UT
## Are there any user-facing changes?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: parquet coerce_int96 schema [datafusion]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on PR #15750:
URL: https://github.com/apache/datafusion/pull/15750#issuecomment-2813507122
Maybe a title change too: `fix: describe Parquet schema with coerce_int96`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: describe Parquet schema with coerce_int96 [datafusion]
Posted by "chenkovsky (via GitHub)" <gi...@apache.org>.
chenkovsky closed pull request #15750: fix: describe Parquet schema with coerce_int96
URL: https://github.com/apache/datafusion/pull/15750
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: parquet coerce_int96 schema [datafusion]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on code in PR #15750:
URL: https://github.com/apache/datafusion/pull/15750#discussion_r2049184667
##########
datafusion/sqllogictest/test_files/create_external_table.slt:
##########
@@ -297,3 +297,21 @@ CREATE EXTERNAL TABLE staging.foo STORED AS parquet LOCATION '../../parquet-test
# Create external table with qualified name, but no schema should error
statement error DataFusion error: Error during planning: failed to resolve schema: release
CREATE EXTERNAL TABLE release.bar STORED AS parquet LOCATION '../../parquet-testing/data/alltypes_plain.parquet';
+
+
+statement ok
Review Comment:
Can we put this test cast in `parquet.slt` like #15723 does?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: parquet coerce_int96 schema [datafusion]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on code in PR #15750:
URL: https://github.com/apache/datafusion/pull/15750#discussion_r2049184667
##########
datafusion/sqllogictest/test_files/create_external_table.slt:
##########
@@ -297,3 +297,21 @@ CREATE EXTERNAL TABLE staging.foo STORED AS parquet LOCATION '../../parquet-test
# Create external table with qualified name, but no schema should error
statement error DataFusion error: Error during planning: failed to resolve schema: release
CREATE EXTERNAL TABLE release.bar STORED AS parquet LOCATION '../../parquet-testing/data/alltypes_plain.parquet';
+
+
+statement ok
Review Comment:
Can we put this test case in `parquet.slt` like #15723 does?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: parquet coerce_int96 schema [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on PR #15750:
URL: https://github.com/apache/datafusion/pull/15750#issuecomment-2813202353
@mbutrovich cc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: describe Parquet schema with coerce_int96 [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #15750:
URL: https://github.com/apache/datafusion/pull/15750#discussion_r2050828175
##########
datafusion/datasource-parquet/src/file_format.rs:
##########
@@ -825,13 +831,21 @@ async fn fetch_schema(
store: &dyn ObjectStore,
file: &ObjectMeta,
metadata_size_hint: Option<usize>,
+ coerce_int96: Option<TimeUnit>,
) -> Result<Schema> {
let metadata = fetch_parquet_metadata(store, file, metadata_size_hint).await?;
let file_metadata = metadata.file_metadata();
let schema = parquet_to_arrow_schema(
file_metadata.schema_descr(),
file_metadata.key_value_metadata(),
)?;
+ let schema = match coerce_int96 {
Review Comment:
a nit: it might be rewritten more concise ?
```
let schema = coerce_int96
.and_then(|time_unit| coerce_int96_to_resolution(file_metadata.schema_descr(), &schema, &time_unit).ok())
.unwrap_or(schema);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: describe Parquet schema with coerce_int96 [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #15750:
URL: https://github.com/apache/datafusion/pull/15750#issuecomment-2836855789
Nice
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: describe Parquet schema with coerce_int96 [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead merged PR #15750:
URL: https://github.com/apache/datafusion/pull/15750
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org