You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "chenkovsky (via GitHub)" <gi...@apache.org> on 2025/04/16 12:47:31 UTC
[PR] fix: serialize listing table without partition column [datafusion]
chenkovsky opened a new pull request, #15737:
URL: https://github.com/apache/datafusion/pull/15737
## Which issue does this PR close?
- Closes #15718.
## Rationale for this change
partition columns should exclude file schema in listing table
## What changes are included in this PR?
1. update partition column type in proto
2. remove partition columns from ListingTableScanNode schema.
## Are these changes tested?
UT
## Are there any user-facing changes?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: serialize listing table without partition column [datafusion]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on PR #15737:
URL: https://github.com/apache/datafusion/pull/15737#issuecomment-2812298253
Thanks @chenkovsky i think changes make sense, very clean implementation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: serialize listing table without partition column [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #15737:
URL: https://github.com/apache/datafusion/pull/15737#issuecomment-2813800026
Thanks again @chenkovsky
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: serialize listing table without partition column [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #15737:
URL: https://github.com/apache/datafusion/pull/15737
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: serialize listing table without partition column [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #15737:
URL: https://github.com/apache/datafusion/pull/15737#discussion_r2049073244
##########
datafusion/proto/src/logical_plan/mod.rs:
##########
@@ -458,23 +458,24 @@ impl AsLogicalPlan for LogicalPlanNode {
.map(ListingTableUrl::parse)
.collect::<Result<Vec<_>, _>>()?;
+ let partition_columns = scan
+ .table_partition_cols
+ .iter()
+ .map(|col| {
+ let arrow_type =
+ col.arrow_type.as_ref().unwrap().try_into().map_err(|e| {
+ proto_error(format!(
+ "Received an unknown ArrowType: {}",
+ e
+ ))
+ })?;
Review Comment:
I think it would be good to avoid an unwrap here as that will panic rather than returning an error on invalid protobuf
```suggestion
let Some(arrow_type) = col.arrow_type.as_ref() else {
return Err(proto_error(format!(
"Missing Arrow type in partition columns"
)));
};
let arrow_type = DataType::try_from(arrow_type).map_err(|e| {
proto_error(format!("Received an unknown ArrowType: {}", e))
})?;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org