You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "officialasishkumar (via GitHub)" <gi...@apache.org> on 2026/04/08 08:19:02 UTC
[PR] fix(dataframe): handle FixedSizeBinary in describe [datafusion]
officialasishkumar opened a new pull request, #21455:
URL: https://github.com/apache/datafusion/pull/21455
## Which issue does this PR close?
- Closes #20273.
## Rationale for this change
`DataFrame::describe()` builds min/max aggregates for non-numeric columns and then casts the results to `Utf8` for display. That works for strings, but it fails for unsupported binary-like outputs such as `FixedSizeBinary`, which currently causes describe to error instead of falling back to null summary values.
## What changes are included in this PR?
- Treat `FixedSizeBinary` as an unsupported min/max describe type, alongside the other binary-like types that cannot be rendered through the current `Utf8` cast path.
- Reuse the same type predicate for both min and max summary construction.
- Add a regression test covering `describe()` on a `FixedSizeBinary` column.
## Are these changes tested?
- `cargo test -p datafusion --test core_integration describe_fixed_size_binary -- --nocapture`
- `cargo test -p datafusion --test core_integration dataframe::describe:: -- --nocapture`
## Are there any user-facing changes?
`describe()` no longer errors on `FixedSizeBinary` columns; unsupported min/max summaries now fall back to `null` as intended.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix(dataframe): handle FixedSizeBinary in describe [datafusion]
Posted by "officialasishkumar (via GitHub)" <gi...@apache.org>.
officialasishkumar commented on PR #21455:
URL: https://github.com/apache/datafusion/pull/21455#issuecomment-4216410721
@kumarUjjawal Updated the PR body to call out both new tests: `describe_fixed_size_binary` and `describe_mixed_numeric_and_fixed_size_binary`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix(dataframe): handle FixedSizeBinary in describe [datafusion]
Posted by "officialasishkumar (via GitHub)" <gi...@apache.org>.
officialasishkumar commented on code in PR #21455:
URL: https://github.com/apache/datafusion/pull/21455#discussion_r3093963185
##########
datafusion/core/src/dataframe/mod.rs:
##########
@@ -1015,6 +1015,16 @@ impl DataFrame {
//the functions now supported
let supported_describe_functions =
vec!["count", "null_count", "mean", "std", "min", "max", "median"];
+ let supports_describe_min_max = |data_type: &DataType| {
+ !matches!(
+ data_type,
+ DataType::Boolean
+ | DataType::Binary
+ | DataType::LargeBinary
+ | DataType::BinaryView
+ | DataType::FixedSizeBinary(_)
Review Comment:
Updated the describe min/max predicate to use `DataType::is_binary()` while keeping boolean excluded.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix(dataframe): handle FixedSizeBinary in describe [datafusion]
Posted by "kumarUjjawal (via GitHub)" <gi...@apache.org>.
kumarUjjawal commented on PR #21455:
URL: https://github.com/apache/datafusion/pull/21455#issuecomment-4211535791
@officialasishkumar Looks good, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix(dataframe): handle FixedSizeBinary in describe [datafusion]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on code in PR #21455:
URL: https://github.com/apache/datafusion/pull/21455#discussion_r3090932493
##########
datafusion/core/src/dataframe/mod.rs:
##########
@@ -1015,6 +1015,16 @@ impl DataFrame {
//the functions now supported
let supported_describe_functions =
vec!["count", "null_count", "mean", "std", "min", "max", "median"];
+ let supports_describe_min_max = |data_type: &DataType| {
+ !matches!(
+ data_type,
+ DataType::Boolean
+ | DataType::Binary
+ | DataType::LargeBinary
+ | DataType::BinaryView
+ | DataType::FixedSizeBinary(_)
Review Comment:
use [`is_binary`](https://docs.rs/arrow/latest/arrow/datatypes/enum.DataType.html#method.is_binary) here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix(dataframe): handle FixedSizeBinary in describe [datafusion]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on PR #21455:
URL: https://github.com/apache/datafusion/pull/21455#issuecomment-4593291544
superseded by
- https://github.com/apache/datafusion/pull/21728
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix(dataframe): handle FixedSizeBinary in describe [datafusion]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey closed pull request #21455: fix(dataframe): handle FixedSizeBinary in describe
URL: https://github.com/apache/datafusion/pull/21455
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org