You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "xiedeyantu (via GitHub)" <gi...@apache.org> on 2026/03/19 16:29:45 UTC
[PR] fix: preserve duplicate GROUPING SETS rows [datafusion]
xiedeyantu opened a new pull request, #21058:
URL: https://github.com/apache/datafusion/pull/21058
## Which issue does this PR close?
- Closes #.
## Rationale for this change
`GROUPING SETS` with duplicate grouping lists were incorrectly collapsed during execution. The internal grouping id only encoded the null mask, so repeated grouping sets shared the same key and were merged, which caused rows to be lost compared with PostgreSQL behavior.
I verified the expected result in PostgreSQL, where the same query returns 30 rows for the full `emp` example instead of being collapsed.
## What changes are included in this PR?
- Preserve duplicate grouping sets by adding a per-occurrence ordinal into the internal grouping id used during execution.
- Keep `GROUPING()` semantics unchanged.
- Add regression coverage for the duplicate `GROUPING SETS` case in:
- `datafusion/core/tests/sql/aggregates/basic.rs`
- `datafusion/sqllogictest/test_files/group_by.slt`
## Are these changes tested?
- `cargo fmt --all`
- `cargo test -p datafusion duplicate_grouping_sets_are_preserved -- --nocapture`
- PostgreSQL validation against the same query/result shape
## Are there any user-facing changes?
- Yes. Queries that contain duplicate `GROUPING SETS` entries now return the correct duplicated result rows, matching PostgreSQL behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: preserve duplicate GROUPING SETS rows [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #21058:
URL: https://github.com/apache/datafusion/pull/21058#issuecomment-4103312202
Thanks @xiedeyantu -- this is great. As before, can you please:
1. File a ticket showing the bug and the postgres behavior as well? This will make it much easier / faster to review these PRs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: preserve duplicate GROUPING SETS rows [datafusion]
Posted by "neilconway (via GitHub)" <gi...@apache.org>.
neilconway commented on code in PR #21058:
URL: https://github.com/apache/datafusion/pull/21058#discussion_r3027989377
##########
datafusion/core/src/dataframe/mod.rs:
##########
@@ -654,14 +654,14 @@ impl DataFrame {
.aggregate(group_expr, aggr_expr)?
.build()?;
let plan = if is_grouping_set {
- let grouping_id_pos = plan.schema().fields().len() - 1 - aggr_expr_len;
+ let grouping_id_pos = plan.schema().fields().len() - 2 - aggr_expr_len;
Review Comment:
`2` magic number in a few places is a bit inscrutable. Maybe create a named constant?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: preserve duplicate GROUPING SETS rows [datafusion]
Posted by "xiedeyantu (via GitHub)" <gi...@apache.org>.
xiedeyantu commented on PR #21058:
URL: https://github.com/apache/datafusion/pull/21058#issuecomment-4103332182
> Thanks @xiedeyantu -- this is great. As before, can you please:
>
> 1. File a ticket showing the bug and the postgres behavior as well? This will make it much easier / faster to review these PRs
I completely agree. I may not be very familiar with DataFusion's contribution process yet. Thank you for your thoughtful suggestions. I might add the necessary information in a few days, as I'm currently on vacation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: preserve duplicate GROUPING SETS rows [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #21058:
URL: https://github.com/apache/datafusion/pull/21058
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: preserve duplicate GROUPING SETS rows [datafusion]
Posted by "neilconway (via GitHub)" <gi...@apache.org>.
neilconway commented on PR #21058:
URL: https://github.com/apache/datafusion/pull/21058#issuecomment-4202058168
@xiedeyantu Can you update the PR description? It still talks about the old approach.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: preserve duplicate GROUPING SETS rows [datafusion]
Posted by "neilconway (via GitHub)" <gi...@apache.org>.
neilconway commented on code in PR #21058:
URL: https://github.com/apache/datafusion/pull/21058#discussion_r2991557200
##########
datafusion/physical-plan/src/aggregates/mod.rs:
##########
@@ -1937,15 +1937,53 @@ fn evaluate_optional(
.collect()
}
-fn group_id_array(group: &[bool], batch: &RecordBatch) -> Result<ArrayRef> {
+fn group_id_array(
+ group: &[bool],
+ group_ordinal: usize,
+ batch: &RecordBatch,
+) -> Result<ArrayRef> {
if group.len() > 64 {
return not_impl_err!(
"Grouping sets with more than 64 columns are not supported"
);
}
+ let width_bits = if group.len() <= 8 {
+ 8
+ } else if group.len() <= 16 {
+ 16
+ } else if group.len() <= 32 {
+ 32
+ } else {
+ 64
+ };
+ let extra_bits = width_bits - group.len();
+ if extra_bits == 0 && group_ordinal > 0 {
+ return not_impl_err!(
+ "Duplicate grouping sets with more than {} grouping columns are not supported",
Review Comment:
If `group.len() == 64`, seems like this will produce an inaccurate error message ("more than 64" vs "64").
##########
datafusion/physical-plan/src/aggregates/mod.rs:
##########
@@ -1937,15 +1937,53 @@ fn evaluate_optional(
.collect()
}
-fn group_id_array(group: &[bool], batch: &RecordBatch) -> Result<ArrayRef> {
+fn group_id_array(
+ group: &[bool],
+ group_ordinal: usize,
+ batch: &RecordBatch,
+) -> Result<ArrayRef> {
if group.len() > 64 {
return not_impl_err!(
"Grouping sets with more than 64 columns are not supported"
);
}
+ let width_bits = if group.len() <= 8 {
+ 8
+ } else if group.len() <= 16 {
+ 16
+ } else if group.len() <= 32 {
+ 32
+ } else {
+ 64
+ };
+ let extra_bits = width_bits - group.len();
+ if extra_bits == 0 && group_ordinal > 0 {
+ return not_impl_err!(
+ "Duplicate grouping sets with more than {} grouping columns are not supported",
+ width_bits
+ );
+ }
+ if extra_bits < usize::BITS as usize {
+ let max_group_ordinal = 1usize << extra_bits;
+ if group_ordinal >= max_group_ordinal {
+ return not_impl_err!(
+ "Duplicate grouping sets exceed the supported grouping id capacity"
+ );
+ }
+ }
let group_id = group.iter().fold(0u64, |acc, &is_null| {
(acc << 1) | if is_null { 1 } else { 0 }
});
+ let group_id = if group.len() == 64 {
Review Comment:
Personally, I would opt for tightening the check on `group.len() > 64` to `group.len() >= 64` at the top of the function, which would simplify this logic.
##########
datafusion/physical-plan/src/aggregates/mod.rs:
##########
@@ -1972,6 +2010,7 @@ pub fn evaluate_group_by(
group_by: &PhysicalGroupBy,
batch: &RecordBatch,
) -> Result<Vec<Vec<ArrayRef>>> {
+ let mut group_ordinals: HashMap<Vec<bool>, usize> = HashMap::new();
Review Comment:
`HashMap<&[bool], usize>` instead?
##########
datafusion/physical-plan/src/aggregates/mod.rs:
##########
@@ -1937,15 +1937,53 @@ fn evaluate_optional(
.collect()
}
-fn group_id_array(group: &[bool], batch: &RecordBatch) -> Result<ArrayRef> {
+fn group_id_array(
Review Comment:
I think this function would benefit from some commentary: you need to read the implementation pretty carefully to understand the encoded bit format.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: preserve duplicate GROUPING SETS rows [datafusion]
Posted by "xiedeyantu (via GitHub)" <gi...@apache.org>.
xiedeyantu commented on PR #21058:
URL: https://github.com/apache/datafusion/pull/21058#issuecomment-4133099021
@neilconway Thank you for the thorough review. I will complete these modifications this weekend and update the problem description accordingly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: preserve duplicate GROUPING SETS rows [datafusion]
Posted by "xiedeyantu (via GitHub)" <gi...@apache.org>.
xiedeyantu commented on PR #21058:
URL: https://github.com/apache/datafusion/pull/21058#issuecomment-4190591441
@neilconway I have made the changes based on the comments; could you please review it again? Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: preserve duplicate GROUPING SETS rows [datafusion]
Posted by "xiedeyantu (via GitHub)" <gi...@apache.org>.
xiedeyantu commented on PR #21058:
URL: https://github.com/apache/datafusion/pull/21058#issuecomment-4148408141
> It's a little unfortunate that we don't support duplicate grouping sets for 8/16/32 grouping columns. I guess there's no easy way around that?
>
> `grouping_function_on_id` has this code:
>
> ```rust
> let grouping_id_column = Expr::Column(Column::from(Aggregate::INTERNAL_GROUPING_ID));
> // The grouping call is exactly our internal grouping id
> if args.len() == group_by_expr_count
> && args
> .iter()
> .rev()
> .enumerate()
> .all(|(idx, expr)| group_by_expr.get(expr) == Some(&idx))
> {
> return Ok(cast(grouping_id_column, DataType::Int32));
> }
> ```
>
> We probably want to mask out the ordinal bits that this PR adds in to the grouping ID?
@neilconway Apologies for the delayed response. I have reviewed all your thoughtful comments and agree that the approach using high-bit flags indeed has significant flaws. I have now refactored the entire logic and updated the PR description; if you have the time, could you please help review it again? Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: preserve duplicate GROUPING SETS rows [datafusion]
Posted by "neilconway (via GitHub)" <gi...@apache.org>.
neilconway commented on code in PR #21058:
URL: https://github.com/apache/datafusion/pull/21058#discussion_r2991095057
##########
datafusion/core/tests/sql/aggregates/basic.rs:
##########
@@ -175,6 +175,57 @@ async fn count_aggregated_cube() -> Result<()> {
Ok(())
}
+#[tokio::test]
+async fn duplicate_grouping_sets_are_preserved() -> Result<()> {
Review Comment:
Seems like this duplicates the SLT test that this PR already adds, so I think it can be omitted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: preserve duplicate GROUPING SETS rows [datafusion]
Posted by "xiedeyantu (via GitHub)" <gi...@apache.org>.
xiedeyantu commented on PR #21058:
URL: https://github.com/apache/datafusion/pull/21058#issuecomment-4171000122
@neilconway Could you please help me to have a look again?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: preserve duplicate GROUPING SETS rows [datafusion]
Posted by "xiedeyantu (via GitHub)" <gi...@apache.org>.
xiedeyantu commented on PR #21058:
URL: https://github.com/apache/datafusion/pull/21058#issuecomment-4178647113
> Sorry for the delayed response, @xiedeyantu !
>
> Thanks for revising this. I'm a bit concerned by the overhead here; we are added a `UInt32` column to _every_ query with grouping sets just to handle a relatively rare situation.
>
> I actually liked the approach you took in the initial PR better; encoding duplicates into the high bits of the grouping ID is a nice approach. We'd just need to take care to mask out the high bits in the `GROUPING` function, and if possible it would be nice to avoid arbitrary limits like not supporting duplicate grouping sets for queries with 8/16/etc. grouping sets.
>
> Alternatively we could represent grouping IDs as an index into the list of GROUPING SETS. That would provide an ID without concern for duplicates, and then we'd implement the `GROUPING` function with a `CASE` or similar construct.
@neilconway Thanks for your kind guidance. I have refactored this PR according to your comments. Due to the significant changes, I force-pushed the commit. I think this version is better than the previous one. If you have time, please take a look again. Sorry for making multiple changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: preserve duplicate GROUPING SETS rows [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #21058:
URL: https://github.com/apache/datafusion/pull/21058#issuecomment-4112485780
Thank you @xiedeyantu
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: preserve duplicate GROUPING SETS rows [datafusion]
Posted by "neilconway (via GitHub)" <gi...@apache.org>.
neilconway commented on code in PR #21058:
URL: https://github.com/apache/datafusion/pull/21058#discussion_r3036984058
##########
datafusion/physical-plan/src/aggregates/mod.rs:
##########
@@ -397,6 +397,14 @@ impl PhysicalGroupBy {
self.expr.len() + usize::from(self.has_grouping_set)
}
+ /// Returns the Arrow data type of the `__grouping_id` column.
+ ///
+ /// The type is chosen to be wide enough to hold both the semantic bitmask
+ /// (in the low `n` bits) and the duplicate ordinal (in the high bits).
Review Comment:
Not clear what `n` is here based on the context of the function itself.
##########
datafusion/expr/src/logical_plan/plan.rs:
##########
@@ -3737,6 +3686,24 @@ impl PartialOrd for Aggregate {
}
}
+/// Returns the highest duplicate ordinal across all grouping sets in `group_expr`.
+///
+/// The ordinal counts how many times a given grouping set pattern has already
+/// appeared before the current occurrence. For example, if the same set
+/// appears three times the ordinals are 0, 1, 2 and this function returns 2.
+/// Returns 0 when no grouping set is duplicated.
+fn max_grouping_set_duplicate_ordinal(group_expr: &[Expr]) -> usize {
+ if let Some(Expr::GroupingSet(GroupingSet::GroupingSets(sets))) = group_expr.first() {
+ let mut counts: HashMap<&Vec<Expr>, usize> = HashMap::new();
Review Comment:
`HashMap<&[Expr], usize>`
##########
datafusion/optimizer/src/analyzer/resolve_grouping_function.rs:
##########
@@ -204,17 +204,22 @@ fn grouping_function_on_id(
Expr::Literal(ScalarValue::from(value as u64), None)
}
};
-
let grouping_id_column = Expr::Column(Column::from(Aggregate::INTERNAL_GROUPING_ID));
- // The grouping call is exactly our internal grouping id
+ // The grouping call is exactly our internal grouping id — mask the ordinal
+ // bits (above position `n`) so only the semantic bitmask is visible.
+ let n = group_by_expr_count;
+ // (1 << n) - 1 masks the low n bits.
+ let semantic_mask: u64 = if n >= 64 { u64::MAX } else { (1u64 << n) - 1 };
Review Comment:
As I suggested before, move this inside the `if`
##########
datafusion/sqllogictest/test_files/group_by.slt:
##########
@@ -5206,6 +5203,41 @@ NULL NULL 1
statement ok
drop table t;
+# regression: duplicate grouping sets must not be collapsed into one
Review Comment:
I think it's worth having a test case for the situation where adding the duplicate ordinal widens the size of the grouping ID field, that's a bit tricky.
##########
datafusion/expr/src/logical_plan/plan.rs:
##########
@@ -3737,6 +3686,24 @@ impl PartialOrd for Aggregate {
}
}
+/// Returns the highest duplicate ordinal across all grouping sets in `group_expr`.
+///
+/// The ordinal counts how many times a given grouping set pattern has already
+/// appeared before the current occurrence. For example, if the same set
Review Comment:
I mentioned this before but "the current occurrence" is not well-defined in this context. Something like this instead:
```
/// The ordinal for each occurrence of a grouping set pattern is its 0-based index among
/// identical entries. For example, if the same set appears three times, the ordinals are
/// 0, 1, 2 and this function returns 2.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: preserve duplicate GROUPING SETS rows [datafusion]
Posted by "xiedeyantu (via GitHub)" <gi...@apache.org>.
xiedeyantu commented on PR #21058:
URL: https://github.com/apache/datafusion/pull/21058#issuecomment-4186156266
@neilconway Thank you very much for your detailed review! I have addressed your comments line by line and included the changes in a single commit. Please take another look to see if everything now aligns with your feedback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: preserve duplicate GROUPING SETS rows [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #21058:
URL: https://github.com/apache/datafusion/pull/21058#issuecomment-4209068308
Thank you @xiedeyantu and @neilconway
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: preserve duplicate GROUPING SETS rows [datafusion]
Posted by "xiedeyantu (via GitHub)" <gi...@apache.org>.
xiedeyantu commented on PR #21058:
URL: https://github.com/apache/datafusion/pull/21058#issuecomment-4148379291
> Thanks @xiedeyantu -- this is great. As before, can you please:
>
> 1. File a ticket showing the bug and the postgres behavior as well? This will make it much easier / faster to review these PRs
@alamb Apologies for the late reply. I have just updated the problem description in the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: preserve duplicate GROUPING SETS rows [datafusion]
Posted by "xiedeyantu (via GitHub)" <gi...@apache.org>.
xiedeyantu commented on PR #21058:
URL: https://github.com/apache/datafusion/pull/21058#issuecomment-4202839556
@neilconway @alamb Thank you for the review! I've made revisions based on the comments. Please help me take another look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: preserve duplicate GROUPING SETS rows [datafusion]
Posted by "neilconway (via GitHub)" <gi...@apache.org>.
neilconway commented on code in PR #21058:
URL: https://github.com/apache/datafusion/pull/21058#discussion_r3047128518
##########
datafusion/optimizer/src/analyzer/resolve_grouping_function.rs:
##########
@@ -207,17 +207,21 @@ fn grouping_function_on_id(
Expr::Literal(ScalarValue::from(value as u64), None)
}
};
-
let grouping_id_column = Expr::Column(Column::from(Aggregate::INTERNAL_GROUPING_ID));
- // The grouping call is exactly our internal grouping id
if args.len() == group_by_expr_count
&& args
.iter()
.rev()
.enumerate()
.all(|(idx, expr)| group_by_expr.get(expr) == Some(&idx))
{
- return Ok(cast(grouping_id_column, DataType::Int32));
+ let n = group_by_expr_count;
+ // Mask the ordinal bits above position `n` so only the semantic bitmask is visible.
+ // (1 << n) - 1 masks the low n bits.
+ let semantic_mask: u64 = if n >= 64 { u64::MAX } else { (1u64 << n) - 1 };
Review Comment:
`n >= 64` is rejected earlier, right? I wonder if it would be cleaner to just assert `n < 64` and then remove the conditional.
##########
datafusion/optimizer/src/analyzer/resolve_grouping_function.rs:
##########
@@ -207,17 +207,21 @@ fn grouping_function_on_id(
Expr::Literal(ScalarValue::from(value as u64), None)
}
};
Review Comment:
It's a little unfortunate that we'll undercount the required bit width here if there are duplicates in `group_by_expr`. Would it be easy to calculate the true width required, including dups?
##########
datafusion/physical-plan/src/aggregates/mod.rs:
##########
@@ -1972,6 +2026,8 @@ pub fn evaluate_group_by(
group_by: &PhysicalGroupBy,
batch: &RecordBatch,
) -> Result<Vec<Vec<ArrayRef>>> {
+ let max_ordinal = max_duplicate_ordinal(&group_by.groups);
+ let mut ordinal_per_pattern: HashMap<&Vec<bool>, usize> = HashMap::new();
Review Comment:
`HashMap<&[bool], usize>`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: preserve duplicate GROUPING SETS rows [datafusion]
Posted by "xiedeyantu (via GitHub)" <gi...@apache.org>.
xiedeyantu commented on PR #21058:
URL: https://github.com/apache/datafusion/pull/21058#issuecomment-4098281204
Hi @alamb , there's another one here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: preserve duplicate GROUPING SETS rows [datafusion]
Posted by "xiedeyantu (via GitHub)" <gi...@apache.org>.
xiedeyantu commented on PR #21058:
URL: https://github.com/apache/datafusion/pull/21058#issuecomment-4158684942
@alamb Could you please take a look and let me know if my current revisions are okay?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: preserve duplicate GROUPING SETS rows [datafusion]
Posted by "neilconway (via GitHub)" <gi...@apache.org>.
neilconway commented on code in PR #21058:
URL: https://github.com/apache/datafusion/pull/21058#discussion_r3033661965
##########
datafusion/physical-plan/src/aggregates/mod.rs:
##########
@@ -1942,25 +1945,74 @@ fn evaluate_optional(
.collect()
}
-fn group_id_array(group: &[bool], batch: &RecordBatch) -> Result<ArrayRef> {
- if group.len() > 64 {
+/// Builds the internal `__grouping_id` array for a single grouping set.
+///
+/// The returned array packs two values into a single integer:
+///
+/// - Low `n` bits (positions 0 .. n-1): the semantic bitmask. A `1` bit
+/// at position `i` means that the `i`-th grouping column (counting from the
+/// least significant bit, i.e. the *last* column in the `group` slice) is
+/// `NULL` for this grouping set.
+/// - High bits (positions n and above): the duplicate `ordinal`, which
+/// distinguishes multiple occurrences of the same grouping-set pattern. The
+/// ordinal is `0` for the first occurrence, `1` for the second, and so on.
+///
+/// The integer type is chosen to be the smallest `UInt8 / UInt16 / UInt32 /
+/// UInt64` that can represent both parts. It matches the type returned by
+/// [`Aggregate::grouping_id_type`].
+fn group_id_array(
+ group: &[bool],
+ ordinal: usize,
+ max_ordinal: usize,
+ batch: &RecordBatch,
+) -> Result<ArrayRef> {
+ let n = group.len();
+ if n > 64 {
return not_impl_err!(
"Grouping sets with more than 64 columns are not supported"
);
}
- let group_id = group.iter().fold(0u64, |acc, &is_null| {
+ let ordinal_bits = usize::BITS as usize - max_ordinal.leading_zeros() as usize;
+ let total_bits = n + ordinal_bits;
+ if total_bits > 64 {
+ return not_impl_err!(
+ "Grouping sets with {n} columns and a maximum duplicate ordinal of \
+ {max_ordinal} require {total_bits} bits, which exceeds 64"
+ );
+ }
+ let semantic_id = group.iter().fold(0u64, |acc, &is_null| {
(acc << 1) | if is_null { 1 } else { 0 }
});
+ let full_id = semantic_id | ((ordinal as u64) << n);
let num_rows = batch.num_rows();
- if group.len() <= 8 {
- Ok(Arc::new(UInt8Array::from(vec![group_id as u8; num_rows])))
- } else if group.len() <= 16 {
- Ok(Arc::new(UInt16Array::from(vec![group_id as u16; num_rows])))
- } else if group.len() <= 32 {
- Ok(Arc::new(UInt32Array::from(vec![group_id as u32; num_rows])))
+ if total_bits <= 8 {
+ Ok(Arc::new(UInt8Array::from(vec![full_id as u8; num_rows])))
+ } else if total_bits <= 16 {
+ Ok(Arc::new(UInt16Array::from(vec![full_id as u16; num_rows])))
+ } else if total_bits <= 32 {
+ Ok(Arc::new(UInt32Array::from(vec![full_id as u32; num_rows])))
} else {
- Ok(Arc::new(UInt64Array::from(vec![group_id; num_rows])))
+ Ok(Arc::new(UInt64Array::from(vec![full_id; num_rows])))
+ }
+}
+
+/// Returns the highest duplicate ordinal across all grouping sets.
+///
+/// The ordinal counts how many times a grouping-set pattern has already
+/// appeared before the current occurrence. If the same `Vec<bool>` appears
+/// three times the ordinals are 0, 1, 2 and this function returns 2.
+/// Returns 0 when no grouping set is duplicated.
+fn max_duplicate_ordinal(groups: &[Vec<bool>]) -> usize {
+ let mut counts: HashMap<&Vec<bool>, usize> = HashMap::new();
+ for group in groups {
+ *counts.entry(group).or_insert(0) += 1;
}
+ counts
+ .values()
+ .copied()
+ .max()
+ .unwrap_or(1)
+ .saturating_sub(1)
Review Comment:
`counts.into_values().max().unwrap_or(0).saturating_sub(1)`
##########
datafusion/physical-plan/src/aggregates/mod.rs:
##########
@@ -1942,25 +1945,74 @@ fn evaluate_optional(
.collect()
}
-fn group_id_array(group: &[bool], batch: &RecordBatch) -> Result<ArrayRef> {
- if group.len() > 64 {
+/// Builds the internal `__grouping_id` array for a single grouping set.
+///
+/// The returned array packs two values into a single integer:
+///
+/// - Low `n` bits (positions 0 .. n-1): the semantic bitmask. A `1` bit
+/// at position `i` means that the `i`-th grouping column (counting from the
+/// least significant bit, i.e. the *last* column in the `group` slice) is
+/// `NULL` for this grouping set.
+/// - High bits (positions n and above): the duplicate `ordinal`, which
+/// distinguishes multiple occurrences of the same grouping-set pattern. The
+/// ordinal is `0` for the first occurrence, `1` for the second, and so on.
+///
+/// The integer type is chosen to be the smallest `UInt8 / UInt16 / UInt32 /
+/// UInt64` that can represent both parts. It matches the type returned by
+/// [`Aggregate::grouping_id_type`].
+fn group_id_array(
+ group: &[bool],
+ ordinal: usize,
+ max_ordinal: usize,
+ batch: &RecordBatch,
+) -> Result<ArrayRef> {
+ let n = group.len();
+ if n > 64 {
return not_impl_err!(
"Grouping sets with more than 64 columns are not supported"
);
}
- let group_id = group.iter().fold(0u64, |acc, &is_null| {
+ let ordinal_bits = usize::BITS as usize - max_ordinal.leading_zeros() as usize;
+ let total_bits = n + ordinal_bits;
+ if total_bits > 64 {
+ return not_impl_err!(
+ "Grouping sets with {n} columns and a maximum duplicate ordinal of \
+ {max_ordinal} require {total_bits} bits, which exceeds 64"
+ );
+ }
+ let semantic_id = group.iter().fold(0u64, |acc, &is_null| {
(acc << 1) | if is_null { 1 } else { 0 }
});
+ let full_id = semantic_id | ((ordinal as u64) << n);
let num_rows = batch.num_rows();
- if group.len() <= 8 {
- Ok(Arc::new(UInt8Array::from(vec![group_id as u8; num_rows])))
- } else if group.len() <= 16 {
- Ok(Arc::new(UInt16Array::from(vec![group_id as u16; num_rows])))
- } else if group.len() <= 32 {
- Ok(Arc::new(UInt32Array::from(vec![group_id as u32; num_rows])))
+ if total_bits <= 8 {
+ Ok(Arc::new(UInt8Array::from(vec![full_id as u8; num_rows])))
+ } else if total_bits <= 16 {
+ Ok(Arc::new(UInt16Array::from(vec![full_id as u16; num_rows])))
+ } else if total_bits <= 32 {
+ Ok(Arc::new(UInt32Array::from(vec![full_id as u32; num_rows])))
} else {
- Ok(Arc::new(UInt64Array::from(vec![group_id; num_rows])))
+ Ok(Arc::new(UInt64Array::from(vec![full_id; num_rows])))
+ }
+}
+
+/// Returns the highest duplicate ordinal across all grouping sets.
+///
+/// The ordinal counts how many times a grouping-set pattern has already
+/// appeared before the current occurrence. If the same `Vec<bool>` appears
Review Comment:
"the current occurrence" doesn't seem to make sense when you look at the function itself, it is talking about the call-site.
##########
datafusion/physical-plan/src/aggregates/mod.rs:
##########
@@ -1942,25 +1945,74 @@ fn evaluate_optional(
.collect()
}
-fn group_id_array(group: &[bool], batch: &RecordBatch) -> Result<ArrayRef> {
- if group.len() > 64 {
+/// Builds the internal `__grouping_id` array for a single grouping set.
+///
+/// The returned array packs two values into a single integer:
+///
+/// - Low `n` bits (positions 0 .. n-1): the semantic bitmask. A `1` bit
+/// at position `i` means that the `i`-th grouping column (counting from the
+/// least significant bit, i.e. the *last* column in the `group` slice) is
+/// `NULL` for this grouping set.
+/// - High bits (positions n and above): the duplicate `ordinal`, which
+/// distinguishes multiple occurrences of the same grouping-set pattern. The
+/// ordinal is `0` for the first occurrence, `1` for the second, and so on.
+///
+/// The integer type is chosen to be the smallest `UInt8 / UInt16 / UInt32 /
+/// UInt64` that can represent both parts. It matches the type returned by
+/// [`Aggregate::grouping_id_type`].
+fn group_id_array(
+ group: &[bool],
+ ordinal: usize,
+ max_ordinal: usize,
+ batch: &RecordBatch,
+) -> Result<ArrayRef> {
+ let n = group.len();
+ if n > 64 {
return not_impl_err!(
"Grouping sets with more than 64 columns are not supported"
);
}
- let group_id = group.iter().fold(0u64, |acc, &is_null| {
+ let ordinal_bits = usize::BITS as usize - max_ordinal.leading_zeros() as usize;
+ let total_bits = n + ordinal_bits;
+ if total_bits > 64 {
+ return not_impl_err!(
+ "Grouping sets with {n} columns and a maximum duplicate ordinal of \
+ {max_ordinal} require {total_bits} bits, which exceeds 64"
+ );
+ }
+ let semantic_id = group.iter().fold(0u64, |acc, &is_null| {
(acc << 1) | if is_null { 1 } else { 0 }
});
+ let full_id = semantic_id | ((ordinal as u64) << n);
let num_rows = batch.num_rows();
- if group.len() <= 8 {
- Ok(Arc::new(UInt8Array::from(vec![group_id as u8; num_rows])))
- } else if group.len() <= 16 {
- Ok(Arc::new(UInt16Array::from(vec![group_id as u16; num_rows])))
- } else if group.len() <= 32 {
- Ok(Arc::new(UInt32Array::from(vec![group_id as u32; num_rows])))
+ if total_bits <= 8 {
+ Ok(Arc::new(UInt8Array::from(vec![full_id as u8; num_rows])))
+ } else if total_bits <= 16 {
+ Ok(Arc::new(UInt16Array::from(vec![full_id as u16; num_rows])))
+ } else if total_bits <= 32 {
+ Ok(Arc::new(UInt32Array::from(vec![full_id as u32; num_rows])))
} else {
- Ok(Arc::new(UInt64Array::from(vec![group_id; num_rows])))
+ Ok(Arc::new(UInt64Array::from(vec![full_id; num_rows])))
+ }
+}
+
+/// Returns the highest duplicate ordinal across all grouping sets.
+///
+/// The ordinal counts how many times a grouping-set pattern has already
+/// appeared before the current occurrence. If the same `Vec<bool>` appears
+/// three times the ordinals are 0, 1, 2 and this function returns 2.
+/// Returns 0 when no grouping set is duplicated.
+fn max_duplicate_ordinal(groups: &[Vec<bool>]) -> usize {
+ let mut counts: HashMap<&Vec<bool>, usize> = HashMap::new();
Review Comment:
`HashMap<&[bool], usize>`
##########
datafusion/optimizer/src/analyzer/resolve_grouping_function.rs:
##########
@@ -204,17 +204,26 @@ fn grouping_function_on_id(
Expr::Literal(ScalarValue::from(value as u64), None)
}
};
-
let grouping_id_column = Expr::Column(Column::from(Aggregate::INTERNAL_GROUPING_ID));
- // The grouping call is exactly our internal grouping id
+ // The grouping call is exactly our internal grouping id — mask the ordinal
+ // bits (above position `n`) so only the semantic bitmask is visible.
+ let n = group_by_expr_count;
+ // (1 << n) - 1 masks the low n bits. Use saturating arithmetic to handle n == 0.
Review Comment:
Although I don't think `wrapping_sub` is necessary anyway, `n == 0` -> `1 << 0` -> `1`, so it won't underflow.
##########
datafusion/optimizer/src/analyzer/resolve_grouping_function.rs:
##########
@@ -204,17 +204,26 @@ fn grouping_function_on_id(
Expr::Literal(ScalarValue::from(value as u64), None)
}
};
-
let grouping_id_column = Expr::Column(Column::from(Aggregate::INTERNAL_GROUPING_ID));
- // The grouping call is exactly our internal grouping id
+ // The grouping call is exactly our internal grouping id — mask the ordinal
+ // bits (above position `n`) so only the semantic bitmask is visible.
+ let n = group_by_expr_count;
+ // (1 << n) - 1 masks the low n bits. Use saturating arithmetic to handle n == 0.
+ let semantic_mask: u64 = if n >= 64 {
+ u64::MAX
+ } else {
+ (1u64 << n).wrapping_sub(1)
+ };
+ let masked_id =
Review Comment:
Don't compute this outside the `if` in which it is used.
##########
datafusion/optimizer/src/analyzer/resolve_grouping_function.rs:
##########
@@ -204,17 +204,26 @@ fn grouping_function_on_id(
Expr::Literal(ScalarValue::from(value as u64), None)
}
};
-
let grouping_id_column = Expr::Column(Column::from(Aggregate::INTERNAL_GROUPING_ID));
- // The grouping call is exactly our internal grouping id
+ // The grouping call is exactly our internal grouping id — mask the ordinal
+ // bits (above position `n`) so only the semantic bitmask is visible.
+ let n = group_by_expr_count;
+ // (1 << n) - 1 masks the low n bits. Use saturating arithmetic to handle n == 0.
Review Comment:
The code doesn't use saturating arithmetic?
##########
datafusion/sql/src/unparser/utils.rs:
##########
@@ -247,6 +247,11 @@ fn find_agg_expr<'a>(agg: &'a Aggregate, column: &Column) -> Result<Option<&'a E
)
}
Ordering::Greater => {
+ if index < grouping_expr.len() + 1 {
Review Comment:
This seems like dead code? i.e., index > grouping_expr.len -> index >= (grouping_expr.len + 1)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: preserve duplicate GROUPING SETS rows [datafusion]
Posted by "neilconway (via GitHub)" <gi...@apache.org>.
neilconway commented on code in PR #21058:
URL: https://github.com/apache/datafusion/pull/21058#discussion_r3048457709
##########
datafusion/optimizer/src/analyzer/resolve_grouping_function.rs:
##########
@@ -184,40 +191,43 @@ fn validate_args(
fn grouping_function_on_id(
function: &AggregateFunction,
group_by_expr: &HashMap<&Expr, usize>,
- is_grouping_set: bool,
+ // None means not a grouping set (result is always 0).
+ grouping_id_type: Option<DataType>,
) -> Result<Expr> {
validate_args(function, group_by_expr)?;
let args = &function.params.args;
// Postgres allows grouping function for group by without grouping sets, the result is then
// always 0
- if !is_grouping_set {
+ let Some(grouping_id_type) = grouping_id_type else {
return Ok(Expr::Literal(ScalarValue::from(0i32), None));
- }
-
- let group_by_expr_count = group_by_expr.len();
- let literal = |value: usize| {
- if group_by_expr_count < 8 {
- Expr::Literal(ScalarValue::from(value as u8), None)
- } else if group_by_expr_count < 16 {
- Expr::Literal(ScalarValue::from(value as u16), None)
- } else if group_by_expr_count < 32 {
- Expr::Literal(ScalarValue::from(value as u32), None)
- } else {
- Expr::Literal(ScalarValue::from(value as u64), None)
- }
};
+ // Use the actual __grouping_id column type to size literals correctly. This
+ // accounts for duplicate-ordinal bits that `Aggregate::grouping_id_type`
+ // packs into the high bits of the column, which a simple count of grouping
+ // expressions would miss.
+ let literal = |value: usize| match &grouping_id_type {
+ DataType::UInt8 => Expr::Literal(ScalarValue::from(value as u8), None),
+ DataType::UInt16 => Expr::Literal(ScalarValue::from(value as u16), None),
+ DataType::UInt32 => Expr::Literal(ScalarValue::from(value as u32), None),
+ _ => Expr::Literal(ScalarValue::from(value as u64), None),
Review Comment:
Seems safer to have an explicit match arm for `DataType::UInt64` and then a fallthrough that raises an error message.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: preserve duplicate GROUPING SETS rows [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #21058:
URL: https://github.com/apache/datafusion/pull/21058#discussion_r3047702575
##########
datafusion/physical-plan/src/aggregates/mod.rs:
##########
@@ -1937,15 +1937,53 @@ fn evaluate_optional(
.collect()
}
-fn group_id_array(group: &[bool], batch: &RecordBatch) -> Result<ArrayRef> {
+fn group_id_array(
Review Comment:
the new comments help a lot
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: preserve duplicate GROUPING SETS rows [datafusion]
Posted by "xiedeyantu (via GitHub)" <gi...@apache.org>.
xiedeyantu commented on code in PR #21058:
URL: https://github.com/apache/datafusion/pull/21058#discussion_r3051594558
##########
datafusion/optimizer/src/analyzer/resolve_grouping_function.rs:
##########
@@ -184,40 +191,43 @@ fn validate_args(
fn grouping_function_on_id(
function: &AggregateFunction,
group_by_expr: &HashMap<&Expr, usize>,
- is_grouping_set: bool,
+ // None means not a grouping set (result is always 0).
+ grouping_id_type: Option<DataType>,
) -> Result<Expr> {
validate_args(function, group_by_expr)?;
let args = &function.params.args;
// Postgres allows grouping function for group by without grouping sets, the result is then
// always 0
- if !is_grouping_set {
+ let Some(grouping_id_type) = grouping_id_type else {
return Ok(Expr::Literal(ScalarValue::from(0i32), None));
- }
-
- let group_by_expr_count = group_by_expr.len();
- let literal = |value: usize| {
- if group_by_expr_count < 8 {
- Expr::Literal(ScalarValue::from(value as u8), None)
- } else if group_by_expr_count < 16 {
- Expr::Literal(ScalarValue::from(value as u16), None)
- } else if group_by_expr_count < 32 {
- Expr::Literal(ScalarValue::from(value as u32), None)
- } else {
- Expr::Literal(ScalarValue::from(value as u64), None)
- }
};
+ // Use the actual __grouping_id column type to size literals correctly. This
+ // accounts for duplicate-ordinal bits that `Aggregate::grouping_id_type`
+ // packs into the high bits of the column, which a simple count of grouping
+ // expressions would miss.
+ let literal = |value: usize| match &grouping_id_type {
+ DataType::UInt8 => Expr::Literal(ScalarValue::from(value as u8), None),
+ DataType::UInt16 => Expr::Literal(ScalarValue::from(value as u16), None),
+ DataType::UInt32 => Expr::Literal(ScalarValue::from(value as u32), None),
+ _ => Expr::Literal(ScalarValue::from(value as u64), None),
Review Comment:
done!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org