You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "brancz (via GitHub)" <gi...@apache.org> on 2024/09/20 08:50:35 UTC
[I] Panic on aggregations on struct of dictionaries [datafusion]
brancz opened a new issue, #12542:
URL: https://github.com/apache/datafusion/issues/12542
### Describe the bug
Aggregations on a struct of dictionaries produces data with a different schema than expected (plain arrays instead of dictionaries).
For example:
```
thread 'tokio-runtime-worker' panicked at /Users/brancz/.cargo/registry/src/index.crates.io-6f17d22bba15001f/arrow-array-53.0.0/src/array/struct_array.rs:90:46:
called `Result::unwrap()` on an `Err` value: InvalidArgumentError("Incorrect datatype for StructArray field \"a\", expected Dictionary(Int32, Utf8) got Utf8")
```
### To Reproduce
Have a schema with a struct of dictionaries, and perform an aggregation on it, like `count_distinct`.
Full code example here:
https://gist.github.com/brancz/fa12a3ae0f5d09620e9c274384ffd506
### Expected behavior
No panic.
### Additional context
I can see two ways to solve this:
1) Currently, the aggregation says it will emit data with the dictionaries being dictionaries. Instead, if all it did was declare it would emit plain arrays instead of dictionary-encoded ones, it would not panic.
2) Have `RowConverter` emit the same `DataType` as its input.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Panic on aggregations on struct of dictionaries [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #12542:
URL: https://github.com/apache/datafusion/issues/12542#issuecomment-2363877930
> I think the right fix in that case is 1, adding support for nested schemas.
I agree this sounds like it makes sense. There even seems to be an existing ticket: #7647
Note that there is a PR by @jayzhan211 to rework how grouping is done to avoid the RowConverter in many cases in https://github.com/apache/datafusion/pull/12269. I haven't reivewed it thoroughly, but I would suggest that you ensure your fix for this issue is well covered by end to end `.slt` tests (not just unit tests)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Panic on aggregations on struct of dictionaries [datafusion]
Posted by "brancz (via GitHub)" <gi...@apache.org>.
brancz commented on issue #12542:
URL: https://github.com/apache/datafusion/issues/12542#issuecomment-2363812088
Ok found it. [This is the place](https://github.com/apache/datafusion/blob/8397855f24ce548fe53ed0d7a8bb4d9cbbeb866b/datafusion/physical-plan/src/aggregates/group_values/row.rs#L221-L233) where if the data is of an unexpected type, it's casted to the expected type.
I think the right fix in that case is 1, adding support for nested schemas.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Panic on aggregations on struct of dictionaries [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #12542: Panic on aggregations on struct of dictionaries
URL: https://github.com/apache/datafusion/issues/12542
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Panic on aggregations on struct of dictionaries [datafusion]
Posted by "brancz (via GitHub)" <gi...@apache.org>.
brancz commented on issue #12542:
URL: https://github.com/apache/datafusion/issues/12542#issuecomment-2363763884
I actually tried to understand what is different about dicts that are not in structs, and it turns out that the row converter also emits plain arrays in those cases, but something turns them back into dictionaries at some point (I'm guessing this has to be in datafusion somewhere).
Example: https://gist.github.com/brancz/9ff04f6263b710ad8215933590026500
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org