You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "rluvaton (via GitHub)" <gi...@apache.org> on 2024/12/12 10:30:12 UTC
[I] Allow to filter null in `array_agg` [datafusion]
rluvaton opened a new issue, #13742:
URL: https://github.com/apache/datafusion/issues/13742
### Is your feature request related to a problem or challenge?
Yes, I want nulls to be filtered from `array_agg` when I specify `with_ignore_nulls: true` in `AggregateExprBuilder` to have behavior similar to Spark, this should also work when `with_distinct` is `false` or `true`
### Describe the solution you'd like
Specifing `with_ignore_nulls: true` should ignore
### Describe alternatives you've considered
Adding a filter for removing nulls to the child before creating the `array_agg`
### Additional context
This PR might be relevant:
- #11447
I'm willing to open a pull request
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Allow to filter null in `array_agg` [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #13742:
URL: https://github.com/apache/datafusion/issues/13742#issuecomment-2549409310
> Thank you @alamb. do you have an example of an agg that doesn't ignore them? I tried find one, but maybe i was searching the wrong way
I think `first_value` supports nulls / ignore nulls
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Allow to filter null in `array_agg` [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on issue #13742:
URL: https://github.com/apache/datafusion/issues/13742#issuecomment-2541269280
I like the idea that we leverage existing filter in AggregateExec for this usecase
https://github.com/apache/datafusion/blob/c030bfe1e197fdbb385511a2e4fedf753f284d6f/datafusion/physical-plan/src/aggregates/mod.rs#L355-L356
I guess you can create `AggregateExec` with predicate that only non-nulls goes to array-agg, similar to the following syntax
```
select array_agg(column1) from table where column1 != NUll;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Allow to filter null in `array_agg` [datafusion]
Posted by "findepi (via GitHub)" <gi...@apache.org>.
findepi commented on issue #13742:
URL: https://github.com/apache/datafusion/issues/13742#issuecomment-2548152661
Thank you @alamb. do you have an example of an agg that doesn't ignore them?
I tried find one, but maybe i was searching the wrong way
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Allow to filter null in `array_agg` [datafusion]
Posted by "thinkharderdev (via GitHub)" <gi...@apache.org>.
thinkharderdev closed issue #13742: Allow to filter null in `array_agg`
URL: https://github.com/apache/datafusion/issues/13742
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Allow to filter null in `array_agg` [datafusion]
Posted by "findepi (via GitHub)" <gi...@apache.org>.
findepi commented on issue #13742:
URL: https://github.com/apache/datafusion/issues/13742#issuecomment-2538750719
AFAICT, the `with_ignore_nulls: true` is currently... ignored?
Given it's part of the "logical plan", it should be obeyed. It's probably part of LP because the LP is half-way between parser's AST and real plan (see also https://github.com/apache/datafusion/issues/12604 and cc @alamb).
I'm OK for respecting `with_ignore_nulls` in the plan, but I would actually consider removing it completely. Simpler things are better. The same functionality is available using `AggregateFunction.filter`.
What calls for keeping `with_ignore_nulls` is that aggregate functions -- in general -- skip over null inputs. SQL standard `array_agg` is one of few exceptions (and some systems like Snowflake and apparently Spark skip nulls for array_agg). Ideally this could be decided by the function itself, so could be a parameter of the `ArrayAgg` struct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Allow to filter null in `array_agg` [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #13742:
URL: https://github.com/apache/datafusion/issues/13742#issuecomment-2545961628
To be clear, I think the SQL parser supports predicates like `ARRAY_AGG(x) IGNORE NULL` and we have logical plan support for them.
However, some aggregates (silently) ignore this, see
- https://github.com/apache/datafusion/issues/9924
I think filling out support (or at least making it clear that this syntax is ignored) would be a great idea
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org