You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "Weijun-H (via GitHub)" <gi...@apache.org> on 2024/08/30 02:42:42 UTC
[I] [Epic] Extend the Aggregation function [datafusion]
Weijun-H opened a new issue, #12254:
URL: https://github.com/apache/datafusion/issues/12254
### Is your feature request related to a problem or challenge?
_No response_
### Describe the solution you'd like
- [ ] #12252
- [ ] #12253
- [ ] #12251
- [ ] #12250
- [ ] #12249
- [ ] #12248
- [ ] #12247
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [Epic] A collection of issues for extending the Aggregation function [datafusion]
Posted by "austin362667 (via GitHub)" <gi...@apache.org>.
austin362667 commented on issue #12254:
URL: https://github.com/apache/datafusion/issues/12254#issuecomment-2353657029
Thank you @alamb for proposing this initiative. I like this idea. What about others' thought?
It clearly draws a line between the `core` and the `extensions`. And we can still leverage those functions as extension in `dft`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [Epic] A collection of issues for extending the Aggregation function [datafusion]
Posted by "Weijun-H (via GitHub)" <gi...@apache.org>.
Weijun-H commented on issue #12254:
URL: https://github.com/apache/datafusion/issues/12254#issuecomment-2358321684
> > @alamb I’m generally in favor of moving additional features from the core to separate subprojects. However, could we consider using a more general name than datafusion-functions-duckdb? What are your thoughts on using something like datafusion-functions-sql or another broader name?
>
> I do not have a strong preference -- I think it likely depends on the usecase:
>
> * trying to migrate spark workloads
> * trying to migrate duckdb workloads
> * triynt to make the most useful sql system you can
>
> However, let's not let get too carried away with details at the moment.
>
> I created [datafusion-contrib/datafusion-functions-extra](https://github.com/datafusion-contrib/datafusion-functions-extra) and added @dmitrybugakov and @austin362667 as admins. If anyone else wants to help let me know and we can add you too.
>
> @dmitrybugakov or @austin362667 would you be willing to setup the basic skeleton of the repo?
>
> Perhaps you could follow the model of [datafusion-contrib/datafusion-functions-json](https://github.com/datafusion-contrib/datafusion-functions-json) for readme and registration function
>
> And then try to put the `mode` function there: #12385
I agree that 'datafusion-functions-extra' is sufficient for the current scenario because our goal is to decouple the extra functions from the core and make them lightweight. Additionally, it would be easier to split 'datafusion-functions-extra' into many specific use cases if necessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [Epic] A collection of issues for extending the Aggregation function [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #12254:
URL: https://github.com/apache/datafusion/issues/12254#issuecomment-2374446425
I believe the consensus is that we are going to implement these functions in https://github.com/datafusion-contrib/datafusion-functions-extra -- I will close the related tickets so we are not confusing people about what we are looking for
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [Epic] A collection of issues for extending the Aggregation function [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #12254: [Epic] A collection of issues for extending the Aggregation function
URL: https://github.com/apache/datafusion/issues/12254
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [Epic] A collection of issues for extending the Aggregation function [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #12254:
URL: https://github.com/apache/datafusion/issues/12254#issuecomment-2353615046
@Weijun-H and @dmitrybugakov and @dharanad -- what do you think about creating a `datafusion-functions-duckdb` repo in datafusion-contrib similar to https://github.com/datafusion-contrib/datafusion-functions-json for JSON from @samuelcolvin and co.
It would be a pretty neat way to help build out the function library in DataFUsion and would show off its extensibility
I could then try an integrate it into `dft` that @matthewmturner and I have been working on: https://github.com/datafusion-contrib/datafusion-dft which would make it easer to use
Originally from: https://github.com/apache/datafusion/pull/12476#issuecomment-2353611810
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [Epic] A collection of issues for extending the Aggregation function [datafusion]
Posted by "dmitrybugakov (via GitHub)" <gi...@apache.org>.
dmitrybugakov commented on issue #12254:
URL: https://github.com/apache/datafusion/issues/12254#issuecomment-2355267281
@alamb
I’m generally in favor of moving additional features from the core to separate subprojects. However, could we consider using a more general name than datafusion-functions-duckdb? What are your thoughts on using something like datafusion-functions-sql or another broader name?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [Epic] A collection of issues for extending the Aggregation function [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #12254:
URL: https://github.com/apache/datafusion/issues/12254#issuecomment-2331260267
I wonder if we should consider where to draw the line on what aggregate functions to include in the core (i.e. should we include all these new functions?)
Now that all aggregate functions use the same API, we could potentially keep more specialized functions such as listed here outside the ore -- either in its own crate or even own repo -- and then have other code integrate it in -- e.g. https://github.com/apache/datafusion/issues/11979
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [Epic] A collection of issues for extending the Aggregation function [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #12254:
URL: https://github.com/apache/datafusion/issues/12254#issuecomment-2334101103
I started a discussion about if we should be adding all these functions directly in the core here: https://github.com/apache/datafusion/issues/12357
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [Epic] A collection of issues for extending the Aggregation function [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #12254:
URL: https://github.com/apache/datafusion/issues/12254#issuecomment-2356571864
> @alamb I’m generally in favor of moving additional features from the core to separate subprojects. However, could we consider using a more general name than datafusion-functions-duckdb? What are your thoughts on using something like datafusion-functions-sql or another broader name?
I do not have a strong preference -- I think it likely depends on the usecase:
* trying to migrate spark workloads
* trying to migrate duckdb workloads
* triynt to make the most useful sql system you can
However, let's not let get too carried away with details at the moment.
I created https://github.com/datafusion-contrib/datafusion-functions-extra and added @dmitrybugakov and @austin362667 as admins. If anyone else wants to help let me know and we can add you too.
@dmitrybugakov or @austin362667 would you be willing to setup the basic skeleton of the repo?
Perhaps you could follow the model of https://github.com/datafusion-contrib/datafusion-functions-json for readme and registration function
And then try to put the `mode` function there: https://github.com/apache/datafusion/pull/12385
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [Epic] A collection of issues for extending the Aggregation function [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #12254:
URL: https://github.com/apache/datafusion/issues/12254#issuecomment-2356576346
Update: I also added @Lordworms (a longtime DataFusion contributor) per https://github.com/apache/datafusion/pull/12284#issuecomment-2354194830
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [Epic] A collection of issues for extending the Aggregation function [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #12254:
URL: https://github.com/apache/datafusion/issues/12254#issuecomment-2374477282
I filed https://github.com/apache/datafusion/issues/12625 to propose moving `kertosis_pop`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [Epic] A collection of issues for extending the Aggregation function [datafusion]
Posted by "Weijun-H (via GitHub)" <gi...@apache.org>.
Weijun-H commented on issue #12254:
URL: https://github.com/apache/datafusion/issues/12254#issuecomment-2334206044
> I wonder if we should consider where to draw the line on what aggregate functions to include in the core (i.e. should we include all these new functions?)
>
> Now that all aggregate functions use the same API, we could potentially keep more specialized functions such as listed here outside the ore -- either in its own crate or even own repo -- and then have other code integrate it in -- e.g. #11979
I like this idea! 🚀
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org