You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2024/10/21 13:15:33 UTC
[I] Oct 21, 2024: This week in DataFusion [datafusion]
alamb opened a new issue, #13035:
URL: https://github.com/apache/datafusion/issues/13035
## Introduction
This ticket is a weekly summary of interesting things happening in DataFusion. Note this is not a complete list (it is what I remember / can find). Please feel free to leave comments on this ticket about things that I may have missed or you think should get wider attention by the community
Loosely inspired by https://this-week-in-rust.org/
## Highlights from last week(s):
(I am sorry if I missed you -- please add a note to this ticket with anything you would like to add)
- @goldmedal [became a committer](https://lists.apache.org/thread/2b66yw4sddq2cqybn8ff7h074p9oy1ro) 🎉
- @peasee, @sgrebnov @goldmedal @phillipleblanc continue to make the plan --> SQL text first class https://github.com/apache/datafusion/pull/12994 https://github.com/apache/datafusion/pull/12987 https://github.com/apache/datafusion/pull/12983 and more
- @duongcongtoai cleaned up `unnest` even more: https://github.com/apache/datafusion/pull/12836
- @akurmustafa made several PRs to keep the code clean like https://github.com/apache/datafusion/pull/13013
- @eejbyfeldt pushed ahead getting all of TPCDS working https://github.com/apache/datafusion/pull/12945 and fixed bugs https://github.com/apache/datafusion/pull/12902
- @tokoko @Blizzara @vbarua and @westonpace keep on improving substrait https://github.com/apache/datafusion/pull/12863
- @Omega359 added new regexp functions https://github.com/apache/datafusion/pull/12970
- @askalt https://github.com/apache/datafusion/pull/12950
- @jcsherin kept the momentum up on converting window functions: https://github.com/apache/datafusion/pull/12857
- @peter-toth began implementing CSE for physical plans; https://github.com/apache/datafusion/issues/12599
## Looking to get more involved? Try code review!
DataFusion has a long history of community members [contributing in all aspects of the project](https://datafusion.apache.org/contributor-guide/index.html). Reviewing PRs is an especially great way to get introduced to the project, help the community and grow your own knowledge -- researching and understanding the code enough to review PRs also often inspires additional ideas for improvements.
We have [docs about reviews](https://datafusion.apache.org/contributor-guide/index.html#reviewing-pull-requests). TLDR is: look for test coverage, if the change is understandable and well documented, and if the code can be improved. When you think the PR looks good to merge, try `@` mentioning [one of the committers](https://projects.apache.org/committee.html?datafusion).
## Help wanted
Please feel leave your own comments on the ticket if you are looking for help
* Reviews (especially for committers) to help @notfillipo and @findepi on[ logical types](https://github.com/apache/datafusion/issues/12622)
* Reviews of new FFI interface https://github.com/apache/datafusion/pull/12920 from @timsaucer
* Committer review / approval of https://github.com/apache/datafusion/pull/12871
## Andrew's Focus Areas:
We are preparing for the [43.0.0 release](https://github.com/apache/datafusion/issues/12470) and I am personally pretty excited about (and thus actively help / put to the top of my review list):
- https://github.com/apache/datafusion/issues/12821 (thanks to the epic work of @Rachelint, @goldmedal, @jayzhan211, @Dandandan @XiangpengHao and others, we are quite close)
- https://github.com/apache/datafusion/issues/8709 (very close to finishing thanks @jcsherin @jatin510)
- https://github.com/apache/datafusion/issues/12740 (Thanks to @jonathanc-n)
- https://github.com/apache/datafusion/issues/12114 (thanks @Rachelint for all your help so far)
- https://github.com/apache/datafusion-sqlparser-rs/issues/1294
## Recent and Upcoming Releases
- [x] https://github.com/apache/datafusion/issues/12813 (thanks @Xuanwo and @matthewmturner)
- [ ] https://github.com/apache/datafusion-sqlparser-rs/issues/1423 (huge kudos to @iffyio for all the reviews)
- [ ] https://github.com/apache/arrow-rs/issues/6341 (to support turning on string view by default)
- [ ] https://github.com/apache/datafusion/issues/12470 (thanks @andygrove)
## Interesting discussions underway:
- https://github.com/apache/datafusion/issues/11442
- https://github.com/apache/datafusion/issues/12357
## Community
* [Weekly Call](https://docs.google.com/document/d/1NBpkIAuU7O9h8Br5CbFksDhX-L9TyO9wmGLPMe0Plc8/edit#heading=h.kpjkpncdmt1g)
* Slack/Discord: [info links](https://datafusion.apache.org/contributor-guide/communication.html#slack-and-discord)
## Upcoming meetups:
* [Oct 14 Seattle](https://github.com/apache/datafusion/discussions/10348): https://lu.ma/tnwl866b @phillipleblanc @likekim
* [Dec 18 Chicago](https://github.com/apache/datafusion/discussions/12894): https://lu.ma/eq5myc5i @adriangb @timsaucer
* TBD: https://github.com/apache/datafusion/discussions/12988
## Background:
Previous update: https://github.com/apache/datafusion/issues/12973
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Oct 21, 2024: This week in DataFusion [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #13035:
URL: https://github.com/apache/datafusion/issues/13035#issuecomment-2435322881
@SamSynnada started a great discussion about better spreading the word about DataFusion. Thank you 🙏 -- it is going to be a great year
- https://github.com/apache/datafusion/discussions/13049
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Oct 21, 2024: This week in DataFusion [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #13035:
URL: https://github.com/apache/datafusion/issues/13035#issuecomment-2439489033
@2010YOUY01 is rallying a team for improvements in externalized hash aggregation:
- https://github.com/apache/datafusion/issues/13123
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Oct 21, 2024: This week in DataFusion [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #13035:
URL: https://github.com/apache/datafusion/issues/13035#issuecomment-2444801114
Next week: https://github.com/apache/datafusion/issues/13167
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Oct 21, 2024: This week in DataFusion [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #13035: Oct 21, 2024: This week in DataFusion
URL: https://github.com/apache/datafusion/issues/13035
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Oct 21, 2024: This week in DataFusion [datafusion]
Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on issue #13035:
URL: https://github.com/apache/datafusion/issues/13035#issuecomment-2426833261
Many thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Oct 21, 2024: This week in DataFusion [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #13035:
URL: https://github.com/apache/datafusion/issues/13035#issuecomment-2429383079
@XiangpengHao and @tustvold are also working on Adaptive Predicate Pushdown in the parquet reader of https://github.com/apache/arrow-rs/issues/5523 -- this has the potential to allow DataFusion to always push predicates into the parquet decoder itself, further improving parquet read performance
See the great writeup from @tustvold here with analysis: https://github.com/apache/arrow-rs/issues/6454#issuecomment-2402118020
cc @samuelcolvin
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Oct 21, 2024: This week in DataFusion [datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on issue #13035:
URL: https://github.com/apache/datafusion/issues/13035#issuecomment-2426702122
Congrats @goldmedal for becoming a committer! Thanks for all your hard work 🚀
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Oct 21, 2024: This week in DataFusion [datafusion]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #13035:
URL: https://github.com/apache/datafusion/issues/13035#issuecomment-2435335718
Hi, I want to highlight https://github.com/datafusion-contrib/datafusion-orc/issues/120 as a great example of community collaboration.
Given DataFusion's unique position, I believe there are many opportunities for such collaboration. It's also a great chance to thank everyone who made it possible, especially those who have maintained the project for a year.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org