You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "adp2201 (via GitHub)" <gi...@apache.org> on 2026/03/17 23:57:35 UTC
[D] Q2 2026 roadmap planning [datafusion]
GitHub user adp2201 created a discussion: Q2 2026 roadmap planning
## Q2 2026 roadmap planning
Hi all,
Q2 is coming up, so it seems like a good time to start a fresh roadmap discussion building on the Q1 thread:
- Q1 roadmap: https://github.com/apache/datafusion/issues/18494
The goals here are simple:
1. Identify what should carry forward from Q1 into Q2
2. Identify what new projects we think should be prioritized in Q2
## Likely carry-forward themes
From the Q1 discussion, these seem like the clearest themes to continue:
- **Performance**
- ClickBench / Parquet leaderboard
- Parquet pushdown / late materialization
- planning speed
- sort / join / spilling improvements
- **Developer experience / coordination**
- CI reliability
- contributor workflow
- better coordination across active efforts
- **Extensibility / SQL surface**
- extension types / UDTs
- `VARIANT`
- lambda syntax / array functionality
- async UDF / UDTF work
- datetime / timezone improvements
## Some concrete Q2 candidates
A few open epics / proposal epics that seem especially relevant:
- **Improve query planning speed**: #19795
- **ClickBench Parquet leaderboard**: #18489
- **Improve NUMA-awareness**: #20902
- **Sort-merge join execution improvements**: #20910
- **Support `VARIANT`**: #16116
- **Support Extension Types / UDTs**: #12644
- **Improve developer / CI speed**: #13813
- **Dynamic filtering**: #15512
- **Statistics improvements**: #8227
## Questions
- Which Q1 themes should clearly carry into Q2?
- Which open epics feel most important right now?
- What important Q2 projects are missing?
- Which efforts need more implementation help, benchmarking, or design work?
If you already plan to work on something in Q2, please comment with the issue / epic and the kind of help you’d welcome from others.
GitHub link: https://github.com/apache/datafusion/discussions/21017
----
This is an automatically sent email for github@datafusion.apache.org.
To unsubscribe, please send an email to: github-unsubscribe@datafusion.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [D] Q2 2026 roadmap planning [datafusion]
Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
GitHub user LiaCastaneda edited a comment on the discussion: Q2 2026 roadmap planning
👋 Hello! We’re very interested in having DataFusion support lambda functions. There has been an ongoing epic for this, and some work has already started toward a basic e2e implementation:
- EPIC - https://github.com/apache/datafusion/issues/14205
- First PR – https://github.com/apache/datafusion/pull/18921
It feels like we’re currently missing consensus on the approach to follow. We’d be happy to help -- whether it's reviewing, contributing, or anything else that would move this forward.
Also, @benbellick from substrait-io has added lambda support in Substrait, so it seems the remaining big piece is the function implementation.
cc @gstvg as the one driving this effort!
GitHub link: https://github.com/apache/datafusion/discussions/21017#discussioncomment-16214853
----
This is an automatically sent email for github@datafusion.apache.org.
To unsubscribe, please send an email to: github-unsubscribe@datafusion.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [D] Q2 2026 roadmap planning [datafusion]
Posted by "akoshchiy (via GitHub)" <gi...@apache.org>.
GitHub user akoshchiy added a comment to the discussion: Q2 2026 roadmap planning
Hello! I would be happy to get involved and contribute to any epic.
Btw I'd been working on window range optimization within #15607, but seems that the discussion is not active.
GitHub link: https://github.com/apache/datafusion/discussions/21017#discussioncomment-16661281
----
This is an automatically sent email for github@datafusion.apache.org.
To unsubscribe, please send an email to: github-unsubscribe@datafusion.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [D] Q2 2026 roadmap planning [datafusion]
Posted by "sdf-jkl (via GitHub)" <gi...@apache.org>.
GitHub user sdf-jkl added a comment to the discussion: Q2 2026 roadmap planning
I plan to work on:
- Pushing `VARIANT` forward
- https://github.com/apache/arrow-rs/issues/8480
- Variant integration in DataFusion https://github.com/datafusion-contrib/datafusion-variant/issues/2 with @friendlymatthew. We have some `good first issues`
- Abovementioned https://github.com/apache/datafusion/issues/14535
- ALP encoding support for Parquet reader https://github.com/apache/arrow-rs/issues/8748
GitHub link: https://github.com/apache/datafusion/discussions/21017#discussioncomment-16209719
----
This is an automatically sent email for github@datafusion.apache.org.
To unsubscribe, please send an email to: github-unsubscribe@datafusion.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [D] Q2 2026 roadmap planning [datafusion]
Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
GitHub user LiaCastaneda edited a comment on the discussion: Q2 2026 roadmap planning
👋 Hello! We’re very interested in having DataFusion support lambda functions. There has been an ongoing epic for this, and some work has already started toward a basic e2e implementation:
- EPIC - https://github.com/apache/datafusion/issues/14205
- (updated) EPIC - https://github.com/apache/datafusion/issues/21172
- First PR – https://github.com/apache/datafusion/pull/18921
It feels like we’re currently missing consensus on the approach to follow. We’d be happy to help -- whether it's reviewing, contributing, or anything else that would move this forward.
Also, @benbellick from substrait-io has added lambda support in Substrait, so it seems the remaining big piece is the function implementation.
cc @gstvg as the one driving this effort!
GitHub link: https://github.com/apache/datafusion/discussions/21017#discussioncomment-16214853
----
This is an automatically sent email for github@datafusion.apache.org.
To unsubscribe, please send an email to: github-unsubscribe@datafusion.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [D] Q2 2026 roadmap planning [datafusion]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
GitHub user paleolimbot added a comment to the discussion: Q2 2026 roadmap planning
I'd be happy to help with statistics generally...my primary interest is in ensuring they can represent geometry statistics in a way that doesn't obliterate them when propagating through DataFusion internals. I know @adriangb is interested in making them lazier and also able to represent nested types (not sure the "lazier" bit is captured in an issue yet).
GitHub link: https://github.com/apache/datafusion/discussions/21017#discussioncomment-16228504
----
This is an automatically sent email for github@datafusion.apache.org.
To unsubscribe, please send an email to: github-unsubscribe@datafusion.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [D] Q2 2026 roadmap planning [datafusion]
Posted by "adp2201 (via GitHub)" <gi...@apache.org>.
GitHub user adp2201 closed a discussion: Q2 2026 roadmap planning
## Q2 2026 roadmap planning
Hi all,
Q2 is coming up, so it seems like a good time to start a fresh roadmap discussion building on the Q1 thread:
- Q1 roadmap: https://github.com/apache/datafusion/issues/18494
The goals here are simple:
1. Identify what should carry forward from Q1 into Q2
2. Identify what new projects we think should be prioritized in Q2
## Likely carry-forward themes
From the Q1 discussion, these seem like the clearest themes to continue:
- **Performance**
- ClickBench / Parquet leaderboard
- Parquet pushdown / late materialization
- planning speed
- sort / join / spilling improvements
- **Developer experience / coordination**
- CI reliability
- contributor workflow
- better coordination across active efforts
- **Extensibility / SQL surface**
- extension types / UDTs
- `VARIANT`
- lambda syntax / array functionality
- async UDF / UDTF work
- datetime / timezone improvements
## Some concrete Q2 candidates
A few open epics / proposal epics that seem especially relevant:
- **Improve query planning speed**: #19795
- **ClickBench Parquet leaderboard**: #18489
- **Improve NUMA-awareness**: #20902
- **Sort-merge join execution improvements**: #20910
- **Support `VARIANT`**: #16116
- **Support Extension Types / UDTs**: #12644
- **Improve developer / CI speed**: #13813
- **Dynamic filtering**: #15512
- **Statistics improvements**: #8227
## Questions
- Which Q1 themes should clearly carry into Q2?
- Which open epics feel most important right now?
- What important Q2 projects are missing?
- Which efforts need more implementation help, benchmarking, or design work?
If you already plan to work on something in Q2, please comment with the issue / epic and the kind of help you’d welcome from others.
GitHub link: https://github.com/apache/datafusion/discussions/21017
----
This is an automatically sent email for github@datafusion.apache.org.
To unsubscribe, please send an email to: github-unsubscribe@datafusion.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [D] Q2 2026 roadmap planning [datafusion]
Posted by "gene-bordegaray (via GitHub)" <gi...@apache.org>.
GitHub user gene-bordegaray added a comment to the discussion: Q2 2026 roadmap planning
Hello! I would like to continue working on [statistics improvements](https://github.com/apache/datafusion/issues/8227) as @asolimando mentioned.
Along with this we have found `RepartitionExec` to be expensive in some cases so diving in on this and making some improvements here is another goal I have. I have not filed an issue or EPIC for this but have been making micro-benches to isolate behavior under different workloads
GitHub link: https://github.com/apache/datafusion/discussions/21017#discussioncomment-16224690
----
This is an automatically sent email for github@datafusion.apache.org.
To unsubscribe, please send an email to: github-unsubscribe@datafusion.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [D] Q2 2026 roadmap planning [datafusion]
Posted by "jonathanc-n (via GitHub)" <gi...@apache.org>.
GitHub user jonathanc-n added a comment to the discussion: Q2 2026 roadmap planning
> Do you have any particular direction for this?
Currently I feel like the current statistics EPIC is a bit old. We can revamp it to point at all the ongoing statistic improvements under their own section:
ex. Geometry statistics, NDV counts, histograms, etc. each having their own section and explaining the reason we need them.
GitHub link: https://github.com/apache/datafusion/discussions/21017#discussioncomment-16259695
----
This is an automatically sent email for github@datafusion.apache.org.
To unsubscribe, please send an email to: github-unsubscribe@datafusion.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [D] Q2 2026 roadmap planning [datafusion]
Posted by "jonathanc-n (via GitHub)" <gi...@apache.org>.
GitHub user jonathanc-n added a comment to the discussion: Q2 2026 roadmap planning
Do we think we should revamp the #8227 ticket for statistics improvements?
GitHub link: https://github.com/apache/datafusion/discussions/21017#discussioncomment-16227386
----
This is an automatically sent email for github@datafusion.apache.org.
To unsubscribe, please send an email to: github-unsubscribe@datafusion.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [D] Q2 2026 roadmap planning [datafusion]
Posted by "benbellick (via GitHub)" <gi...@apache.org>.
GitHub user benbellick added a comment to the discussion: Q2 2026 roadmap planning
From the @substrait-io side, I am happy to help however I can, including leading the implementation of the substrait consumer / producer once consensus is reached on the approach for lambdas. Excited to see this one through!
GitHub link: https://github.com/apache/datafusion/discussions/21017#discussioncomment-16227041
----
This is an automatically sent email for github@datafusion.apache.org.
To unsubscribe, please send an email to: github-unsubscribe@datafusion.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [D] Q2 2026 roadmap planning [datafusion]
Posted by "asolimando (via GitHub)" <gi...@apache.org>.
GitHub user asolimando added a comment to the discussion: Q2 2026 roadmap planning
Hello! I'd like to continue helping out on "statistics improvements", broadly tracked by https://github.com/apache/datafusion/issues/8227, as mentioned in the main description.
I can contribute with PRs and reviews from other active contributors on the topic (@jonathanc-n, @buraksenn, @gene-bordegaray, @kumarUjjawal and I might be forgetting someone!), it would be great if some committer could help us get contributions over the merge line.
cc: (no commitment needed and no pressure, just wanted to make sure you are aware)
- @paleolimbot as you have shown interest in the aforementioned epic
- @xudong963 as you have been very active in the area, notably with https://github.com/apache/datafusion/issues/15873 and recent PRs around stats
- @gabotechs as you have worked and helped with stats already
GitHub link: https://github.com/apache/datafusion/discussions/21017#discussioncomment-16223589
----
This is an automatically sent email for github@datafusion.apache.org.
To unsubscribe, please send an email to: github-unsubscribe@datafusion.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [D] Q2 2026 roadmap planning [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
GitHub user gstvg added a comment to the discussion: Q2 2026 roadmap planning
@benbellick I managed to implement substrait producing and consuming on another branch https://github.com/apache/datafusion/compare/main...gstvg:arrow-datafusion:lambda_substrait, existing lambda tests pass when running `cargo test --test sqllogictests -- --substrait-round-trip lambda.slt`
GitHub link: https://github.com/apache/datafusion/discussions/21017#discussioncomment-16278944
----
This is an automatically sent email for github@datafusion.apache.org.
To unsubscribe, please send an email to: github-unsubscribe@datafusion.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [D] Q2 2026 roadmap planning [datafusion]
Posted by "2010YOUY01 (via GitHub)" <gi...@apache.org>.
GitHub user 2010YOUY01 added a comment to the discussion: Q2 2026 roadmap planning
Thank you for opening this discussion!
This is an advanced predicate pruning feature that I believe is important in the long term. I’d like to help implement it, and would appreciate any help with reviewing.
- https://github.com/apache/datafusion/pull/19609
GitHub link: https://github.com/apache/datafusion/discussions/21017#discussioncomment-16201784
----
This is an automatically sent email for github@datafusion.apache.org.
To unsubscribe, please send an email to: github-unsubscribe@datafusion.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [D] Q2 2026 roadmap planning [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
GitHub user alamb closed the discussion with a comment: Q2 2026 roadmap planning
I filed a ticket to discuss 2026 Q3/Q4
- https://github.com/apache/datafusion/issues/22882
Let's move our conversations there
GitHub link: https://github.com/apache/datafusion/discussions/21017#discussioncomment-17253162
----
This is an automatically sent email for github@datafusion.apache.org.
To unsubscribe, please send an email to: github-unsubscribe@datafusion.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [D] Q2 2026 roadmap planning [datafusion]
Posted by "gene-bordegaray (via GitHub)" <gi...@apache.org>.
GitHub user gene-bordegaray edited a comment on the discussion: Q2 2026 roadmap planning
Hello! I would like to continue working on [statistics improvements](https://github.com/apache/datafusion/issues/8227) as @asolimando mentioned.
Along with this we have found `RepartitionExec` to be expensive in some cases so diving in on this and making some improvements here is another goal I have. I have not filed an issue or EPIC for this but have been making micro-benches to isolate behavior under different workloads but once done think it will be a good effort for others to collaborate on.
GitHub link: https://github.com/apache/datafusion/discussions/21017#discussioncomment-16224690
----
This is an automatically sent email for github@datafusion.apache.org.
To unsubscribe, please send an email to: github-unsubscribe@datafusion.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [D] Q2 2026 roadmap planning [datafusion]
Posted by "kumarUjjawal (via GitHub)" <gi...@apache.org>.
GitHub user kumarUjjawal added a comment to the discussion: Q2 2026 roadmap planning
> > Do you have any particular direction for this?
>
> Currently I feel like the current statistics EPIC is a bit old. We can revamp it to point at all the ongoing statistic improvements under their own section: ex. Geometry statistics, NDV counts, histograms, etc. each having their own section and explaining the reason we need them.
Sounds good!
GitHub link: https://github.com/apache/datafusion/discussions/21017#discussioncomment-16260085
----
This is an automatically sent email for github@datafusion.apache.org.
To unsubscribe, please send an email to: github-unsubscribe@datafusion.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [D] Q2 2026 roadmap planning [datafusion]
Posted by "kumarUjjawal (via GitHub)" <gi...@apache.org>.
GitHub user kumarUjjawal added a comment to the discussion: Q2 2026 roadmap planning
Do you have any particular direction for this?
GitHub link: https://github.com/apache/datafusion/discussions/21017#discussioncomment-16227987
----
This is an automatically sent email for github@datafusion.apache.org.
To unsubscribe, please send an email to: github-unsubscribe@datafusion.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [D] Q2 2026 roadmap planning [datafusion]
Posted by "kumarUjjawal (via GitHub)" <gi...@apache.org>.
GitHub user kumarUjjawal added a comment to the discussion: Q2 2026 roadmap planning
Sound good! I would be happy to help with this. We should make sure all the relevant issue related to statistics improvements are tracked by https://github.com/apache/datafusion/issues/8227.
GitHub link: https://github.com/apache/datafusion/discussions/21017#discussioncomment-16224439
----
This is an automatically sent email for github@datafusion.apache.org.
To unsubscribe, please send an email to: github-unsubscribe@datafusion.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [D] Q2 2026 roadmap planning [datafusion]
Posted by "LiaCastaneda (via GitHub)" <gi...@apache.org>.
GitHub user LiaCastaneda added a comment to the discussion: Q2 2026 roadmap planning
👋 Hello! We’re very interested in having DataFusion support lambda functions. There has been an ongoing epic for this, and some work has already started toward a basic e2e implementation:
- EPIC - https://github.com/apache/datafusion/issues/14205
- First PR – https://github.com/apache/datafusion/pull/18921
It feels like we’re currently missing consensus on the approach to follow. We’d be happy to help -- whether it's reviewing, contributing, or anything else that would move this forward.
Also, @benbellick from substrait-io has added lambda support in Substrait, so it seems the remaining big piece is the function implementation.
GitHub link: https://github.com/apache/datafusion/discussions/21017#discussioncomment-16214853
----
This is an automatically sent email for github@datafusion.apache.org.
To unsubscribe, please send an email to: github-unsubscribe@datafusion.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [D] Q2 2026 roadmap planning [datafusion]
Posted by "2010YOUY01 (via GitHub)" <gi...@apache.org>.
GitHub user 2010YOUY01 edited a comment on the discussion: Q2 2026 roadmap planning
Thank you for opening this discussion!
This is an advanced predicate pruning feature that I believe is important in the long term. I’d like to help implement it, and would appreciate any help with reviewing.
- https://github.com/apache/datafusion/pull/19609
Also for testing coverage, @sdf-jkl and I are working on a SQL fuzzing framework for DataFusion:
- https://github.com/apache/datafusion/issues/14535
- https://github.com/2010YOUY01/datafusion-fuzzer
We hope to complete the initial version soon and start deploying it in CI.
GitHub link: https://github.com/apache/datafusion/discussions/21017#discussioncomment-16201784
----
This is an automatically sent email for github@datafusion.apache.org.
To unsubscribe, please send an email to: github-unsubscribe@datafusion.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org