You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "gianm (via GitHub)" <gi...@apache.org> on 2023/04/25 07:06:05 UTC
[GitHub] [druid] gianm opened a new issue, #14157: Community roadmap 2023
gianm opened a new issue, #14157:
URL: https://github.com/apache/druid/issues/14157
It would be cool to have a publicly published roadmap of things the community is working on, so I thought we should do one for 2023. The year isn't even half over, so it seems like a good time 🙂.
I'm starting this list off with some things that are on my radar. I hope everyone reading this sees some exciting features on the list! Please feel free to post here if you've got any feedback, questions, or comments about items that are or aren't here.
Btw, note that items being on this list doesn't mean they will definitely ship in 2023. It is rather a statement that there is interest in working on them, and some likelihood they will get done. So, it's a good way to get a sense for what kinds of things are receiving active development interest and activity.
### Query language
- Window functions (#13458, #13514, #13902, #13773)
- Distributed shuffling JOIN (#13506)
- Array columns (#13803)
- Array UNNEST (#13268)
- Standardize on SQL-compatible null handling (#14154)
- ORDER BY non-`__time` in native Scan query
### Query engine
- Detect unavailable segments (#13967)
- Larger, self-adjusting subquery limit (#13952)
- High-cardinality GROUP BY performance enhancements
- Interactive API for multi-stage query
- Export multi-stage query task results to object storage
- Query directly from object storage
- Update to Calcite 1.34 or later (#13532)
- EXPLAIN ANALYZE
### Data management
- Batch DELETE DML (#14145)
- Batch UPDATE, UPSERT DML
- DDL (CREATE / ALTER / DROP TABLE)
- Schema auto-discovery
- Compaction concurrent with streaming ingestion
- Metadata catalog (#12546)
- Validation of load rules, partitioning, etc
- Broadcast tables (alternative to lookups)
- Reset to arbitrary Kafka offsets
### Operations
- Java 17 full support (#12838)
- Java 21 experimental support
- Overlord scalability to larger numbers of tasks
- Coordinator scalability to larger numbers of segments
- K8s task runner (#13156 and follow-ups)
- SysMonitor on ARM
### For Druid developers
- Standardized extension API
- Improve build pipeline and testing system
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] gianm commented on issue #14157: Community roadmap 2023
Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm commented on issue #14157:
URL: https://github.com/apache/druid/issues/14157#issuecomment-1530862348
> Kubernetes - Control Plane For Apache Druid
@AdheipSingh Very cool! I added some of this stuff to a new "Ecosystem" section.
> > shall I add https://github.com/apache/druid/issues/13923? I think a lot of people would be into this.
> Yes please, I'll try to find sometime soon to put up a PR.
@a2l007 Excellent, I added this to the "Data management" section.
> and would also like to add https://github.com/apache/druid/pull/10920 to the roadmap
> I will pick it up and get it over the line in q3 of this year
@maytasm Thanks! I added this to the "Data management" section.
> > With regard to https://github.com/apache/druid/issues/12526 ...
> I plan on picking it up this year if others don't have cycles.
@suneet-s Thanks! I added this to the "Data management" section.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] suneet-s commented on issue #14157: Community roadmap 2023
Posted by "suneet-s (via GitHub)" <gi...@apache.org>.
suneet-s commented on issue #14157:
URL: https://github.com/apache/druid/issues/14157#issuecomment-1522715219
> With regard to https://github.com/apache/druid/issues/12526 — @capistrant would you be able to pick this back up this year? Or if not, @suneet-s do you think you or someone else would be able to take it over the finish line? If so, I'd love to add it.
I plan on picking it up this year if others don't have cycles.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] maytasm commented on issue #14157: Community roadmap 2023
Posted by "maytasm (via GitHub)" <gi...@apache.org>.
maytasm commented on issue #14157:
URL: https://github.com/apache/druid/issues/14157#issuecomment-1522711510
@gianm
> @maytasm I would like to add it too 🙂 although it seems stuck. Do you know if @JulianJaffePinterest or you would be able to pick it back up? I know it's gone through the wringer during review, so if it gets picked back up I will set some time aside to personally stay with you til it gets in. That should help.
I will pick it up and get it over the line in q3 of this year
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] abhishekagarwal87 commented on issue #14157: Community roadmap 2023
Posted by "abhishekagarwal87 (via GitHub)" <gi...@apache.org>.
abhishekagarwal87 commented on issue #14157:
URL: https://github.com/apache/druid/issues/14157#issuecomment-1600564312
I will also recommend adding https://github.com/apache/druid/pull/14424 to this roadmap. :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] JulianJaffePinterest commented on issue #14157: Community roadmap 2023
Posted by "JulianJaffePinterest (via GitHub)" <gi...@apache.org>.
JulianJaffePinterest commented on issue #14157:
URL: https://github.com/apache/druid/issues/14157#issuecomment-1535686475
@maytasm I should have some time in the next month or so to put up my Spark 3.3 update to #10920. There are a few pieces I need to pull out and patch over so if you already have a version in progress I'll defer to you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] suneet-s commented on issue #14157: Community roadmap 2023
Posted by "suneet-s (via GitHub)" <gi...@apache.org>.
suneet-s commented on issue #14157:
URL: https://github.com/apache/druid/issues/14157#issuecomment-1522188368
I think something under Operations for better kill tasks to help operators save costs would be nice to add. #14131 makes them faster and #12526 would make it safer for operators to use.
Under data management - I see there is already a note about concurrent compaction. Related to that I'd like to add #13852 Both seem to be in service of making compaction faster and more robust.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] gianm commented on issue #14157: Community roadmap 2023
Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm commented on issue #14157:
URL: https://github.com/apache/druid/issues/14157#issuecomment-1522588674
> I think something under Operations for better kill tasks to help operators save costs would be nice to add. #14131 makes them faster and #12526 would make it safer for operators to use.
With regard to #12526 — @capistrant would you be able to pick this back up this year? Or if not, @suneet-s do you think you or someone else would be able to take it over the finish line? If so, I'd love to add it.
> Under data management - I see there is already a note about concurrent compaction. Related to that I'd like to add #13852 Both seem to be in service of making compaction faster and more robust.
@suneet-s added as "Continuous auto-compaction"! 🚀
> and would also like to add #10920 to the roadmap
@maytasm I would like to add it too 🙂 although it seems stuck. Do you know if @JulianJaffePinterest or you would be able to pick it back up? I know it's gone through the wringer during review, so if it gets picked back up I will set some time aside to personally stay with you til it gets in. That should help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] a2l007 commented on issue #14157: Community roadmap 2023
Posted by "a2l007 (via GitHub)" <gi...@apache.org>.
a2l007 commented on issue #14157:
URL: https://github.com/apache/druid/issues/14157#issuecomment-1522692274
> @a2l007 shall I add #13923? I think a lot of people would be into this.
Yes please, I'll try to find sometime soon to put up a PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Community roadmap 2023 (druid)
Posted by "maytasm (via GitHub)" <gi...@apache.org>.
maytasm commented on issue #14157:
URL: https://github.com/apache/druid/issues/14157#issuecomment-2711844674
@gianm Should we do another one of this roadmap for 2025 (given that we are still early in 2025)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] gianm commented on issue #14157: Community roadmap 2023
Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm commented on issue #14157:
URL: https://github.com/apache/druid/issues/14157#issuecomment-1521275588
@a2l007 shall I add #13923? I think a lot of people would be into this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] maytasm commented on issue #14157: Community roadmap 2023
Posted by "maytasm (via GitHub)" <gi...@apache.org>.
maytasm commented on issue #14157:
URL: https://github.com/apache/druid/issues/14157#issuecomment-1522322666
+1 on https://github.com/apache/druid/issues/13923
and would also like to add https://github.com/apache/druid/pull/10920
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] AdheipSingh commented on issue #14157: Community roadmap 2023
Posted by "AdheipSingh (via GitHub)" <gi...@apache.org>.
AdheipSingh commented on issue #14157:
URL: https://github.com/apache/druid/issues/14157#issuecomment-1522602656
### Kubernetes - Control Plane For Apache Druid
- We believe that Kubernetes can serve as a control plane for any application, including those with data and stateful sets. While Helm charts are useful for configuration management, they don't maintain the state of the application. That's why we've plan to build a control plane based on kubernetes operator pattern that acts as a bridge between your application's requirements and Kubernetes infrastructure.
Here are few pointers to be worked upon :
- Building Ingestion Spec controller [WIP](https://github.com/datainfrahq/druid-operator/pull/22)
- Scaling Historicals based on [usage](https://github.com/datainfrahq/druid-operator/issues/40)
- Rest a lot of work is needed to build up documentation, which is being worked upon :)
Looking forward for collaboration and feedback
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] gianm commented on issue #14157: Community roadmap 2023
Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm commented on issue #14157:
URL: https://github.com/apache/druid/issues/14157#issuecomment-1521275517
Some wish-list items that I have heard people talk about, but I don't know if someone is planning to work on:
- Cloud object supervisor
- Single-process Druid
- Non-time based partitioning
- Realtime upsert from Kafka / Kinesis
- Scheduled queries
If you are interested in these (using, or building) let us know.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org