You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2021/01/13 20:49:41 UTC
[GitHub] [druid] jihoonson opened a new issue #10752: [Draft] 0.21.0 Release Notes
jihoonson opened a new issue #10752:
URL: https://github.com/apache/druid/issues/10752
Apache Druid 0.21.0 contains around 120 new features, bug fixes, performance enhancements, documentation improvements, and additional test coverage from 35 contributors. Refer to the complete list of changes and everything tagged to the milestone for further details.
# <a name="21-new-features" href="#21-new-features">#</a> New features
## Ingestion and querying
### Segment-level caching for join queries
Join queries now support segment-level caching in historicals.
https://github.com/apache/druid/pull/10366
### Query timeout handling
https://github.com/apache/druid/pull/10464
### New `grouping` function
A new `grouping` SQL function can be used with `GROUPING SETS` or `CUBE`. The result of this function indicates which grouping dimensions are included in the current grouping set. See [Aggregation functions](https://druid.apache.org/docs/latest/querying/sql.html#aggregation-functions) for more details.
https://github.com/apache/druid/pull/10518
### Consistent type handling for expressions
https://github.com/apache/druid/pull/10499
### Old kafka version support
https://github.com/apache/druid/pull/10551
### Native parallel ingestion can run without explicit intervals
https://github.com/apache/druid/pull/10592
### `context` dimension in query metrics
https://github.com/apache/druid/pull/10578
### Query timeout metrics
https://github.com/apache/druid/pull/10567
### Add zero period support to TIMESTAMPADD
https://github.com/apache/druid/pull/10550
### Tidy up HTTP status codes for query errors
https://github.com/apache/druid/pull/10746
## Web console
### Web console: targetRowsPerSegment for hashed partioning
https://github.com/apache/druid/pull/10500
### Web console: show segment sizes in rows not bytes
https://github.com/apache/druid/pull/10496
### Web console: refresh and tighten up the console styles ✨💅💫
https://github.com/apache/druid/pull/10515
### Web console: improve make type preservation in ingestion configs in the data loader
https://github.com/apache/druid/pull/10533
### Web console: Double confirm before permanently delete unused segments
https://github.com/apache/druid/pull/10581
### Match the column order between in the Schema table and dimensionsSpec
https://github.com/apache/druid/pull/10588
### Web console: improve how code is imported, use API instance
https://github.com/apache/druid/pull/10597
### Web console: Improve the handling of extreme data (funky datasources, longs)
https://github.com/apache/druid/pull/10641
### Web console: reflect the changes to interval requirement in the data loader flow
https://github.com/apache/druid/pull/10647
## Others
### k8s disco
https://github.com/apache/druid/pull/10544
https://github.com/apache/druid/pull/9507
https://github.com/apache/druid/pull/10537
### New password provider based on AWS RDS token
https://github.com/apache/druid/pull/9518
### Add https to druid-influxdb-emitter extension
https://github.com/apache/druid/pull/9938
### make dimension column extensible with COMPLEX type
https://github.com/apache/druid/pull/10277
### Add dynamic coordinator config that allows control over how many segments are considered when picking a segment to move.
https://github.com/apache/druid/pull/10284
### introduce DynamicConfigProvider interface and make kafka consumer props extensible
https://github.com/apache/druid/pull/10309
### Shuffle metrics for batch ingestion
https://github.com/apache/druid/pull/10359
### Added CronScheduler support as a proof to clock drift while emitting metrics
https://github.com/apache/druid/pull/10448
https://github.com/apache/druid/pull/10732
### Add support for domain allow/deny list for HTTPInputSource
https://github.com/apache/druid/pull/10535
### New coordinator duty runtime metrics
https://github.com/apache/druid/pull/10603
### Historical load Segments enhancement
https://github.com/apache/druid/pull/10650
### DruidInputSource: Sort segments by ID before grouping into splits.
https://github.com/apache/druid/pull/10646
### Adding example config files for indexer process. Also added status/selfDiscovered endpoint to indexer for self discovery
https://github.com/apache/druid/pull/10679
### Servers table shows leaders
https://github.com/apache/druid/pull/10680
## Docker
### Reduce docker image size by half
https://github.com/apache/druid/pull/10506
### druid-docker-image: add DRUID_DIRS_TO_CREATE variable to customize directories created on startup
https://github.com/apache/druid/pull/10591
### Fix issue causing incorrect config in Docker
https://github.com/apache/druid/pull/10595
### Return appropriate config directory path for indexer process.
https://github.com/apache/druid/pull/10657
# <a name="21-bug-fixes" href="#21-bug-fixes">#</a> Bug fixes
### All fields in segments table are now json-serialized
https://github.com/apache/druid/pull/10481
### Post-aggregator computation with subtotals
https://github.com/apache/druid/pull/10653
### Indexers announce themselves as segment servers
https://github.com/apache/druid/pull/10631
# <a name="21-known-issues" href="#21-known-issues">#</a> Known issues
TBD
# <a name="21-upgrading-from-0.20.0" href="#21-upgrading-from-0.20.0">#</a> Upgrading from 0.20.0
TBD
# <a name="21-credits" href="#21-credits">#</a> Credits
Thanks to everyone who contributed to this release!
@a2l007
@abhishekagarwal87
@asdf2014
@AshishKapoor
@awelsh93
@ayushkul2910
@capistrant
@ccaominh
@clintropolis
@cloventt
@FrankChen021
@gianm
@harinirajendran
@himanshug
@jihoonson
@jon-wei
@kroeders
@liran-funaro
@martin-g
@maytasm
@mghosh4
@michaelschiff
@nishantmonu51
@pcarrier
@QingdongZeng3
@sthetland
@suneet-s
@tdt17
@techdocsmith
@valdemar-giosg
@viatcheslavmogilevsky
@viongpanzi
@vogievetsky
@xvrl
@zhangyue19921010
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jihoonson commented on issue #10752: [Draft] 0.21.0 Release Notes
Posted by GitBox <gi...@apache.org>.
jihoonson commented on issue #10752:
URL: https://github.com/apache/druid/issues/10752#issuecomment-827009490
Hi @jp707049, no worries and sorry for the delay. The 0.21.0 release vote has just passed! We will announce the release tomorrow.
When you try the k8s extension, please keep it mind that the extension is experimental and so you might be able to see some unexpected bugs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] himanshug commented on issue #10752: [Draft] 0.21.0 Release Notes
Posted by GitBox <gi...@apache.org>.
himanshug commented on issue #10752:
URL: https://github.com/apache/druid/issues/10752#issuecomment-760547707
hey @jihoonson , I am very excited about `Service discovery and leader election based on Kubernetes` but I am in the process of writing a PR that runs #10680 for k8s based deployment to stress test the k8s based leader election mechanism ... In that process I found two improvements,
(1) upgrade to newer version of `kubernetes-client` library which has some leader election improvements and makes leader changes happen quicker if a node is gracefully stopped
(2) a bug which shows up in specific deployments where `druid.host` gets set so something more than 63 characters which is the max length limit on kubernetes label values
I will address all of the above including introducing the new build in a new PR, those updates would make this feature a lot more robust and production ready. Thanks to @clintropolis for introducing #10680 which really stresses the overlord/coordinator startup, discovery, leader election machinery to help discover above improvements.
that said, I think, 0.21.0 should continue to go on its schedule. For that reason, I think we should remove the mention of this new feature in 0.21.0 release notes and we can document its availability in next release.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jihoonson commented on issue #10752: [Draft] 0.21.0 Release Notes
Posted by GitBox <gi...@apache.org>.
jihoonson commented on issue #10752:
URL: https://github.com/apache/druid/issues/10752#issuecomment-828059789
I just announced the 0.21.0 release. Thanks everyone for your patience!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] xvrl commented on issue #10752: [Draft] 0.21.0 Release Notes
Posted by GitBox <gi...@apache.org>.
xvrl commented on issue #10752:
URL: https://github.com/apache/druid/issues/10752#issuecomment-828047008
@jihoonson do you plan to update the release tag with the release notes and close this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jihoonson commented on issue #10752: [Draft] 0.21.0 Release Notes
Posted by GitBox <gi...@apache.org>.
jihoonson commented on issue #10752:
URL: https://github.com/apache/druid/issues/10752#issuecomment-760575610
Thanks. That sounds good :slightly_smiling_face:
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jihoonson closed issue #10752: [Draft] 0.21.0 Release Notes
Posted by GitBox <gi...@apache.org>.
jihoonson closed issue #10752:
URL: https://github.com/apache/druid/issues/10752
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jihoonson commented on issue #10752: [Draft] 0.21.0 Release Notes
Posted by GitBox <gi...@apache.org>.
jihoonson commented on issue #10752:
URL: https://github.com/apache/druid/issues/10752#issuecomment-828051575
Apologize for the delay. I'm finalizing the release process now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] Gabriel39 edited a comment on issue #10752: [Draft] 0.21.0 Release Notes
Posted by GitBox <gi...@apache.org>.
Gabriel39 edited a comment on issue #10752:
URL: https://github.com/apache/druid/issues/10752#issuecomment-774485685
Hi @jihoonson this link https://druid.apache.org/docs/latest/development/extensions-core/kubernetes.html seems like outdated with 404 error. Is there a new site to refer about k8s extension?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jp707049 commented on issue #10752: [Draft] 0.21.0 Release Notes
Posted by GitBox <gi...@apache.org>.
jp707049 commented on issue #10752:
URL: https://github.com/apache/druid/issues/10752#issuecomment-826481916
Hello,
- When the `0.21.0` will be released?
- Sorry to keep asking about updates on when it will be released. I'm looking forward to test out `Kubernetes extension` in our Druid clusters.
Thank you :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jihoonson commented on issue #10752: [Draft] 0.21.0 Release Notes
Posted by GitBox <gi...@apache.org>.
jihoonson commented on issue #10752:
URL: https://github.com/apache/druid/issues/10752#issuecomment-760555607
> that said, I think, 0.21.0 should continue to go on its schedule. For that reason, I think we should remove the mention of this new feature in 0.21.0 release notes and we can document its availability in next release.
@himanshug thanks for taking a look. I understand it's not matured enough to use in production yet, but would like to keep it because it's one of the most exciting new features that 0.21.0 introduces and worth drawing people's attention. From my experience, most people don't want to test new features out in their production right away when they are first introduced. Rather, they would like to wait for a couple of releases until those features become matured and stable. In this case, as it is stated as experimental, people would be especially careful before trying it out. Later, once it becomes production ready, we can promote it again in the release notes by stating like "this cool feature is now production ready. everyone should try it out".
If we didn't mention it in this release, but in some future release when it becomes more stable, I think that people could still underestimate its maturity because it would be a feature that is promoted in the release notes first time. What do you think?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jp707049 commented on issue #10752: [Draft] 0.21.0 Release Notes
Posted by GitBox <gi...@apache.org>.
jp707049 commented on issue #10752:
URL: https://github.com/apache/druid/issues/10752#issuecomment-826481916
Hello,
- When the `0.21.0` will be released?
- Sorry to keep asking about updates on when it will be released. I'm looking forward to test out `Kubernetes extension` in our Druid clusters.
Thank you :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jihoonson commented on issue #10752: [Draft] 0.21.0 Release Notes
Posted by GitBox <gi...@apache.org>.
jihoonson commented on issue #10752:
URL: https://github.com/apache/druid/issues/10752#issuecomment-759885436
@FrankChen021 hmm, sure. It seems worth calling it out. I added it under the bug fixes section.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] Gabriel39 commented on issue #10752: [Draft] 0.21.0 Release Notes
Posted by GitBox <gi...@apache.org>.
Gabriel39 commented on issue #10752:
URL: https://github.com/apache/druid/issues/10752#issuecomment-774485685
Hi @jihoonson this link https://druid.apache.org/docs/latest/development/extensions-core/kubernetes.html seems like outdated. Is there a new site to refer about k8s extension?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jihoonson commented on issue #10752: [Draft] 0.21.0 Release Notes
Posted by GitBox <gi...@apache.org>.
jihoonson commented on issue #10752:
URL: https://github.com/apache/druid/issues/10752#issuecomment-822632033
Hi @stroeovidiu, the vote for 0.21.0 release is in progress. It will pass in a couple of days unless we find some issue in the release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] FrankChen021 commented on issue #10752: [Draft] 0.21.0 Release Notes
Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on issue #10752:
URL: https://github.com/apache/druid/issues/10752#issuecomment-759867857
Hi @jihoonson , should #10363 be mentioned in the release note since it has changed the user-facing configurations ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] stroeovidiu commented on issue #10752: [Draft] 0.21.0 Release Notes
Posted by GitBox <gi...@apache.org>.
stroeovidiu commented on issue #10752:
URL: https://github.com/apache/druid/issues/10752#issuecomment-822459052
Hello,
Any update when the 0.21.0 will be released ?
Thank you !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] himanshug edited a comment on issue #10752: [Draft] 0.21.0 Release Notes
Posted by GitBox <gi...@apache.org>.
himanshug edited a comment on issue #10752:
URL: https://github.com/apache/druid/issues/10752#issuecomment-760547707
hey @jihoonson , I am very excited about `Service discovery and leader election based on Kubernetes` but I am in the process of writing a PR that runs #10680 for k8s based deployment to stress test the k8s based leader election mechanism ... In that process I found two improvements,
(1) upgrade to newer version of `kubernetes-client` library which has some leader election improvements and makes leader changes happen quicker if a node is gracefully stopped
(2) a bug which shows up in specific deployments where `druid.host` gets set to something more than 63 characters which is the max length limit on kubernetes label values
I will address all of the above including introducing the new build in a new PR, those updates would make this feature a lot more robust and production ready. Thanks to @clintropolis for introducing #10680 which really stresses the overlord/coordinator startup, discovery, leader election machinery to help discover above improvements.
that said, I think, 0.21.0 should continue to go on its schedule. For that reason, I think we should remove the mention of this new feature in 0.21.0 release notes and we can document its availability in next release.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jihoonson commented on issue #10752: [Draft] 0.21.0 Release Notes
Posted by GitBox <gi...@apache.org>.
jihoonson commented on issue #10752:
URL: https://github.com/apache/druid/issues/10752#issuecomment-774529269
Hi @Gabriel39, the kubernetes extension is added in 0.21.0 and not available in the latest release (0.20.1) yet. The doc will be updated once 0.21.0 is released.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] himanshug commented on issue #10752: [Draft] 0.21.0 Release Notes
Posted by GitBox <gi...@apache.org>.
himanshug commented on issue #10752:
URL: https://github.com/apache/druid/issues/10752#issuecomment-760562141
sure, you are the release manager :)
anyways, now that the conversation is here in this issue, power users would likely note the upcoming improvements. however, by that time new PR would be done and merged for them to cherry pick that specific commit.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org