You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "317brian (via GitHub)" <gi...@apache.org> on 2026/08/09 16:02:21 UTC
[PR] docs: 38.0.0 release notes (druid)
317brian opened a new pull request, #19945:
URL: https://github.com/apache/druid/pull/19945
WIP release notes for 38.0.0.
<!-- Thanks for trying to help us make Apache Druid be the best it can be! Please fill out as much of the following information as is possible (where relevant, and remove it when irrelevant) to help make the intention and scope of this PR clear in order to ease review. -->
<!-- Please read the doc for contribution (https://github.com/apache/druid/blob/master/CONTRIBUTING.md) before making this PR. Also, once you open a PR, please _avoid using force pushes and rebasing_ since these make it difficult for reviewers to see what you've changed in response to their reviews. See [the 'If your pull request shows conflicts with master' section](https://github.com/apache/druid/blob/master/CONTRIBUTING.md#if-your-pull-request-shows-conflicts-with-master) for more details. -->
Fixes #XXXX.
<!-- Replace XXXX with the id of the issue fixed in this PR. Remove this section if there is no corresponding issue. Don't reference the issue in the title of this pull-request. -->
<!-- If you are a committer, follow the PR action item checklist for committers:
https://github.com/apache/druid/blob/master/dev/committer-instructions.md#pr-and-issue-action-item-checklist-for-committers. -->
### Description
<!-- Describe the goal of this PR, what problem are you fixing. If there is a corresponding issue (referenced above), it's not necessary to repeat the description here, however, you may choose to keep one summary sentence. -->
<!-- Describe your patch: what did you change in code? How did you fix the problem? -->
<!-- If there are several relatively logically separate changes in this PR, create a mini-section for each of them. For example: -->
#### Fixed the bug ...
#### Renamed the class ...
#### Added a forbidden-apis entry ...
<!--
In each section, please describe design decisions made, including:
- Choice of algorithms
- Behavioral aspects. What configuration values are acceptable? How are corner cases and error conditions handled, such as when there are insufficient resources?
- Class organization and design (how the logic is split between classes, inheritance, composition, design patterns)
- Method organization and design (how the logic is split between methods, parameters and return types)
- Naming (class, method, API, configuration, HTTP endpoint, names of emitted metrics)
-->
<!-- It's good to describe an alternative design (or mention an alternative name) for every design (or naming) decision point and compare the alternatives with the designs that you've implemented (or the names you've chosen) to highlight the advantages of the chosen designs and names. -->
<!-- If there was a discussion of the design of the feature implemented in this PR elsewhere (e. g. a "Proposal" issue, any other issue, or a thread in the development mailing list), link to that discussion from this PR description and explain what have changed in your final design compared to your original proposal or the consensus version in the end of the discussion. If something hasn't changed since the original discussion, you can omit a detailed discussion of those aspects of the design here, perhaps apart from brief mentioning for the sake of readability of this PR description. -->
<!-- Some of the aspects mentioned above may be omitted for simple and small changes. -->
#### Release note
<!-- Give your best effort to summarize your changes in a couple of sentences aimed toward Druid users.
If your change doesn't have end user impact, you can skip this section.
For tips about how to write a good release note, see [Release notes](https://github.com/apache/druid/blob/master/CONTRIBUTING.md#release-notes).
-->
<hr>
##### Key changed/added classes in this PR
* `MyFoo`
* `OurBar`
* `TheirBaz`
<hr>
<!-- Check the items by putting "x" in the brackets for the done things. Not all of these items apply to every PR. Remove the items which are not done or not relevant to the PR. None of the items from the checklist below are strictly necessary, but it would be very helpful if you at least self-review the PR. -->
This PR has:
- [ ] been self-reviewed.
- [ ] using the [concurrency checklist](https://github.com/apache/druid/blob/master/dev/code-review/concurrency.md) (Remove this item if the PR doesn't have any relation to concurrency.)
- [ ] added documentation for new or modified features or behaviors.
- [ ] a release note entry in the PR description.
- [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
- [ ] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/dev/license.md)
- [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
- [ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
- [ ] added integration tests.
- [ ] been tested in a test Druid cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] docs: 38.0.0 release notes (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #19945:
URL: https://github.com/apache/druid/pull/19945#discussion_r4154618259
##########
docs/release-info/release-notes.md:
##########
@@ -65,42 +134,389 @@ This section contains detailed release notes separated by areas.
#### Other web console improvements
+- Added the following status details to the **Services** view for Historical services: cloning from another Historical, in turbo loading mode, in decommissioning mode [#19253](https://github.com/apache/druid/pull/19253)
+- Added support for resetting a supervisor to the latest offsets and backfilling [#19533](https://github.com/apache/druid/pull/19533)
+- Improved handling for new tabs [#19483](https://github.com/apache/druid/pull/19483)
+- Improved the Home view's **Services** card. It now reports Overlord, Coordinator, Router, Broker, and Indexer counts on clusters where the web console talks to the Coordinator without SQL access [#19481](https://github.com/apache/druid/pull/19481)
+
### Ingestion
+* You can now use the expression aggregator at ingestion time for expressions that produce a LONG or DOUBLE for both fold and combine expressions [#19508](https://github.com/apache/druid/pull/19508)
+* Added `now()` expression function that returns the current system timestamp in milliseconds since epoch. Useful at ingestion time for troubleshooting pipeline delays (e.g., `now() - __time`). Note: `now()` is non-deterministic as it evaluates for every row, so it can break idempotency. This can be added to any column besides `__time` [#19386](https://github.com/apache/druid/pull/19386)
+* Improved resiliency when ingesting from S3. Druid now retries on `SSLException` and transient credential errors instead of failing [#19617](https://github.com/apache/druid/pull/19617) [#19558](https://github.com/apache/druid/pull/19558)
+* Improved S3 performance [#19394](https://github.com/apache/druid/pull/19394)
+* Updated the default S3 connection pool size so that it's computed based on the number of available processors [#19536](https://github.com/apache/druid/pull/19536)
+
#### SQL-based ingestion
+##### EXTERN for S3 now supports role ARN
+
+You can now include the role ARN when running an INSERT INTO EXTERN query. For example:
+
+```
+INSERT INTO
+EXTERN(
+ s3(bucket => 'test2', prefix => 'export', assumeRoleArn => 'arn:aws:iam::00000:role/test-20260520'))
+AS CSV
+SELECT ...
+```
+
+[#19317](https://github.com/apache/druid/pull/19317)
+
##### Other SQL-based ingestion improvements
+- Added storage counters for the amount of bytes and files written to local and durable storage as well as the state of the local `ByteTracker`. Per-worker storage counters have also been added to the web console [#19316](https://github.com/apache/druid/pull/19316)
+
#### Streaming ingestion
+##### Scaling cool down
+
+You can now configure different cool down durations for scaling up and scaling down streaming task autoscalers.
+
+[#19286](https://github.com/apache/druid/pull/19286)
+
+##### Improved supervisor restarts
+
+Supervisors no longer restart for all changes. Based on the type of change, one of the following can occur:
+
+- Druid persists the updated spec without a restart.
+- Druid restarts the supervisor, but running tasks aren't impacted.
+- Druid restarts the supervisor, terminates the current tasks and launches fresh tasks (default behaviour in older Druid versions).
Review Comment:
```suggestion
- Druid restarts the supervisor, terminates the current tasks and launches fresh tasks (default behavior in older Druid versions).
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] docs: 38.0.0 release notes (druid)
Posted by "techdocsmith (via GitHub)" <gi...@apache.org>.
techdocsmith commented on code in PR #19945:
URL: https://github.com/apache/druid/pull/19945#discussion_r3787327874
##########
docs/release-info/release-notes.md:
##########
@@ -57,6 +57,80 @@ For tips about how to write a good release note, see [Release notes](https://git
This section contains important information about new and existing features.
+#### Java
+
+Druid now supports Java 25. While Druid 21 is still supported, we recommend you upgrade to Java 25.
Review Comment:
```suggestion
Druid now supports Java 25. While Java 21 is still supported, we recommend you upgrade to Java 25.
```
Should be Java? See line 435
##########
docs/release-info/release-notes.md:
##########
@@ -57,6 +57,80 @@ For tips about how to write a good release note, see [Release notes](https://git
This section contains important information about new and existing features.
+#### Java
+
+Druid now supports Java 25. While Druid 21 is still supported, we recommend you upgrade to Java 25.
+
+Support for Java 17 has been dropped.
+
+[#19304](https://github.com/apache/druid/pull/19304) [#19336](https://github.com/apache/druid/pull/19336)
+
+#### Historical tier aliases
+
+You can now use the `historicalTierAliases` Coordinator dynamic configuration to map a virtual tier name to a set of real Historical tiers so that a group of Historical tiers has a single identifier. When a load/drop rule references the alias, the Coordinator replaces it with the actual tiers. For example, if you map the Historical tiers `hot_1` and `hot_2` to the alias `hot`, the rule `{"hot": 2}` loads 2 replicas of each onto `hot_1` and `hot_2`.
+
+[#19204](https://github.com/apache/druid/pull/19204) [#19667](https://github.com/apache/druid/pull/19667)
+
+#### New load rule types
+
+Adds a new family of retention rules, `loadPartialByPeriod`, `loadPartialByInterval`, `loadPartialForever`, laying the groundwork for partial loading of version 10 segment projections on Historicals.
+
+[#19374](https://github.com/apache/druid/pull/19374)
+
+#### Realtime segments query context
Review Comment:
```suggestion
#### Realtime segments mode query context
```
##########
docs/release-info/release-notes.md:
##########
@@ -65,42 +139,389 @@ This section contains detailed release notes separated by areas.
#### Other web console improvements
+- Added the following status details to the **Services** view for Historical services: cloning from another Historical, in turbo loading mode, in decommissioning mode [#19253](https://github.com/apache/druid/pull/19253)
+- Added support for resetting a supervisor to the latest offsets and backfilling [#19533](https://github.com/apache/druid/pull/19533)
+- Improved how new tabs are handled [#19483](https://github.com/apache/druid/pull/19483)
Review Comment:
```suggestion
- Improved handling for new tabs [#19483](https://github.com/apache/druid/pull/19483)
```
##########
docs/release-info/release-notes.md:
##########
@@ -65,42 +139,389 @@ This section contains detailed release notes separated by areas.
#### Other web console improvements
+- Added the following status details to the **Services** view for Historical services: cloning from another Historical, in turbo loading mode, in decommissioning mode [#19253](https://github.com/apache/druid/pull/19253)
+- Added support for resetting a supervisor to the latest offsets and backfilling [#19533](https://github.com/apache/druid/pull/19533)
+- Improved how new tabs are handled [#19483](https://github.com/apache/druid/pull/19483)
+- Improved the Home view's **Services** card. It now reports Overlord, Coordinator, Router, Broker, and Indexer counts on clusters where the web console talks to the Coordinator without SQL access [#19481](https://github.com/apache/druid/pull/19481)
+
### Ingestion
+* You can now use the expression aggregator at ingestion time for expressions that produce a LONG or DOUBLE for both fold and combine expressions [#19508](https://github.com/apache/druid/pull/19508)
+* Added `now()` expression function that returns the current system timestamp in milliseconds since epoch. Useful at ingestion time for troubleshooting pipeline delays (e.g., `now() - __time`). Note: `now()` is non-deterministic as it evaluates for every row, so it can break idempotency. This can be added to any besides `__time` [#19386](https://github.com/apache/druid/pull/19386)
+* Improved resiliency when ingesting from S3. Druid now retries on `SSLException` and transient credential errors instead of failing [#19617](https://github.com/apache/druid/pull/19617) [#19558](https://github.com/apache/druid/pull/19558)
+* Improved S3 performance [#19394](https://github.com/apache/druid/pull/19394)
+* Updated the default S3 connection pool size so that it's computed based on the number of available processors [#19536](https://github.com/apache/druid/pull/19536)
+
#### SQL-based ingestion
+##### EXTERN for S3 now supports role ARN
+
+You can now include the role ARN when running an INSERT INTO EXTERN query. For example:
+
+```
+INSERT INTO
+EXTERN(
+ s3(bucket => 'test2', prefix => 'export', assumeRoleArn => 'arn:aws:iam::00000:role/test-20260520'))
+AS CSV
+SELECT ...
+```
+
+[#19317](https://github.com/apache/druid/pull/19317)
+
##### Other SQL-based ingestion improvements
+- Added storage counters for the amount of bytes and files written to local and durable storage as well as the state of the local `ByteTracker`. Per-worker storage counters have also been added to the web console [#19316](https://github.com/apache/druid/pull/19316)
+
#### Streaming ingestion
+##### Scaling cool down
+
+You can now configure different cool downs for scaling up and scaling down streaming task autoscalers.
Review Comment:
```suggestion
You can now configure different cool down durations for scaling up and scaling down streaming task autoscalers.
```
##########
docs/release-info/release-notes.md:
##########
@@ -57,6 +57,80 @@ For tips about how to write a good release note, see [Release notes](https://git
This section contains important information about new and existing features.
+#### Java
+
+Druid now supports Java 25. While Druid 21 is still supported, we recommend you upgrade to Java 25.
+
+Support for Java 17 has been dropped.
+
+[#19304](https://github.com/apache/druid/pull/19304) [#19336](https://github.com/apache/druid/pull/19336)
+
+#### Historical tier aliases
+
+You can now use the `historicalTierAliases` Coordinator dynamic configuration to map a virtual tier name to a set of real Historical tiers so that a group of Historical tiers has a single identifier. When a load/drop rule references the alias, the Coordinator replaces it with the actual tiers. For example, if you map the Historical tiers `hot_1` and `hot_2` to the alias `hot`, the rule `{"hot": 2}` loads 2 replicas of each onto `hot_1` and `hot_2`.
Review Comment:
```suggestion
You can map a virtual tier name to a number of real Historical tiers with the `historicalTierAliases` Coordinator dynamic configuration. This creates a single identifier for a group of Historical tiers. When a load/drop rule references the alias, the Coordinator replaces it with the actual tiers. For example, if you map the Historical tiers `hot_1` and `hot_2` to the alias `hot`, the rule `{"hot": 2}` loads 2 replicas of each onto `hot_1` and `hot_2`.
```
##########
docs/release-info/release-notes.md:
##########
@@ -65,42 +139,389 @@ This section contains detailed release notes separated by areas.
#### Other web console improvements
+- Added the following status details to the **Services** view for Historical services: cloning from another Historical, in turbo loading mode, in decommissioning mode [#19253](https://github.com/apache/druid/pull/19253)
+- Added support for resetting a supervisor to the latest offsets and backfilling [#19533](https://github.com/apache/druid/pull/19533)
+- Improved how new tabs are handled [#19483](https://github.com/apache/druid/pull/19483)
+- Improved the Home view's **Services** card. It now reports Overlord, Coordinator, Router, Broker, and Indexer counts on clusters where the web console talks to the Coordinator without SQL access [#19481](https://github.com/apache/druid/pull/19481)
+
### Ingestion
+* You can now use the expression aggregator at ingestion time for expressions that produce a LONG or DOUBLE for both fold and combine expressions [#19508](https://github.com/apache/druid/pull/19508)
+* Added `now()` expression function that returns the current system timestamp in milliseconds since epoch. Useful at ingestion time for troubleshooting pipeline delays (e.g., `now() - __time`). Note: `now()` is non-deterministic as it evaluates for every row, so it can break idempotency. This can be added to any besides `__time` [#19386](https://github.com/apache/druid/pull/19386)
+* Improved resiliency when ingesting from S3. Druid now retries on `SSLException` and transient credential errors instead of failing [#19617](https://github.com/apache/druid/pull/19617) [#19558](https://github.com/apache/druid/pull/19558)
+* Improved S3 performance [#19394](https://github.com/apache/druid/pull/19394)
+* Updated the default S3 connection pool size so that it's computed based on the number of available processors [#19536](https://github.com/apache/druid/pull/19536)
+
#### SQL-based ingestion
+##### EXTERN for S3 now supports role ARN
+
+You can now include the role ARN when running an INSERT INTO EXTERN query. For example:
+
+```
+INSERT INTO
+EXTERN(
+ s3(bucket => 'test2', prefix => 'export', assumeRoleArn => 'arn:aws:iam::00000:role/test-20260520'))
+AS CSV
+SELECT ...
+```
+
+[#19317](https://github.com/apache/druid/pull/19317)
+
##### Other SQL-based ingestion improvements
+- Added storage counters for the amount of bytes and files written to local and durable storage as well as the state of the local `ByteTracker`. Per-worker storage counters have also been added to the web console [#19316](https://github.com/apache/druid/pull/19316)
+
#### Streaming ingestion
+##### Scaling cool down
+
+You can now configure different cool downs for scaling up and scaling down streaming task autoscalers.
+
+[#19286](https://github.com/apache/druid/pull/19286)
+
+##### Improved supervisor restarts
+
+Supervisors no longer restart for all changes. Based on the type of change, one of the following can occur:
+
+- The updated spec is persisted without a restart
+- The supervisor is restarted but running tasks aren't impacted
+- The supervisor is restarted and its tasks are terminated (the default behavior prior to this change)
+
+For example, cosmetic changes to a supervisor spec no longer trigger a restart.
+
+[#19700](https://github.com/apache/druid/pull/19700) [#19720](https://github.com/apache/druid/pull/19720)
+
+Additionally, the algorithm for determining a change in the spec has been improved. For example, changes to `ioConfig.taskCount` don't trigger a supervisor restart if auto-scaling is enabled.
+
+[#19541](https://github.com/apache/druid/pull/19541)
+
+###### Latest offset and backfill
+
+For Kafka and RabbitMQ, you can now reset a supervisor to the latest offset and start a new bounded backfill supervisor to ingest data from the skipped range. This is a useful feature for operating Druid clusters where the most recent data is the most important, such as for alerting.
+
+Note the following requirements:
+
+- The supervisor's `useEarliestSequenceNumber` property must be `false`.
+- The supervisor context must have `useConcurrentLocks` set to `true` to allow the backfill supervisor's tasks to write concurrently with the main supervisor's tasks.
+- The supervisor must be in a `RUNNING` state.
+
+Use `POST` `/druid/indexer/v1/supervisor/{supervisorId}/resetToLatestAndBackfill` or the web console to perform this action.
+
+[#19477](https://github.com/apache/druid/pull/19477)
+
+##### Prunable shard specs for streaming published segments
+
+Kafka ingestion can now publish segments that the Broker prunes at query time without waiting for compaction. Set `tuningConfig.streamingPartitionsSpec.partitionDimensions` to a list of low-to-medium cardinality dimensions; each task records
+the distinct values it observes per dimension and stamps them onto a new `dim_value_set` shard spec. Queries that filter on a declared dimension then skip segments whose values can't match.
Review Comment:
```suggestion
Kafka ingestion can now publish segments that the Broker prunes at query time without waiting for compaction. Set `tuningConfig.streamingPartitionsSpec.partitionDimensions` to a list of low-to-medium cardinality dimensions; each task records the distinct values it observes per dimension and stamps them onto a new `dim_value_set` shard spec. Queries that filter on a declared dimension then skip segments whose values don't match.
```
odd line break. "values can't match" reads awkwardly
##########
docs/release-info/release-notes.md:
##########
@@ -65,42 +139,389 @@ This section contains detailed release notes separated by areas.
#### Other web console improvements
+- Added the following status details to the **Services** view for Historical services: cloning from another Historical, in turbo loading mode, in decommissioning mode [#19253](https://github.com/apache/druid/pull/19253)
+- Added support for resetting a supervisor to the latest offsets and backfilling [#19533](https://github.com/apache/druid/pull/19533)
+- Improved how new tabs are handled [#19483](https://github.com/apache/druid/pull/19483)
+- Improved the Home view's **Services** card. It now reports Overlord, Coordinator, Router, Broker, and Indexer counts on clusters where the web console talks to the Coordinator without SQL access [#19481](https://github.com/apache/druid/pull/19481)
+
### Ingestion
+* You can now use the expression aggregator at ingestion time for expressions that produce a LONG or DOUBLE for both fold and combine expressions [#19508](https://github.com/apache/druid/pull/19508)
+* Added `now()` expression function that returns the current system timestamp in milliseconds since epoch. Useful at ingestion time for troubleshooting pipeline delays (e.g., `now() - __time`). Note: `now()` is non-deterministic as it evaluates for every row, so it can break idempotency. This can be added to any besides `__time` [#19386](https://github.com/apache/druid/pull/19386)
+* Improved resiliency when ingesting from S3. Druid now retries on `SSLException` and transient credential errors instead of failing [#19617](https://github.com/apache/druid/pull/19617) [#19558](https://github.com/apache/druid/pull/19558)
+* Improved S3 performance [#19394](https://github.com/apache/druid/pull/19394)
+* Updated the default S3 connection pool size so that it's computed based on the number of available processors [#19536](https://github.com/apache/druid/pull/19536)
+
#### SQL-based ingestion
+##### EXTERN for S3 now supports role ARN
+
+You can now include the role ARN when running an INSERT INTO EXTERN query. For example:
+
+```
+INSERT INTO
+EXTERN(
+ s3(bucket => 'test2', prefix => 'export', assumeRoleArn => 'arn:aws:iam::00000:role/test-20260520'))
+AS CSV
+SELECT ...
+```
+
+[#19317](https://github.com/apache/druid/pull/19317)
+
##### Other SQL-based ingestion improvements
+- Added storage counters for the amount of bytes and files written to local and durable storage as well as the state of the local `ByteTracker`. Per-worker storage counters have also been added to the web console [#19316](https://github.com/apache/druid/pull/19316)
+
#### Streaming ingestion
+##### Scaling cool down
+
+You can now configure different cool downs for scaling up and scaling down streaming task autoscalers.
+
+[#19286](https://github.com/apache/druid/pull/19286)
+
+##### Improved supervisor restarts
+
+Supervisors no longer restart for all changes. Based on the type of change, one of the following can occur:
+
+- The updated spec is persisted without a restart
+- The supervisor is restarted but running tasks aren't impacted
+- The supervisor is restarted and its tasks are terminated (the default behavior prior to this change)
+
+For example, cosmetic changes to a supervisor spec no longer trigger a restart.
+
+[#19700](https://github.com/apache/druid/pull/19700) [#19720](https://github.com/apache/druid/pull/19720)
+
+Additionally, the algorithm for determining a change in the spec has been improved. For example, changes to `ioConfig.taskCount` don't trigger a supervisor restart if auto-scaling is enabled.
+
+[#19541](https://github.com/apache/druid/pull/19541)
+
+###### Latest offset and backfill
+
+For Kafka and RabbitMQ, you can now reset a supervisor to the latest offset and start a new bounded backfill supervisor to ingest data from the skipped range. This is a useful feature for operating Druid clusters where the most recent data is the most important, such as for alerting.
+
+Note the following requirements:
+
+- The supervisor's `useEarliestSequenceNumber` property must be `false`.
+- The supervisor context must have `useConcurrentLocks` set to `true` to allow the backfill supervisor's tasks to write concurrently with the main supervisor's tasks.
+- The supervisor must be in a `RUNNING` state.
+
+Use `POST` `/druid/indexer/v1/supervisor/{supervisorId}/resetToLatestAndBackfill` or the web console to perform this action.
+
+[#19477](https://github.com/apache/druid/pull/19477)
+
+##### Prunable shard specs for streaming published segments
+
+Kafka ingestion can now publish segments that the Broker prunes at query time without waiting for compaction. Set `tuningConfig.streamingPartitionsSpec.partitionDimensions` to a list of low-to-medium cardinality dimensions; each task records
+the distinct values it observes per dimension and stamps them onto a new `dim_value_set` shard spec. Queries that filter on a declared dimension then skip segments whose values can't match.
+
+The feature is opt-in, Kafka-only, and disabled by
+default; when unset, behavior is unchanged.
+
+`dim_value_set` is a new core shard spec type with no fallback, so it is not forward-compatible. Upgrade all services before enabling `streamingPartitionsSpec`. Once `dim_value_set` segments are published, downgrade is unsupported
+until they are compacted away or `streamingPartitionsSpec` is removed.
+
+Added `maxValuesPerDimension` (optional) to `streamingPartitionsSpec`.
+
+[#19571](https://github.com/apache/druid/pull/19571) [#19596](https://github.com/apache/druid/pull/19596)
+
##### Other streaming ingestion improvements
+- Added a property called `boundedStreamConfig` to the `SeekableStreamSupervisorIOConfig`, which allows operators to spin up a Supervisor that consumes only a specified offset range [#19372](https://github.com/apache/druid/pull/19372)
+- Improved the cost-based autoscaler for better throughput [#19646](https://github.com/apache/druid/pull/19646)
+
### Querying
+#### New query laning strategy
+
+The `weighted` query laning strategy scores queries by how many thresholds they breach (segment count, interval duration, data age, segment range) and assigns them to configurable graduated lanes with different capacity limits, providing more nuanced lane assignment than the existing binary high/low strategy.
+
+The weighted query laning strategy supports optional per-threshold cost weights:
+
+- `periodWeight`
+- `durationWeight`
+- `segmentCountWeight`
+- `segmentRangeWeight`
+
+These threshold weights default to 1.
+
+[#19225](https://github.com/apache/druid/pull/19225) [#19665](https://github.com/apache/druid/pull/19665) [#19696](https://github.com/apache/druid/pull/19696)
+
#### Other querying improvements
+- Added nullable `minTime/maxTime` Long fields to `ProjectionMetadata` [#19398](https://github.com/apache/druid/pull/19398)
+- Added `getDimensionRangeSet` support to `LikeDimFilter` for equality and prefix cases [#19524](https://github.com/apache/druid/pull/19524)
+- Added support for aggregate projections with clustered segments [#19599](https://github.com/apache/druid/pull/19599)
+- Optimized performance of aggregators for groupBy queries [#19423](https://github.com/apache/druid/pull/19423)
+
### Cluster management
+#### Improved `diskNormalized` balancer strategy
+
+The `diskNormalized` strategy is now more tunable. The primary changes are making the `utilizationThreshold` more intuitive: increasing the threshold increases the "tolerance" of the strategy while decreasing pushes nodes' disk utilization closer together.
Review Comment:
```suggestion
The `diskNormalized` strategy is now more tunable. The `utilizationThreshold` configuration works more intuitively: increasing the threshold increases the "tolerance" of the strategy while decreasing the threshold pushes nodes' disk utilization closer together.
```
##########
docs/release-info/release-notes.md:
##########
@@ -65,42 +139,389 @@ This section contains detailed release notes separated by areas.
#### Other web console improvements
+- Added the following status details to the **Services** view for Historical services: cloning from another Historical, in turbo loading mode, in decommissioning mode [#19253](https://github.com/apache/druid/pull/19253)
+- Added support for resetting a supervisor to the latest offsets and backfilling [#19533](https://github.com/apache/druid/pull/19533)
+- Improved how new tabs are handled [#19483](https://github.com/apache/druid/pull/19483)
+- Improved the Home view's **Services** card. It now reports Overlord, Coordinator, Router, Broker, and Indexer counts on clusters where the web console talks to the Coordinator without SQL access [#19481](https://github.com/apache/druid/pull/19481)
+
### Ingestion
+* You can now use the expression aggregator at ingestion time for expressions that produce a LONG or DOUBLE for both fold and combine expressions [#19508](https://github.com/apache/druid/pull/19508)
+* Added `now()` expression function that returns the current system timestamp in milliseconds since epoch. Useful at ingestion time for troubleshooting pipeline delays (e.g., `now() - __time`). Note: `now()` is non-deterministic as it evaluates for every row, so it can break idempotency. This can be added to any besides `__time` [#19386](https://github.com/apache/druid/pull/19386)
+* Improved resiliency when ingesting from S3. Druid now retries on `SSLException` and transient credential errors instead of failing [#19617](https://github.com/apache/druid/pull/19617) [#19558](https://github.com/apache/druid/pull/19558)
+* Improved S3 performance [#19394](https://github.com/apache/druid/pull/19394)
+* Updated the default S3 connection pool size so that it's computed based on the number of available processors [#19536](https://github.com/apache/druid/pull/19536)
+
#### SQL-based ingestion
+##### EXTERN for S3 now supports role ARN
+
+You can now include the role ARN when running an INSERT INTO EXTERN query. For example:
+
+```
+INSERT INTO
+EXTERN(
+ s3(bucket => 'test2', prefix => 'export', assumeRoleArn => 'arn:aws:iam::00000:role/test-20260520'))
+AS CSV
+SELECT ...
+```
+
+[#19317](https://github.com/apache/druid/pull/19317)
+
##### Other SQL-based ingestion improvements
+- Added storage counters for the amount of bytes and files written to local and durable storage as well as the state of the local `ByteTracker`. Per-worker storage counters have also been added to the web console [#19316](https://github.com/apache/druid/pull/19316)
+
#### Streaming ingestion
+##### Scaling cool down
+
+You can now configure different cool downs for scaling up and scaling down streaming task autoscalers.
+
+[#19286](https://github.com/apache/druid/pull/19286)
+
+##### Improved supervisor restarts
+
+Supervisors no longer restart for all changes. Based on the type of change, one of the following can occur:
+
+- The updated spec is persisted without a restart
+- The supervisor is restarted but running tasks aren't impacted
+- The supervisor is restarted and its tasks are terminated (the default behavior prior to this change)
+
+For example, cosmetic changes to a supervisor spec no longer trigger a restart.
+
+[#19700](https://github.com/apache/druid/pull/19700) [#19720](https://github.com/apache/druid/pull/19720)
+
+Additionally, the algorithm for determining a change in the spec has been improved. For example, changes to `ioConfig.taskCount` don't trigger a supervisor restart if auto-scaling is enabled.
+
+[#19541](https://github.com/apache/druid/pull/19541)
+
+###### Latest offset and backfill
+
+For Kafka and RabbitMQ, you can now reset a supervisor to the latest offset and start a new bounded backfill supervisor to ingest data from the skipped range. This is a useful feature for operating Druid clusters where the most recent data is the most important, such as for alerting.
+
+Note the following requirements:
+
+- The supervisor's `useEarliestSequenceNumber` property must be `false`.
+- The supervisor context must have `useConcurrentLocks` set to `true` to allow the backfill supervisor's tasks to write concurrently with the main supervisor's tasks.
+- The supervisor must be in a `RUNNING` state.
+
+Use `POST` `/druid/indexer/v1/supervisor/{supervisorId}/resetToLatestAndBackfill` or the web console to perform this action.
+
+[#19477](https://github.com/apache/druid/pull/19477)
+
+##### Prunable shard specs for streaming published segments
+
+Kafka ingestion can now publish segments that the Broker prunes at query time without waiting for compaction. Set `tuningConfig.streamingPartitionsSpec.partitionDimensions` to a list of low-to-medium cardinality dimensions; each task records
+the distinct values it observes per dimension and stamps them onto a new `dim_value_set` shard spec. Queries that filter on a declared dimension then skip segments whose values can't match.
+
+The feature is opt-in, Kafka-only, and disabled by
+default; when unset, behavior is unchanged.
+
+`dim_value_set` is a new core shard spec type with no fallback, so it is not forward-compatible. Upgrade all services before enabling `streamingPartitionsSpec`. Once `dim_value_set` segments are published, downgrade is unsupported
+until they are compacted away or `streamingPartitionsSpec` is removed.
+
+Added `maxValuesPerDimension` (optional) to `streamingPartitionsSpec`.
+
+[#19571](https://github.com/apache/druid/pull/19571) [#19596](https://github.com/apache/druid/pull/19596)
+
##### Other streaming ingestion improvements
+- Added a property called `boundedStreamConfig` to the `SeekableStreamSupervisorIOConfig`, which allows operators to spin up a Supervisor that consumes only a specified offset range [#19372](https://github.com/apache/druid/pull/19372)
+- Improved the cost-based autoscaler for better throughput [#19646](https://github.com/apache/druid/pull/19646)
+
### Querying
+#### New query laning strategy
+
+The `weighted` query laning strategy scores queries by how many thresholds they breach (segment count, interval duration, data age, segment range) and assigns them to configurable graduated lanes with different capacity limits, providing more nuanced lane assignment than the existing binary high/low strategy.
+
+The weighted query laning strategy supports optional per-threshold cost weights:
+
+- `periodWeight`
+- `durationWeight`
+- `segmentCountWeight`
+- `segmentRangeWeight`
+
+These threshold weights default to 1.
+
+[#19225](https://github.com/apache/druid/pull/19225) [#19665](https://github.com/apache/druid/pull/19665) [#19696](https://github.com/apache/druid/pull/19696)
+
#### Other querying improvements
+- Added nullable `minTime/maxTime` Long fields to `ProjectionMetadata` [#19398](https://github.com/apache/druid/pull/19398)
+- Added `getDimensionRangeSet` support to `LikeDimFilter` for equality and prefix cases [#19524](https://github.com/apache/druid/pull/19524)
+- Added support for aggregate projections with clustered segments [#19599](https://github.com/apache/druid/pull/19599)
+- Optimized performance of aggregators for groupBy queries [#19423](https://github.com/apache/druid/pull/19423)
+
### Cluster management
+#### Improved `diskNormalized` balancer strategy
+
+The `diskNormalized` strategy is now more tunable. The primary changes are making the `utilizationThreshold` more intuitive: increasing the threshold increases the "tolerance" of the strategy while decreasing pushes nodes' disk utilization closer together.
+
+[#19663](https://github.com/apache/druid/pull/19663)
+
+#### MiddleManager and Indexer restarts
+
+MiddleManagers and Indexers now persist their enabled or disabled state across restarts. If you prefer the old behavior, where the server re-enables itself after a restart, set `druid.worker.startAlwaysEnabled = true`.
+
+[#19373](https://github.com/apache/druid/pull/19373)
+
+#### Kafka idle signal
+
+Improved the cost-based auto scaler for Kafka. The `poll-idle ratio` only reflected the time spent polling, whether there is spare processing capacity. You can now configure the autoscaler to use a utilization ratio instead:
+
+```
+1 - (avgProcessingRate / maxObservedRate)
+```
+
+Set `useUtilizationRatio` to `true` to use this new ratio for autoscaling.
+
+[#19622](https://github.com/apache/druid/pull/19622)
+
+
#### Other cluster management improvements
+- Added `datasource` filter pushdown to `sys.segments` table [#19718](https://github.com/apache/druid/pull/19718) [#19731](https://github.com/apache/druid/pull/19731)
+- Added a `restarted` boolean field to the supervisor POST endpoint response to indicate whether the supervisor was actually restarted [#19349](https://github.com/apache/druid/pull/19349)
+- Added `error_message` column to `sys.server_properties` table and made the table resilient to unreachable servers. Previously, the entire query would fail if any server was unreachable; now a row is returned with `error_message` populated. The table also now supports filter and projection pushdown [#19459](https://github.com/apache/druid/pull/19459)
+- Added `druid.expressions.useVectorApi` config to support the incubating JDK Vector API. To use the API, set the config to `true` and start Druid with the `--add-modules=jdk.incubator.vector` flag [#19512](https://github.com/apache/druid/pull/19512)
+- Added debug logging at the INFO level for projections if the debug flag is set [#19613](https://github.com/apache/druid/pull/19613)
+- Changed MSQ task engine logging. It now logs the full stack trace when `debug` is set in the context [#19361](https://github.com/apache/druid/pull/19361)
+- Improved the cost-based autoscaler so that it scales down over-provisioned supervisors running above the ideal idle ratio with low lag [#19562](https://github.com/apache/druid/pull/19562)
+- Improved how Druid handles Java. MiddleManagers now honor `JAVA_HOME` [#19709](https://github.com/apache/druid/pull/19709)
+
### Data management
#### Other data management improvements
+* Improved how compaction supervisor specs start up. They no longer attempt to create tasks with invalid configs [#19223](https://github.com/apache/druid/pull/19223)
+* Changed Historical tiers so that they can only be associated with one tier alias [#19595](https://github.com/apache/druid/pull/19595)
+* Sped up segment metadata cache syncs [#19672](https://github.com/apache/druid/pull/19672)
+
### Metrics and monitoring
+#### New metadata cache metrics
+
+Added the following metrics for the segment metadata cache:
+
+- `segment/metadataCache/unused/count`
+- `segment/metadataCache/fetchIds/time`
+- `segment/metadataCache/fetchPayloads/time`
+- `segment/metadataCache/fetchPending/time`
+- `segment/metadataCache/fetchSchemas/time`
+- `segment/metadataCache/fetchIndexingStates/time`
+- `segment/metadataCache/updateIds/time`
+- `segment/metadataCache/updateSnapshot/time`
+- `segment/metadataCache/schema/skipped`
+- `segment/metadataCache/indexingState/added`
+- `segment/metadataCache/indexingState/deleted`
+
+[#19672](https://github.com/apache/druid/pull/19672)
+
+#### Changed metrics for cost-based autoscaling
+
+Removed the following metrics:
+
+- `task/autoScaler/costBased/lagCost`
+- `task/autoScaler/costBased/idleCost`
+
+Added the following metrics:
+
+- `task/autoScaler/costBased/avgProcessingRate`
+- `task/autoScaler/costBased/avgPollIdleRatio`
+- `task/autoScaler/costBased/lagWeight`
+- `task/autoScaler/costBased/costWeight`
+
+[#19631](https://github.com/apache/druid/pull/19631)
+
+#### Auth metrics
+
+You can now configure Druid to emit metrics for authorization events.
+
+The `auth/forbidden` and `auth/exception` metrics have the following dimensions to support precise alerting on security events:
+
+- `identity`
+- `authorizerName`
+- `resourceName`
+- `resourceType`
+- `action`
+- `errorMessage` where applicable
+
+
+To enable this functionality, set `druid.auth.emitAuthMetrics` to `true`.
+
+[#19552](https://github.com/apache/druid/pull/19552)
+
+#### Storage metrics
+
+The `storage/load/bytes` and `storage/virtual/load/bytes` metrics now measure once the load is complete. Previously, they measured when the load starts.
Review Comment:
"they measured when the load starts" mix of past/present reads awkwardly. Consider revising
##########
docs/release-info/release-notes.md:
##########
@@ -57,6 +57,80 @@ For tips about how to write a good release note, see [Release notes](https://git
This section contains important information about new and existing features.
+#### Java
+
+Druid now supports Java 25. While Druid 21 is still supported, we recommend you upgrade to Java 25.
+
+Support for Java 17 has been dropped.
+
+[#19304](https://github.com/apache/druid/pull/19304) [#19336](https://github.com/apache/druid/pull/19336)
+
+#### Historical tier aliases
+
+You can now use the `historicalTierAliases` Coordinator dynamic configuration to map a virtual tier name to a set of real Historical tiers so that a group of Historical tiers has a single identifier. When a load/drop rule references the alias, the Coordinator replaces it with the actual tiers. For example, if you map the Historical tiers `hot_1` and `hot_2` to the alias `hot`, the rule `{"hot": 2}` loads 2 replicas of each onto `hot_1` and `hot_2`.
+
+[#19204](https://github.com/apache/druid/pull/19204) [#19667](https://github.com/apache/druid/pull/19667)
+
+#### New load rule types
+
+Adds a new family of retention rules, `loadPartialByPeriod`, `loadPartialByInterval`, `loadPartialForever`, laying the groundwork for partial loading of version 10 segment projections on Historicals.
+
+[#19374](https://github.com/apache/druid/pull/19374)
+
+#### Realtime segments query context
+
+The `realtimeSegmentsOnly` query context parameter has been deprecated and replaced with `realtimeSegmentsMode`.
+
+You can set `realtimeSegmentsMode` to one of the following:
+
+- `include` (default): query all segments, including realtime segments
+- `exclude`: skip realtime segments for the query
+- `exclusive`: query only realtime segments
+
+This is useful when performing things like blue/green deployments and you only want to query new Historical replica autoscaling groups and not touch any "live" nodes (neither realtime nor historical).
+
+[#19486](https://github.com/apache/druid/pull/19486)
+
+#### Faster segment metadata search
+
+You can set `druid.segment.timeline.fastIntervalSearch` to `true` so that Druid uses an index based on interval trees to store that metadata in memory for faster identification and retrieval.
+
+This feature is off by default.
+
+[#19138](https://github.com/apache/druid/issues/19138) [#19850](https://github.com/apache/druid/issues/19850)
+
+#### Fetching files in the background
+
+The MSQ task engine now supports workers fetching input files from cloud storage asynchronously. This overlaps downloading with processing, which generally improves throughput when reading larger numbers of files. Otherwise, workers stream each file directly from cloud storage while processing it.
+
+This feature is on by default and is controlled by the `backgroundFetchExternalFiles` setting.
+
+[#19539](https://github.com/apache/druid/pull/19539)
+
+#### Segment prefetching for Dart
+
+Dart now supports the runtime property `druid.msq.dart.worker.segmentLoadAheadCount`, which controls the number of segments that Dart prefetches. If configured to be greater than 0 for a worker, this becomes the default `segmentLoadAheadCount` value for the worker. If a query includes the `segmentLoadAheadCount` query context parameter, the query context takes precedence.
Review Comment:
```suggestion
Dart now supports the runtime property `druid.msq.dart.worker.segmentLoadAheadCount`, which controls the number of segments that Dart prefetches. If set greater than 0 for a worker, this setting becomes the default `segmentLoadAheadCount` value for the worker. If a query includes the `segmentLoadAheadCount` query context parameter, the query context takes precedence.
```
##########
docs/release-info/release-notes.md:
##########
@@ -65,42 +139,389 @@ This section contains detailed release notes separated by areas.
#### Other web console improvements
+- Added the following status details to the **Services** view for Historical services: cloning from another Historical, in turbo loading mode, in decommissioning mode [#19253](https://github.com/apache/druid/pull/19253)
+- Added support for resetting a supervisor to the latest offsets and backfilling [#19533](https://github.com/apache/druid/pull/19533)
+- Improved how new tabs are handled [#19483](https://github.com/apache/druid/pull/19483)
+- Improved the Home view's **Services** card. It now reports Overlord, Coordinator, Router, Broker, and Indexer counts on clusters where the web console talks to the Coordinator without SQL access [#19481](https://github.com/apache/druid/pull/19481)
+
### Ingestion
+* You can now use the expression aggregator at ingestion time for expressions that produce a LONG or DOUBLE for both fold and combine expressions [#19508](https://github.com/apache/druid/pull/19508)
+* Added `now()` expression function that returns the current system timestamp in milliseconds since epoch. Useful at ingestion time for troubleshooting pipeline delays (e.g., `now() - __time`). Note: `now()` is non-deterministic as it evaluates for every row, so it can break idempotency. This can be added to any besides `__time` [#19386](https://github.com/apache/druid/pull/19386)
Review Comment:
missing word
```suggestion
* Added `now()` expression function that returns the current system timestamp in milliseconds since epoch. Useful at ingestion time for troubleshooting pipeline delays (e.g., `now() - __time`). Note: `now()` is non-deterministic as it evaluates for every row, so it can break idempotency. This can be added to any column besides `__time` [#19386](https://github.com/apache/druid/pull/19386)
```
##########
docs/release-info/release-notes.md:
##########
@@ -65,42 +139,389 @@ This section contains detailed release notes separated by areas.
#### Other web console improvements
+- Added the following status details to the **Services** view for Historical services: cloning from another Historical, in turbo loading mode, in decommissioning mode [#19253](https://github.com/apache/druid/pull/19253)
+- Added support for resetting a supervisor to the latest offsets and backfilling [#19533](https://github.com/apache/druid/pull/19533)
+- Improved how new tabs are handled [#19483](https://github.com/apache/druid/pull/19483)
+- Improved the Home view's **Services** card. It now reports Overlord, Coordinator, Router, Broker, and Indexer counts on clusters where the web console talks to the Coordinator without SQL access [#19481](https://github.com/apache/druid/pull/19481)
+
### Ingestion
+* You can now use the expression aggregator at ingestion time for expressions that produce a LONG or DOUBLE for both fold and combine expressions [#19508](https://github.com/apache/druid/pull/19508)
+* Added `now()` expression function that returns the current system timestamp in milliseconds since epoch. Useful at ingestion time for troubleshooting pipeline delays (e.g., `now() - __time`). Note: `now()` is non-deterministic as it evaluates for every row, so it can break idempotency. This can be added to any besides `__time` [#19386](https://github.com/apache/druid/pull/19386)
+* Improved resiliency when ingesting from S3. Druid now retries on `SSLException` and transient credential errors instead of failing [#19617](https://github.com/apache/druid/pull/19617) [#19558](https://github.com/apache/druid/pull/19558)
+* Improved S3 performance [#19394](https://github.com/apache/druid/pull/19394)
+* Updated the default S3 connection pool size so that it's computed based on the number of available processors [#19536](https://github.com/apache/druid/pull/19536)
+
#### SQL-based ingestion
+##### EXTERN for S3 now supports role ARN
+
+You can now include the role ARN when running an INSERT INTO EXTERN query. For example:
+
+```
+INSERT INTO
+EXTERN(
+ s3(bucket => 'test2', prefix => 'export', assumeRoleArn => 'arn:aws:iam::00000:role/test-20260520'))
+AS CSV
+SELECT ...
+```
+
+[#19317](https://github.com/apache/druid/pull/19317)
+
##### Other SQL-based ingestion improvements
+- Added storage counters for the amount of bytes and files written to local and durable storage as well as the state of the local `ByteTracker`. Per-worker storage counters have also been added to the web console [#19316](https://github.com/apache/druid/pull/19316)
+
#### Streaming ingestion
+##### Scaling cool down
+
+You can now configure different cool downs for scaling up and scaling down streaming task autoscalers.
+
+[#19286](https://github.com/apache/druid/pull/19286)
+
+##### Improved supervisor restarts
+
+Supervisors no longer restart for all changes. Based on the type of change, one of the following can occur:
+
+- The updated spec is persisted without a restart
+- The supervisor is restarted but running tasks aren't impacted
+- The supervisor is restarted and its tasks are terminated (the default behavior prior to this change)
Review Comment:
```suggestion
- Druid persists the updated spec without a restart.
- Druid restarts the supervisor, but running tasks aren't impacted.
- Druid restarts the supervisor is restarted and its tasks are terminated, which was the default behavior before this change.
```
##########
docs/release-info/release-notes.md:
##########
@@ -65,42 +139,389 @@ This section contains detailed release notes separated by areas.
#### Other web console improvements
+- Added the following status details to the **Services** view for Historical services: cloning from another Historical, in turbo loading mode, in decommissioning mode [#19253](https://github.com/apache/druid/pull/19253)
+- Added support for resetting a supervisor to the latest offsets and backfilling [#19533](https://github.com/apache/druid/pull/19533)
+- Improved how new tabs are handled [#19483](https://github.com/apache/druid/pull/19483)
+- Improved the Home view's **Services** card. It now reports Overlord, Coordinator, Router, Broker, and Indexer counts on clusters where the web console talks to the Coordinator without SQL access [#19481](https://github.com/apache/druid/pull/19481)
+
### Ingestion
+* You can now use the expression aggregator at ingestion time for expressions that produce a LONG or DOUBLE for both fold and combine expressions [#19508](https://github.com/apache/druid/pull/19508)
+* Added `now()` expression function that returns the current system timestamp in milliseconds since epoch. Useful at ingestion time for troubleshooting pipeline delays (e.g., `now() - __time`). Note: `now()` is non-deterministic as it evaluates for every row, so it can break idempotency. This can be added to any besides `__time` [#19386](https://github.com/apache/druid/pull/19386)
+* Improved resiliency when ingesting from S3. Druid now retries on `SSLException` and transient credential errors instead of failing [#19617](https://github.com/apache/druid/pull/19617) [#19558](https://github.com/apache/druid/pull/19558)
+* Improved S3 performance [#19394](https://github.com/apache/druid/pull/19394)
+* Updated the default S3 connection pool size so that it's computed based on the number of available processors [#19536](https://github.com/apache/druid/pull/19536)
+
#### SQL-based ingestion
+##### EXTERN for S3 now supports role ARN
+
+You can now include the role ARN when running an INSERT INTO EXTERN query. For example:
+
+```
+INSERT INTO
+EXTERN(
+ s3(bucket => 'test2', prefix => 'export', assumeRoleArn => 'arn:aws:iam::00000:role/test-20260520'))
+AS CSV
+SELECT ...
+```
+
+[#19317](https://github.com/apache/druid/pull/19317)
+
##### Other SQL-based ingestion improvements
+- Added storage counters for the amount of bytes and files written to local and durable storage as well as the state of the local `ByteTracker`. Per-worker storage counters have also been added to the web console [#19316](https://github.com/apache/druid/pull/19316)
+
#### Streaming ingestion
+##### Scaling cool down
+
+You can now configure different cool downs for scaling up and scaling down streaming task autoscalers.
+
+[#19286](https://github.com/apache/druid/pull/19286)
+
+##### Improved supervisor restarts
+
+Supervisors no longer restart for all changes. Based on the type of change, one of the following can occur:
+
+- The updated spec is persisted without a restart
+- The supervisor is restarted but running tasks aren't impacted
+- The supervisor is restarted and its tasks are terminated (the default behavior prior to this change)
+
+For example, cosmetic changes to a supervisor spec no longer trigger a restart.
+
+[#19700](https://github.com/apache/druid/pull/19700) [#19720](https://github.com/apache/druid/pull/19720)
+
+Additionally, the algorithm for determining a change in the spec has been improved. For example, changes to `ioConfig.taskCount` don't trigger a supervisor restart if auto-scaling is enabled.
+
+[#19541](https://github.com/apache/druid/pull/19541)
+
+###### Latest offset and backfill
+
+For Kafka and RabbitMQ, you can now reset a supervisor to the latest offset and start a new bounded backfill supervisor to ingest data from the skipped range. This is a useful feature for operating Druid clusters where the most recent data is the most important, such as for alerting.
Review Comment:
```suggestion
For Kafka and RabbitMQ, you can now reset a supervisor to the latest offset and start a new bounded backfill supervisor to ingest data from the skipped range. This is a useful feature for operating Druid clusters where the most recent data is the most important, like alerting scenarios.
```
##########
docs/release-info/release-notes.md:
##########
@@ -65,42 +139,389 @@ This section contains detailed release notes separated by areas.
#### Other web console improvements
+- Added the following status details to the **Services** view for Historical services: cloning from another Historical, in turbo loading mode, in decommissioning mode [#19253](https://github.com/apache/druid/pull/19253)
+- Added support for resetting a supervisor to the latest offsets and backfilling [#19533](https://github.com/apache/druid/pull/19533)
+- Improved how new tabs are handled [#19483](https://github.com/apache/druid/pull/19483)
+- Improved the Home view's **Services** card. It now reports Overlord, Coordinator, Router, Broker, and Indexer counts on clusters where the web console talks to the Coordinator without SQL access [#19481](https://github.com/apache/druid/pull/19481)
+
### Ingestion
+* You can now use the expression aggregator at ingestion time for expressions that produce a LONG or DOUBLE for both fold and combine expressions [#19508](https://github.com/apache/druid/pull/19508)
+* Added `now()` expression function that returns the current system timestamp in milliseconds since epoch. Useful at ingestion time for troubleshooting pipeline delays (e.g., `now() - __time`). Note: `now()` is non-deterministic as it evaluates for every row, so it can break idempotency. This can be added to any besides `__time` [#19386](https://github.com/apache/druid/pull/19386)
+* Improved resiliency when ingesting from S3. Druid now retries on `SSLException` and transient credential errors instead of failing [#19617](https://github.com/apache/druid/pull/19617) [#19558](https://github.com/apache/druid/pull/19558)
+* Improved S3 performance [#19394](https://github.com/apache/druid/pull/19394)
+* Updated the default S3 connection pool size so that it's computed based on the number of available processors [#19536](https://github.com/apache/druid/pull/19536)
+
#### SQL-based ingestion
+##### EXTERN for S3 now supports role ARN
+
+You can now include the role ARN when running an INSERT INTO EXTERN query. For example:
+
+```
+INSERT INTO
+EXTERN(
+ s3(bucket => 'test2', prefix => 'export', assumeRoleArn => 'arn:aws:iam::00000:role/test-20260520'))
+AS CSV
+SELECT ...
+```
+
+[#19317](https://github.com/apache/druid/pull/19317)
+
##### Other SQL-based ingestion improvements
+- Added storage counters for the amount of bytes and files written to local and durable storage as well as the state of the local `ByteTracker`. Per-worker storage counters have also been added to the web console [#19316](https://github.com/apache/druid/pull/19316)
+
#### Streaming ingestion
+##### Scaling cool down
+
+You can now configure different cool downs for scaling up and scaling down streaming task autoscalers.
+
+[#19286](https://github.com/apache/druid/pull/19286)
+
+##### Improved supervisor restarts
+
+Supervisors no longer restart for all changes. Based on the type of change, one of the following can occur:
+
+- The updated spec is persisted without a restart
+- The supervisor is restarted but running tasks aren't impacted
+- The supervisor is restarted and its tasks are terminated (the default behavior prior to this change)
+
+For example, cosmetic changes to a supervisor spec no longer trigger a restart.
+
+[#19700](https://github.com/apache/druid/pull/19700) [#19720](https://github.com/apache/druid/pull/19720)
+
+Additionally, the algorithm for determining a change in the spec has been improved. For example, changes to `ioConfig.taskCount` don't trigger a supervisor restart if auto-scaling is enabled.
+
+[#19541](https://github.com/apache/druid/pull/19541)
+
+###### Latest offset and backfill
+
+For Kafka and RabbitMQ, you can now reset a supervisor to the latest offset and start a new bounded backfill supervisor to ingest data from the skipped range. This is a useful feature for operating Druid clusters where the most recent data is the most important, such as for alerting.
+
+Note the following requirements:
+
+- The supervisor's `useEarliestSequenceNumber` property must be `false`.
+- The supervisor context must have `useConcurrentLocks` set to `true` to allow the backfill supervisor's tasks to write concurrently with the main supervisor's tasks.
+- The supervisor must be in a `RUNNING` state.
+
+Use `POST` `/druid/indexer/v1/supervisor/{supervisorId}/resetToLatestAndBackfill` or the web console to perform this action.
+
+[#19477](https://github.com/apache/druid/pull/19477)
+
+##### Prunable shard specs for streaming published segments
+
+Kafka ingestion can now publish segments that the Broker prunes at query time without waiting for compaction. Set `tuningConfig.streamingPartitionsSpec.partitionDimensions` to a list of low-to-medium cardinality dimensions; each task records
+the distinct values it observes per dimension and stamps them onto a new `dim_value_set` shard spec. Queries that filter on a declared dimension then skip segments whose values can't match.
+
+The feature is opt-in, Kafka-only, and disabled by
+default; when unset, behavior is unchanged.
+
+`dim_value_set` is a new core shard spec type with no fallback, so it is not forward-compatible. Upgrade all services before enabling `streamingPartitionsSpec`. Once `dim_value_set` segments are published, downgrade is unsupported
+until they are compacted away or `streamingPartitionsSpec` is removed.
+
+Added `maxValuesPerDimension` (optional) to `streamingPartitionsSpec`.
+
+[#19571](https://github.com/apache/druid/pull/19571) [#19596](https://github.com/apache/druid/pull/19596)
+
##### Other streaming ingestion improvements
+- Added a property called `boundedStreamConfig` to the `SeekableStreamSupervisorIOConfig`, which allows operators to spin up a Supervisor that consumes only a specified offset range [#19372](https://github.com/apache/druid/pull/19372)
+- Improved the cost-based autoscaler for better throughput [#19646](https://github.com/apache/druid/pull/19646)
+
### Querying
+#### New query laning strategy
+
+The `weighted` query laning strategy scores queries by how many thresholds they breach (segment count, interval duration, data age, segment range) and assigns them to configurable graduated lanes with different capacity limits, providing more nuanced lane assignment than the existing binary high/low strategy.
+
+The weighted query laning strategy supports optional per-threshold cost weights:
+
+- `periodWeight`
+- `durationWeight`
+- `segmentCountWeight`
+- `segmentRangeWeight`
+
+These threshold weights default to 1.
+
+[#19225](https://github.com/apache/druid/pull/19225) [#19665](https://github.com/apache/druid/pull/19665) [#19696](https://github.com/apache/druid/pull/19696)
+
#### Other querying improvements
+- Added nullable `minTime/maxTime` Long fields to `ProjectionMetadata` [#19398](https://github.com/apache/druid/pull/19398)
+- Added `getDimensionRangeSet` support to `LikeDimFilter` for equality and prefix cases [#19524](https://github.com/apache/druid/pull/19524)
+- Added support for aggregate projections with clustered segments [#19599](https://github.com/apache/druid/pull/19599)
+- Optimized performance of aggregators for groupBy queries [#19423](https://github.com/apache/druid/pull/19423)
+
### Cluster management
+#### Improved `diskNormalized` balancer strategy
+
+The `diskNormalized` strategy is now more tunable. The primary changes are making the `utilizationThreshold` more intuitive: increasing the threshold increases the "tolerance" of the strategy while decreasing pushes nodes' disk utilization closer together.
+
+[#19663](https://github.com/apache/druid/pull/19663)
+
+#### MiddleManager and Indexer restarts
+
+MiddleManagers and Indexers now persist their enabled or disabled state across restarts. If you prefer the old behavior, where the server re-enables itself after a restart, set `druid.worker.startAlwaysEnabled = true`.
+
+[#19373](https://github.com/apache/druid/pull/19373)
+
+#### Kafka idle signal
+
+Improved the cost-based auto scaler for Kafka. The `poll-idle ratio` only reflected the time spent polling, whether there is spare processing capacity. You can now configure the autoscaler to use a utilization ratio instead:
+
+```
+1 - (avgProcessingRate / maxObservedRate)
+```
+
+Set `useUtilizationRatio` to `true` to use this new ratio for autoscaling.
+
+[#19622](https://github.com/apache/druid/pull/19622)
+
+
#### Other cluster management improvements
+- Added `datasource` filter pushdown to `sys.segments` table [#19718](https://github.com/apache/druid/pull/19718) [#19731](https://github.com/apache/druid/pull/19731)
+- Added a `restarted` boolean field to the supervisor POST endpoint response to indicate whether the supervisor was actually restarted [#19349](https://github.com/apache/druid/pull/19349)
+- Added `error_message` column to `sys.server_properties` table and made the table resilient to unreachable servers. Previously, the entire query would fail if any server was unreachable; now a row is returned with `error_message` populated. The table also now supports filter and projection pushdown [#19459](https://github.com/apache/druid/pull/19459)
+- Added `druid.expressions.useVectorApi` config to support the incubating JDK Vector API. To use the API, set the config to `true` and start Druid with the `--add-modules=jdk.incubator.vector` flag [#19512](https://github.com/apache/druid/pull/19512)
+- Added debug logging at the INFO level for projections if the debug flag is set [#19613](https://github.com/apache/druid/pull/19613)
+- Changed MSQ task engine logging. It now logs the full stack trace when `debug` is set in the context [#19361](https://github.com/apache/druid/pull/19361)
Review Comment:
```suggestion
- The MSQ task engine now logs the full stack trace when `debug` is set in the context [#19361](https://github.com/apache/druid/pull/19361)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] docs: 38.0.0 release notes (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz closed pull request #19945: docs: 38.0.0 release notes
URL: https://github.com/apache/druid/pull/19945
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] docs: 38.0.0 release notes (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #19945:
URL: https://github.com/apache/druid/pull/19945#discussion_r3998699892
##########
docs/release-info/release-notes.md:
##########
@@ -57,6 +57,80 @@ For tips about how to write a good release note, see [Release notes](https://git
This section contains important information about new and existing features.
+#### Java
+
+Druid now supports Java 25. While Java 21 is still supported, we recommend you upgrade to Java 25.
+
+Support for Java 17 has been dropped.
+
+[#19304](https://github.com/apache/druid/pull/19304) [#19336](https://github.com/apache/druid/pull/19336)
+
+#### Historical tier aliases
+
+You can map a virtual tier name to a number of real Historical tiers with the `historicalTierAliases` Coordinator dynamic configuration. This creates a single identifier for a group of Historical tiers. When a load/drop rule references the alias, the Coordinator replaces it with the actual tiers. For example, if you map the Historical tiers `hot_1` and `hot_2` to the alias `hot`, the rule `{"hot": 2}` loads 2 replicas of each onto `hot_1` and `hot_2`.
+
+[#19204](https://github.com/apache/druid/pull/19204) [#19667](https://github.com/apache/druid/pull/19667)
+
+#### New load rule types
+
+Adds a new family of retention rules, `loadPartialByPeriod`, `loadPartialByInterval`, `loadPartialForever`, laying the groundwork for partial loading of version 10 segment projections on Historicals.
+
+[#19374](https://github.com/apache/druid/pull/19374)
+
+#### Realtime segments mode query context
+
+The `realtimeSegmentsOnly` query context parameter has been deprecated and replaced with `realtimeSegmentsMode`.
+
+You can set `realtimeSegmentsMode` to one of the following:
+
+- `include` (default): query all segments, including realtime segments
+- `exclude`: skip realtime segments for the query
+- `exclusive`: query only realtime segments
+
+This is useful when performing things like blue/green deployments and you only want to query new Historical replica autoscaling groups and not touch any "live" nodes (neither realtime nor historical).
+
+[#19486](https://github.com/apache/druid/pull/19486)
+
+#### Faster segment metadata search
+
+You can set `druid.segment.timeline.fastIntervalSearch` to `true` so that Druid uses an index based on interval trees to store that metadata in memory for faster identification and retrieval.
+
+This feature is off by default.
+
+[#19138](https://github.com/apache/druid/issues/19138) [#19850](https://github.com/apache/druid/issues/19850)
+
+#### Fetching files in the background
+
+The MSQ task engine now supports workers fetching input files from cloud storage asynchronously. This overlaps downloading with processing, which generally improves throughput when reading larger numbers of files. Otherwise, workers stream each file directly from cloud storage while processing it.
+
+This feature is on by default and is controlled by the `backgroundFetchExternalFiles` setting.
+
+[#19539](https://github.com/apache/druid/pull/19539)
+
+#### Segment prefetching for Dart
+
+Dart now supports the runtime property `druid.msq.dart.worker.segmentLoadAheadCount`, which controls the number of segments that Dart prefetches. If set greater than 0 for a worker, this setting becomes the default `segmentLoadAheadCount` value for the worker. If a query includes the `segmentLoadAheadCount` query context parameter, the query context takes precedence.
+
+[#19559](https://github.com/apache/druid/pull/19559)
+
+#### Partial segment loading
Review Comment:
Shouldn't this section have a description?
##########
docs/release-info/release-notes.md:
##########
@@ -57,6 +57,80 @@ For tips about how to write a good release note, see [Release notes](https://git
This section contains important information about new and existing features.
+#### Java
+
+Druid now supports Java 25. While Java 21 is still supported, we recommend you upgrade to Java 25.
+
+Support for Java 17 has been dropped.
+
+[#19304](https://github.com/apache/druid/pull/19304) [#19336](https://github.com/apache/druid/pull/19336)
+
+#### Historical tier aliases
+
+You can map a virtual tier name to a number of real Historical tiers with the `historicalTierAliases` Coordinator dynamic configuration. This creates a single identifier for a group of Historical tiers. When a load/drop rule references the alias, the Coordinator replaces it with the actual tiers. For example, if you map the Historical tiers `hot_1` and `hot_2` to the alias `hot`, the rule `{"hot": 2}` loads 2 replicas of each onto `hot_1` and `hot_2`.
+
+[#19204](https://github.com/apache/druid/pull/19204) [#19667](https://github.com/apache/druid/pull/19667)
+
+#### New load rule types
+
+Adds a new family of retention rules, `loadPartialByPeriod`, `loadPartialByInterval`, `loadPartialForever`, laying the groundwork for partial loading of version 10 segment projections on Historicals.
+
+[#19374](https://github.com/apache/druid/pull/19374)
+
+#### Realtime segments mode query context
+
+The `realtimeSegmentsOnly` query context parameter has been deprecated and replaced with `realtimeSegmentsMode`.
+
+You can set `realtimeSegmentsMode` to one of the following:
+
+- `include` (default): query all segments, including realtime segments
+- `exclude`: skip realtime segments for the query
+- `exclusive`: query only realtime segments
+
+This is useful when performing things like blue/green deployments and you only want to query new Historical replica autoscaling groups and not touch any "live" nodes (neither realtime nor historical).
+
+[#19486](https://github.com/apache/druid/pull/19486)
+
+#### Faster segment metadata search
+
+You can set `druid.segment.timeline.fastIntervalSearch` to `true` so that Druid uses an index based on interval trees to store that metadata in memory for faster identification and retrieval.
+
+This feature is off by default.
+
+[#19138](https://github.com/apache/druid/issues/19138) [#19850](https://github.com/apache/druid/issues/19850)
+
+#### Fetching files in the background
+
+The MSQ task engine now supports workers fetching input files from cloud storage asynchronously. This overlaps downloading with processing, which generally improves throughput when reading larger numbers of files. Otherwise, workers stream each file directly from cloud storage while processing it.
+
+This feature is on by default and is controlled by the `backgroundFetchExternalFiles` setting.
+
+[#19539](https://github.com/apache/druid/pull/19539)
+
+#### Segment prefetching for Dart
+
+Dart now supports the runtime property `druid.msq.dart.worker.segmentLoadAheadCount`, which controls the number of segments that Dart prefetches. If set greater than 0 for a worker, this setting becomes the default `segmentLoadAheadCount` value for the worker. If a query includes the `segmentLoadAheadCount` query context parameter, the query context takes precedence.
+
+[#19559](https://github.com/apache/druid/pull/19559)
+
+#### Partial segment loading
+
+[#19620](https://github.com/apache/druid/pull/19620)
+[#19535](https://github.com/apache/druid/pull/19535)
+
+#### Clustered segments
Review Comment:
Description missing?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] docs: 38.0.0 release notes (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #19945:
URL: https://github.com/apache/druid/pull/19945#discussion_r4155055365
##########
docs/release-info/release-notes.md:
##########
@@ -57,6 +57,76 @@ For tips about how to write a good release note, see [Release notes](https://git
This section contains important information about new and existing features.
+#### Java
+
+Druid now supports Java 25. While Java 21 is still supported, we recommend you upgrade to Java 25.
+
+Support for Java 17 has been dropped.
+
+[#19304](https://github.com/apache/druid/pull/19304) [#19336](https://github.com/apache/druid/pull/19336)
+
+#### Historical tier aliases
+
+You can map a virtual tier name to a number of real Historical tiers with the `historicalTierAliases` Coordinator dynamic configuration. This creates a single identifier for a group of Historical tiers. When a load/drop rule references the alias, the Coordinator replaces it with the actual tiers. For example, if you map the Historical tiers `hot_1` and `hot_2` to the alias `hot`, the rule `{"hot": 2}` loads 2 replicas of each onto `hot_1` and `hot_2`.
+
+[#19204](https://github.com/apache/druid/pull/19204) [#19667](https://github.com/apache/druid/pull/19667)
+
+#### New load rule types
+
+Adds a new family of retention rules, `loadPartialByPeriod`, `loadPartialByInterval`, `loadPartialForever`, laying the groundwork for partial loading of version 10 segment projections on Historicals.
+
+[#19374](https://github.com/apache/druid/pull/19374)
+
+#### Realtime segments mode query context
+
+The `realtimeSegmentsOnly` query context parameter has been deprecated and replaced with `realtimeSegmentsMode`.
+
+You can set `realtimeSegmentsMode` to one of the following:
+
+- `include` (default): query all segments, including realtime segments
+- `exclude`: skip realtime segments for the query
+- `exclusive`: query only realtime segments
+
+This is useful when performing things like blue/green deployments and you only want to query new Historical replica autoscaling groups and not touch any "live" nodes (neither realtime nor historical).
+
+[#19486](https://github.com/apache/druid/pull/19486)
+
+#### Faster segment metadata search
+
+You can set `druid.segment.timeline.fastIntervalSearch` to `true` so that Druid uses an index based on interval trees to store that metadata in memory for faster identification and retrieval.
+
+This feature is off by default.
+
+[#19138](https://github.com/apache/druid/issues/19138) [#19850](https://github.com/apache/druid/issues/19850)
+
+#### Fetching files in the background
+
+The MSQ task engine now supports workers fetching input files from cloud storage asynchronously. This overlaps downloading with processing, which generally improves throughput when reading larger numbers of files. Otherwise, workers stream each file directly from cloud storage while processing it.
+
+This feature is on by default and is controlled by the `backgroundFetchExternalFiles` setting.
+
+[#19539](https://github.com/apache/druid/pull/19539)
+
+#### Segment prefetching for Dart
+
+Dart now supports the runtime property `druid.msq.dart.worker.segmentLoadAheadCount`, which controls the number of segments that Dart prefetches. If set greater than 0 for a worker, this setting becomes the default `segmentLoadAheadCount` value for the worker. If a query includes the `segmentLoadAheadCount` query context parameter, the query context takes precedence.
Review Comment:
```suggestion
Historicals with virtual storage enabled now support the runtime property `druid.msq.dart.worker.segmentLoadAheadCount`, which controls the number of segments that Dart prefetches. If set greater than 0 for a worker, this setting becomes the default `segmentLoadAheadCount` value for the worker. If a query includes the `segmentLoadAheadCount` query context parameter, the query context takes precedence.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] docs: 38.0.0 release notes (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #19945:
URL: https://github.com/apache/druid/pull/19945#discussion_r4153175375
##########
docs/release-info/release-notes.md:
##########
@@ -57,6 +57,80 @@ For tips about how to write a good release note, see [Release notes](https://git
This section contains important information about new and existing features.
+#### Java
+
+Druid now supports Java 25. While Java 21 is still supported, we recommend you upgrade to Java 25.
+
+Support for Java 17 has been dropped.
+
+[#19304](https://github.com/apache/druid/pull/19304) [#19336](https://github.com/apache/druid/pull/19336)
+
+#### Historical tier aliases
+
+You can map a virtual tier name to a number of real Historical tiers with the `historicalTierAliases` Coordinator dynamic configuration. This creates a single identifier for a group of Historical tiers. When a load/drop rule references the alias, the Coordinator replaces it with the actual tiers. For example, if you map the Historical tiers `hot_1` and `hot_2` to the alias `hot`, the rule `{"hot": 2}` loads 2 replicas of each onto `hot_1` and `hot_2`.
+
+[#19204](https://github.com/apache/druid/pull/19204) [#19667](https://github.com/apache/druid/pull/19667)
+
+#### New load rule types
+
+Adds a new family of retention rules, `loadPartialByPeriod`, `loadPartialByInterval`, `loadPartialForever`, laying the groundwork for partial loading of version 10 segment projections on Historicals.
+
+[#19374](https://github.com/apache/druid/pull/19374)
+
+#### Realtime segments mode query context
+
+The `realtimeSegmentsOnly` query context parameter has been deprecated and replaced with `realtimeSegmentsMode`.
+
+You can set `realtimeSegmentsMode` to one of the following:
+
+- `include` (default): query all segments, including realtime segments
+- `exclude`: skip realtime segments for the query
+- `exclusive`: query only realtime segments
+
+This is useful when performing things like blue/green deployments and you only want to query new Historical replica autoscaling groups and not touch any "live" nodes (neither realtime nor historical).
+
+[#19486](https://github.com/apache/druid/pull/19486)
+
+#### Faster segment metadata search
+
+You can set `druid.segment.timeline.fastIntervalSearch` to `true` so that Druid uses an index based on interval trees to store that metadata in memory for faster identification and retrieval.
+
+This feature is off by default.
+
+[#19138](https://github.com/apache/druid/issues/19138) [#19850](https://github.com/apache/druid/issues/19850)
+
+#### Fetching files in the background
+
+The MSQ task engine now supports workers fetching input files from cloud storage asynchronously. This overlaps downloading with processing, which generally improves throughput when reading larger numbers of files. Otherwise, workers stream each file directly from cloud storage while processing it.
+
+This feature is on by default and is controlled by the `backgroundFetchExternalFiles` setting.
+
+[#19539](https://github.com/apache/druid/pull/19539)
+
+#### Segment prefetching for Dart
+
+Dart now supports the runtime property `druid.msq.dart.worker.segmentLoadAheadCount`, which controls the number of segments that Dart prefetches. If set greater than 0 for a worker, this setting becomes the default `segmentLoadAheadCount` value for the worker. If a query includes the `segmentLoadAheadCount` query context parameter, the query context takes precedence.
+
+[#19559](https://github.com/apache/druid/pull/19559)
+
+#### Partial segment loading
Review Comment:
Hmm, in that case, I feel we should just remove them from the release notes here, as you had suggested earlier. Otherwise, this will just create confusion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] docs: 38.0.0 release notes (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #19945:
URL: https://github.com/apache/druid/pull/19945#discussion_r4151783273
##########
docs/release-info/release-notes.md:
##########
@@ -57,6 +57,80 @@ For tips about how to write a good release note, see [Release notes](https://git
This section contains important information about new and existing features.
+#### Java
+
+Druid now supports Java 25. While Java 21 is still supported, we recommend you upgrade to Java 25.
+
+Support for Java 17 has been dropped.
+
+[#19304](https://github.com/apache/druid/pull/19304) [#19336](https://github.com/apache/druid/pull/19336)
+
+#### Historical tier aliases
+
+You can map a virtual tier name to a number of real Historical tiers with the `historicalTierAliases` Coordinator dynamic configuration. This creates a single identifier for a group of Historical tiers. When a load/drop rule references the alias, the Coordinator replaces it with the actual tiers. For example, if you map the Historical tiers `hot_1` and `hot_2` to the alias `hot`, the rule `{"hot": 2}` loads 2 replicas of each onto `hot_1` and `hot_2`.
+
+[#19204](https://github.com/apache/druid/pull/19204) [#19667](https://github.com/apache/druid/pull/19667)
+
+#### New load rule types
+
+Adds a new family of retention rules, `loadPartialByPeriod`, `loadPartialByInterval`, `loadPartialForever`, laying the groundwork for partial loading of version 10 segment projections on Historicals.
+
+[#19374](https://github.com/apache/druid/pull/19374)
+
+#### Realtime segments mode query context
+
+The `realtimeSegmentsOnly` query context parameter has been deprecated and replaced with `realtimeSegmentsMode`.
+
+You can set `realtimeSegmentsMode` to one of the following:
+
+- `include` (default): query all segments, including realtime segments
+- `exclude`: skip realtime segments for the query
+- `exclusive`: query only realtime segments
+
+This is useful when performing things like blue/green deployments and you only want to query new Historical replica autoscaling groups and not touch any "live" nodes (neither realtime nor historical).
+
+[#19486](https://github.com/apache/druid/pull/19486)
+
+#### Faster segment metadata search
+
+You can set `druid.segment.timeline.fastIntervalSearch` to `true` so that Druid uses an index based on interval trees to store that metadata in memory for faster identification and retrieval.
+
+This feature is off by default.
+
+[#19138](https://github.com/apache/druid/issues/19138) [#19850](https://github.com/apache/druid/issues/19850)
+
+#### Fetching files in the background
+
+The MSQ task engine now supports workers fetching input files from cloud storage asynchronously. This overlaps downloading with processing, which generally improves throughput when reading larger numbers of files. Otherwise, workers stream each file directly from cloud storage while processing it.
+
+This feature is on by default and is controlled by the `backgroundFetchExternalFiles` setting.
+
+[#19539](https://github.com/apache/druid/pull/19539)
+
+#### Segment prefetching for Dart
+
+Dart now supports the runtime property `druid.msq.dart.worker.segmentLoadAheadCount`, which controls the number of segments that Dart prefetches. If set greater than 0 for a worker, this setting becomes the default `segmentLoadAheadCount` value for the worker. If a query includes the `segmentLoadAheadCount` query context parameter, the query context takes precedence.
+
+[#19559](https://github.com/apache/druid/pull/19559)
+
+#### Partial segment loading
Review Comment:
```suggestion
#### Partial segment loading
Historical tiers now support loading partial data from a segment. The exact data to be downloaded is dictated by partial load rules which may list one or more cluster groups or projections as eligible for download.
```
##########
docs/release-info/release-notes.md:
##########
@@ -57,6 +57,80 @@ For tips about how to write a good release note, see [Release notes](https://git
This section contains important information about new and existing features.
+#### Java
+
+Druid now supports Java 25. While Java 21 is still supported, we recommend you upgrade to Java 25.
+
+Support for Java 17 has been dropped.
+
+[#19304](https://github.com/apache/druid/pull/19304) [#19336](https://github.com/apache/druid/pull/19336)
+
+#### Historical tier aliases
+
+You can map a virtual tier name to a number of real Historical tiers with the `historicalTierAliases` Coordinator dynamic configuration. This creates a single identifier for a group of Historical tiers. When a load/drop rule references the alias, the Coordinator replaces it with the actual tiers. For example, if you map the Historical tiers `hot_1` and `hot_2` to the alias `hot`, the rule `{"hot": 2}` loads 2 replicas of each onto `hot_1` and `hot_2`.
+
+[#19204](https://github.com/apache/druid/pull/19204) [#19667](https://github.com/apache/druid/pull/19667)
+
+#### New load rule types
+
+Adds a new family of retention rules, `loadPartialByPeriod`, `loadPartialByInterval`, `loadPartialForever`, laying the groundwork for partial loading of version 10 segment projections on Historicals.
+
+[#19374](https://github.com/apache/druid/pull/19374)
+
+#### Realtime segments mode query context
+
+The `realtimeSegmentsOnly` query context parameter has been deprecated and replaced with `realtimeSegmentsMode`.
+
+You can set `realtimeSegmentsMode` to one of the following:
+
+- `include` (default): query all segments, including realtime segments
+- `exclude`: skip realtime segments for the query
+- `exclusive`: query only realtime segments
+
+This is useful when performing things like blue/green deployments and you only want to query new Historical replica autoscaling groups and not touch any "live" nodes (neither realtime nor historical).
+
+[#19486](https://github.com/apache/druid/pull/19486)
+
+#### Faster segment metadata search
+
+You can set `druid.segment.timeline.fastIntervalSearch` to `true` so that Druid uses an index based on interval trees to store that metadata in memory for faster identification and retrieval.
+
+This feature is off by default.
+
+[#19138](https://github.com/apache/druid/issues/19138) [#19850](https://github.com/apache/druid/issues/19850)
+
+#### Fetching files in the background
+
+The MSQ task engine now supports workers fetching input files from cloud storage asynchronously. This overlaps downloading with processing, which generally improves throughput when reading larger numbers of files. Otherwise, workers stream each file directly from cloud storage while processing it.
+
+This feature is on by default and is controlled by the `backgroundFetchExternalFiles` setting.
+
+[#19539](https://github.com/apache/druid/pull/19539)
+
+#### Segment prefetching for Dart
+
+Dart now supports the runtime property `druid.msq.dart.worker.segmentLoadAheadCount`, which controls the number of segments that Dart prefetches. If set greater than 0 for a worker, this setting becomes the default `segmentLoadAheadCount` value for the worker. If a query includes the `segmentLoadAheadCount` query context parameter, the query context takes precedence.
+
+[#19559](https://github.com/apache/druid/pull/19559)
+
+#### Partial segment loading
+
+[#19620](https://github.com/apache/druid/pull/19620)
+[#19535](https://github.com/apache/druid/pull/19535)
+
+#### Clustered segments
+
+[#19579](https://github.com/apache/druid/pull/19579)
+[#19597](https://github.com/apache/druid/pull/19597) [#19460](https://github.com/apache/druid/pull/19460)
+
+#### Improved Convert to SQL in web console
+
+The web console now supports converting streaming supervisors to SQL-based ingestion queries. This makes it easier for you to run a streaming task as a one-time batch ingestion.
+
+Select the **Convert supervisor to SQL** option from the **...** menu in the Query view. You can select from existing supervisors or provide JSON for a new one.
+
+[#19547](https://github.com/apache/druid/pull/19547)
+
Review Comment:
```suggestion
#### Deprecate legacy zstd frames
Druid 38 will be the last major Druid release that supports reading from legacy v0.4-v0.7 zstd frames. Support will be removed for these frames in future releases for both batch and streaming ingestion.
The primary reason for this deprecation is that the underlying library zstd-jni removes decoding for legacy frames in version 1.5.7-13 onwards.
```
##########
docs/release-info/release-notes.md:
##########
@@ -57,6 +57,80 @@ For tips about how to write a good release note, see [Release notes](https://git
This section contains important information about new and existing features.
+#### Java
+
+Druid now supports Java 25. While Java 21 is still supported, we recommend you upgrade to Java 25.
+
+Support for Java 17 has been dropped.
+
+[#19304](https://github.com/apache/druid/pull/19304) [#19336](https://github.com/apache/druid/pull/19336)
+
+#### Historical tier aliases
+
+You can map a virtual tier name to a number of real Historical tiers with the `historicalTierAliases` Coordinator dynamic configuration. This creates a single identifier for a group of Historical tiers. When a load/drop rule references the alias, the Coordinator replaces it with the actual tiers. For example, if you map the Historical tiers `hot_1` and `hot_2` to the alias `hot`, the rule `{"hot": 2}` loads 2 replicas of each onto `hot_1` and `hot_2`.
+
+[#19204](https://github.com/apache/druid/pull/19204) [#19667](https://github.com/apache/druid/pull/19667)
+
+#### New load rule types
+
+Adds a new family of retention rules, `loadPartialByPeriod`, `loadPartialByInterval`, `loadPartialForever`, laying the groundwork for partial loading of version 10 segment projections on Historicals.
+
+[#19374](https://github.com/apache/druid/pull/19374)
+
+#### Realtime segments mode query context
+
+The `realtimeSegmentsOnly` query context parameter has been deprecated and replaced with `realtimeSegmentsMode`.
+
+You can set `realtimeSegmentsMode` to one of the following:
+
+- `include` (default): query all segments, including realtime segments
+- `exclude`: skip realtime segments for the query
+- `exclusive`: query only realtime segments
+
+This is useful when performing things like blue/green deployments and you only want to query new Historical replica autoscaling groups and not touch any "live" nodes (neither realtime nor historical).
+
+[#19486](https://github.com/apache/druid/pull/19486)
+
+#### Faster segment metadata search
+
+You can set `druid.segment.timeline.fastIntervalSearch` to `true` so that Druid uses an index based on interval trees to store that metadata in memory for faster identification and retrieval.
+
+This feature is off by default.
+
+[#19138](https://github.com/apache/druid/issues/19138) [#19850](https://github.com/apache/druid/issues/19850)
+
+#### Fetching files in the background
+
+The MSQ task engine now supports workers fetching input files from cloud storage asynchronously. This overlaps downloading with processing, which generally improves throughput when reading larger numbers of files. Otherwise, workers stream each file directly from cloud storage while processing it.
+
+This feature is on by default and is controlled by the `backgroundFetchExternalFiles` setting.
+
+[#19539](https://github.com/apache/druid/pull/19539)
+
+#### Segment prefetching for Dart
+
+Dart now supports the runtime property `druid.msq.dart.worker.segmentLoadAheadCount`, which controls the number of segments that Dart prefetches. If set greater than 0 for a worker, this setting becomes the default `segmentLoadAheadCount` value for the worker. If a query includes the `segmentLoadAheadCount` query context parameter, the query context takes precedence.
+
+[#19559](https://github.com/apache/druid/pull/19559)
+
+#### Partial segment loading
+
+[#19620](https://github.com/apache/druid/pull/19620)
+[#19535](https://github.com/apache/druid/pull/19535)
+
+#### Clustered segments
Review Comment:
```suggestion
#### Clustered segments
Clustered segments contain one or more cluster groups that allow quick pruning of data while querying by scanning only the groups which match a given query.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] docs: 38.0.0 release notes (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #19945:
URL: https://github.com/apache/druid/pull/19945#discussion_r4151783273
##########
docs/release-info/release-notes.md:
##########
@@ -57,6 +57,80 @@ For tips about how to write a good release note, see [Release notes](https://git
This section contains important information about new and existing features.
+#### Java
+
+Druid now supports Java 25. While Java 21 is still supported, we recommend you upgrade to Java 25.
+
+Support for Java 17 has been dropped.
+
+[#19304](https://github.com/apache/druid/pull/19304) [#19336](https://github.com/apache/druid/pull/19336)
+
+#### Historical tier aliases
+
+You can map a virtual tier name to a number of real Historical tiers with the `historicalTierAliases` Coordinator dynamic configuration. This creates a single identifier for a group of Historical tiers. When a load/drop rule references the alias, the Coordinator replaces it with the actual tiers. For example, if you map the Historical tiers `hot_1` and `hot_2` to the alias `hot`, the rule `{"hot": 2}` loads 2 replicas of each onto `hot_1` and `hot_2`.
+
+[#19204](https://github.com/apache/druid/pull/19204) [#19667](https://github.com/apache/druid/pull/19667)
+
+#### New load rule types
+
+Adds a new family of retention rules, `loadPartialByPeriod`, `loadPartialByInterval`, `loadPartialForever`, laying the groundwork for partial loading of version 10 segment projections on Historicals.
+
+[#19374](https://github.com/apache/druid/pull/19374)
+
+#### Realtime segments mode query context
+
+The `realtimeSegmentsOnly` query context parameter has been deprecated and replaced with `realtimeSegmentsMode`.
+
+You can set `realtimeSegmentsMode` to one of the following:
+
+- `include` (default): query all segments, including realtime segments
+- `exclude`: skip realtime segments for the query
+- `exclusive`: query only realtime segments
+
+This is useful when performing things like blue/green deployments and you only want to query new Historical replica autoscaling groups and not touch any "live" nodes (neither realtime nor historical).
+
+[#19486](https://github.com/apache/druid/pull/19486)
+
+#### Faster segment metadata search
+
+You can set `druid.segment.timeline.fastIntervalSearch` to `true` so that Druid uses an index based on interval trees to store that metadata in memory for faster identification and retrieval.
+
+This feature is off by default.
+
+[#19138](https://github.com/apache/druid/issues/19138) [#19850](https://github.com/apache/druid/issues/19850)
+
+#### Fetching files in the background
+
+The MSQ task engine now supports workers fetching input files from cloud storage asynchronously. This overlaps downloading with processing, which generally improves throughput when reading larger numbers of files. Otherwise, workers stream each file directly from cloud storage while processing it.
+
+This feature is on by default and is controlled by the `backgroundFetchExternalFiles` setting.
+
+[#19539](https://github.com/apache/druid/pull/19539)
+
+#### Segment prefetching for Dart
+
+Dart now supports the runtime property `druid.msq.dart.worker.segmentLoadAheadCount`, which controls the number of segments that Dart prefetches. If set greater than 0 for a worker, this setting becomes the default `segmentLoadAheadCount` value for the worker. If a query includes the `segmentLoadAheadCount` query context parameter, the query context takes precedence.
+
+[#19559](https://github.com/apache/druid/pull/19559)
+
+#### Partial segment loading
Review Comment:
```suggestion
#### Partial segment loading (experimental)
Historical tiers now support loading partial data from a segment. The exact data to be downloaded is dictated by partial load rules which may list one or more cluster groups or projections as eligible for download.
```
##########
docs/release-info/release-notes.md:
##########
@@ -57,6 +57,80 @@ For tips about how to write a good release note, see [Release notes](https://git
This section contains important information about new and existing features.
+#### Java
+
+Druid now supports Java 25. While Java 21 is still supported, we recommend you upgrade to Java 25.
+
+Support for Java 17 has been dropped.
+
+[#19304](https://github.com/apache/druid/pull/19304) [#19336](https://github.com/apache/druid/pull/19336)
+
+#### Historical tier aliases
+
+You can map a virtual tier name to a number of real Historical tiers with the `historicalTierAliases` Coordinator dynamic configuration. This creates a single identifier for a group of Historical tiers. When a load/drop rule references the alias, the Coordinator replaces it with the actual tiers. For example, if you map the Historical tiers `hot_1` and `hot_2` to the alias `hot`, the rule `{"hot": 2}` loads 2 replicas of each onto `hot_1` and `hot_2`.
+
+[#19204](https://github.com/apache/druid/pull/19204) [#19667](https://github.com/apache/druid/pull/19667)
+
+#### New load rule types
+
+Adds a new family of retention rules, `loadPartialByPeriod`, `loadPartialByInterval`, `loadPartialForever`, laying the groundwork for partial loading of version 10 segment projections on Historicals.
+
+[#19374](https://github.com/apache/druid/pull/19374)
+
+#### Realtime segments mode query context
+
+The `realtimeSegmentsOnly` query context parameter has been deprecated and replaced with `realtimeSegmentsMode`.
+
+You can set `realtimeSegmentsMode` to one of the following:
+
+- `include` (default): query all segments, including realtime segments
+- `exclude`: skip realtime segments for the query
+- `exclusive`: query only realtime segments
+
+This is useful when performing things like blue/green deployments and you only want to query new Historical replica autoscaling groups and not touch any "live" nodes (neither realtime nor historical).
+
+[#19486](https://github.com/apache/druid/pull/19486)
+
+#### Faster segment metadata search
+
+You can set `druid.segment.timeline.fastIntervalSearch` to `true` so that Druid uses an index based on interval trees to store that metadata in memory for faster identification and retrieval.
+
+This feature is off by default.
+
+[#19138](https://github.com/apache/druid/issues/19138) [#19850](https://github.com/apache/druid/issues/19850)
+
+#### Fetching files in the background
+
+The MSQ task engine now supports workers fetching input files from cloud storage asynchronously. This overlaps downloading with processing, which generally improves throughput when reading larger numbers of files. Otherwise, workers stream each file directly from cloud storage while processing it.
+
+This feature is on by default and is controlled by the `backgroundFetchExternalFiles` setting.
+
+[#19539](https://github.com/apache/druid/pull/19539)
+
+#### Segment prefetching for Dart
+
+Dart now supports the runtime property `druid.msq.dart.worker.segmentLoadAheadCount`, which controls the number of segments that Dart prefetches. If set greater than 0 for a worker, this setting becomes the default `segmentLoadAheadCount` value for the worker. If a query includes the `segmentLoadAheadCount` query context parameter, the query context takes precedence.
+
+[#19559](https://github.com/apache/druid/pull/19559)
+
+#### Partial segment loading
+
+[#19620](https://github.com/apache/druid/pull/19620)
+[#19535](https://github.com/apache/druid/pull/19535)
+
+#### Clustered segments
Review Comment:
```suggestion
#### Clustered segments (experimental)
Clustered segments contain one or more cluster groups that allow quick pruning of data while querying by scanning only the groups which match a given query.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] docs: 38.0.0 release notes (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #19945:
URL: https://github.com/apache/druid/pull/19945#discussion_r4153166430
##########
docs/release-info/release-notes.md:
##########
@@ -57,6 +57,80 @@ For tips about how to write a good release note, see [Release notes](https://git
This section contains important information about new and existing features.
+#### Java
+
+Druid now supports Java 25. While Java 21 is still supported, we recommend you upgrade to Java 25.
+
+Support for Java 17 has been dropped.
+
+[#19304](https://github.com/apache/druid/pull/19304) [#19336](https://github.com/apache/druid/pull/19336)
+
+#### Historical tier aliases
+
+You can map a virtual tier name to a number of real Historical tiers with the `historicalTierAliases` Coordinator dynamic configuration. This creates a single identifier for a group of Historical tiers. When a load/drop rule references the alias, the Coordinator replaces it with the actual tiers. For example, if you map the Historical tiers `hot_1` and `hot_2` to the alias `hot`, the rule `{"hot": 2}` loads 2 replicas of each onto `hot_1` and `hot_2`.
+
+[#19204](https://github.com/apache/druid/pull/19204) [#19667](https://github.com/apache/druid/pull/19667)
+
+#### New load rule types
+
+Adds a new family of retention rules, `loadPartialByPeriod`, `loadPartialByInterval`, `loadPartialForever`, laying the groundwork for partial loading of version 10 segment projections on Historicals.
+
+[#19374](https://github.com/apache/druid/pull/19374)
+
+#### Realtime segments mode query context
+
+The `realtimeSegmentsOnly` query context parameter has been deprecated and replaced with `realtimeSegmentsMode`.
+
+You can set `realtimeSegmentsMode` to one of the following:
+
+- `include` (default): query all segments, including realtime segments
+- `exclude`: skip realtime segments for the query
+- `exclusive`: query only realtime segments
+
+This is useful when performing things like blue/green deployments and you only want to query new Historical replica autoscaling groups and not touch any "live" nodes (neither realtime nor historical).
+
+[#19486](https://github.com/apache/druid/pull/19486)
+
+#### Faster segment metadata search
+
+You can set `druid.segment.timeline.fastIntervalSearch` to `true` so that Druid uses an index based on interval trees to store that metadata in memory for faster identification and retrieval.
+
+This feature is off by default.
+
+[#19138](https://github.com/apache/druid/issues/19138) [#19850](https://github.com/apache/druid/issues/19850)
+
+#### Fetching files in the background
+
+The MSQ task engine now supports workers fetching input files from cloud storage asynchronously. This overlaps downloading with processing, which generally improves throughput when reading larger numbers of files. Otherwise, workers stream each file directly from cloud storage while processing it.
+
+This feature is on by default and is controlled by the `backgroundFetchExternalFiles` setting.
+
+[#19539](https://github.com/apache/druid/pull/19539)
+
+#### Segment prefetching for Dart
+
+Dart now supports the runtime property `druid.msq.dart.worker.segmentLoadAheadCount`, which controls the number of segments that Dart prefetches. If set greater than 0 for a worker, this setting becomes the default `segmentLoadAheadCount` value for the worker. If a query includes the `segmentLoadAheadCount` query context parameter, the query context takes precedence.
+
+[#19559](https://github.com/apache/druid/pull/19559)
+
+#### Partial segment loading
+
+[#19620](https://github.com/apache/druid/pull/19620)
+[#19535](https://github.com/apache/druid/pull/19535)
+
+#### Clustered segments
Review Comment:
So, do you advise just removing these items from the release notes for now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] docs: 38.0.0 release notes (druid)
Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #19945:
URL: https://github.com/apache/druid/pull/19945#discussion_r4152507694
##########
docs/release-info/release-notes.md:
##########
@@ -57,6 +57,80 @@ For tips about how to write a good release note, see [Release notes](https://git
This section contains important information about new and existing features.
+#### Java
+
+Druid now supports Java 25. While Java 21 is still supported, we recommend you upgrade to Java 25.
+
+Support for Java 17 has been dropped.
+
+[#19304](https://github.com/apache/druid/pull/19304) [#19336](https://github.com/apache/druid/pull/19336)
+
+#### Historical tier aliases
+
+You can map a virtual tier name to a number of real Historical tiers with the `historicalTierAliases` Coordinator dynamic configuration. This creates a single identifier for a group of Historical tiers. When a load/drop rule references the alias, the Coordinator replaces it with the actual tiers. For example, if you map the Historical tiers `hot_1` and `hot_2` to the alias `hot`, the rule `{"hot": 2}` loads 2 replicas of each onto `hot_1` and `hot_2`.
+
+[#19204](https://github.com/apache/druid/pull/19204) [#19667](https://github.com/apache/druid/pull/19667)
+
+#### New load rule types
+
+Adds a new family of retention rules, `loadPartialByPeriod`, `loadPartialByInterval`, `loadPartialForever`, laying the groundwork for partial loading of version 10 segment projections on Historicals.
+
+[#19374](https://github.com/apache/druid/pull/19374)
+
+#### Realtime segments mode query context
+
+The `realtimeSegmentsOnly` query context parameter has been deprecated and replaced with `realtimeSegmentsMode`.
+
+You can set `realtimeSegmentsMode` to one of the following:
+
+- `include` (default): query all segments, including realtime segments
+- `exclude`: skip realtime segments for the query
+- `exclusive`: query only realtime segments
+
+This is useful when performing things like blue/green deployments and you only want to query new Historical replica autoscaling groups and not touch any "live" nodes (neither realtime nor historical).
+
+[#19486](https://github.com/apache/druid/pull/19486)
+
+#### Faster segment metadata search
+
+You can set `druid.segment.timeline.fastIntervalSearch` to `true` so that Druid uses an index based on interval trees to store that metadata in memory for faster identification and retrieval.
+
+This feature is off by default.
+
+[#19138](https://github.com/apache/druid/issues/19138) [#19850](https://github.com/apache/druid/issues/19850)
+
+#### Fetching files in the background
+
+The MSQ task engine now supports workers fetching input files from cloud storage asynchronously. This overlaps downloading with processing, which generally improves throughput when reading larger numbers of files. Otherwise, workers stream each file directly from cloud storage while processing it.
+
+This feature is on by default and is controlled by the `backgroundFetchExternalFiles` setting.
+
+[#19539](https://github.com/apache/druid/pull/19539)
+
+#### Segment prefetching for Dart
+
+Dart now supports the runtime property `druid.msq.dart.worker.segmentLoadAheadCount`, which controls the number of segments that Dart prefetches. If set greater than 0 for a worker, this setting becomes the default `segmentLoadAheadCount` value for the worker. If a query includes the `segmentLoadAheadCount` query context parameter, the query context takes precedence.
+
+[#19559](https://github.com/apache/druid/pull/19559)
+
+#### Partial segment loading
Review Comment:
partial load rules are actually separate from the linked PRs (there are a handful of those PRs too). 19535 is about allowing historicals running in virtual storage mode, with v10 segments which are stored in deep storage without zip, to load on demand at the column level at query time (so dictated by what the query needs to read rather than load rules), when using Dart or MSQ (legacy engine still requires full download up front in virtual mode). The storage accounting of the historical cache is done at the 'bundle' level (base table, projections, cluster groups), where the cache has an entry for each bundle and we just lazy load columns contained in that bundle into it as the query needs.
Partial load rules work with this partial load functionality by allowing virtual mode historicals to eagerly load parts of segments at the 'bundle' level (base table, projections, cluster groups) and make those bundles sticky so that they cannot be evicted/are always 'hot' even if the rest of the segment can be loaded and evicted on demand.
All of this stuff is quite experimental, and will probably be in a lot nicer state next release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] docs: 38.0.0 release notes (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #19945:
URL: https://github.com/apache/druid/pull/19945#discussion_r4155050586
##########
docs/release-info/release-notes.md:
##########
@@ -57,6 +57,80 @@ For tips about how to write a good release note, see [Release notes](https://git
This section contains important information about new and existing features.
+#### Java
+
+Druid now supports Java 25. While Java 21 is still supported, we recommend you upgrade to Java 25.
+
+Support for Java 17 has been dropped.
+
+[#19304](https://github.com/apache/druid/pull/19304) [#19336](https://github.com/apache/druid/pull/19336)
+
+#### Historical tier aliases
+
+You can map a virtual tier name to a number of real Historical tiers with the `historicalTierAliases` Coordinator dynamic configuration. This creates a single identifier for a group of Historical tiers. When a load/drop rule references the alias, the Coordinator replaces it with the actual tiers. For example, if you map the Historical tiers `hot_1` and `hot_2` to the alias `hot`, the rule `{"hot": 2}` loads 2 replicas of each onto `hot_1` and `hot_2`.
+
+[#19204](https://github.com/apache/druid/pull/19204) [#19667](https://github.com/apache/druid/pull/19667)
+
+#### New load rule types
+
+Adds a new family of retention rules, `loadPartialByPeriod`, `loadPartialByInterval`, `loadPartialForever`, laying the groundwork for partial loading of version 10 segment projections on Historicals.
+
+[#19374](https://github.com/apache/druid/pull/19374)
+
+#### Realtime segments mode query context
+
+The `realtimeSegmentsOnly` query context parameter has been deprecated and replaced with `realtimeSegmentsMode`.
+
+You can set `realtimeSegmentsMode` to one of the following:
+
+- `include` (default): query all segments, including realtime segments
+- `exclude`: skip realtime segments for the query
+- `exclusive`: query only realtime segments
+
+This is useful when performing things like blue/green deployments and you only want to query new Historical replica autoscaling groups and not touch any "live" nodes (neither realtime nor historical).
+
+[#19486](https://github.com/apache/druid/pull/19486)
+
+#### Faster segment metadata search
+
+You can set `druid.segment.timeline.fastIntervalSearch` to `true` so that Druid uses an index based on interval trees to store that metadata in memory for faster identification and retrieval.
+
+This feature is off by default.
+
+[#19138](https://github.com/apache/druid/issues/19138) [#19850](https://github.com/apache/druid/issues/19850)
+
+#### Fetching files in the background
+
+The MSQ task engine now supports workers fetching input files from cloud storage asynchronously. This overlaps downloading with processing, which generally improves throughput when reading larger numbers of files. Otherwise, workers stream each file directly from cloud storage while processing it.
+
+This feature is on by default and is controlled by the `backgroundFetchExternalFiles` setting.
+
+[#19539](https://github.com/apache/druid/pull/19539)
+
+#### Segment prefetching for Dart
+
+Dart now supports the runtime property `druid.msq.dart.worker.segmentLoadAheadCount`, which controls the number of segments that Dart prefetches. If set greater than 0 for a worker, this setting becomes the default `segmentLoadAheadCount` value for the worker. If a query includes the `segmentLoadAheadCount` query context parameter, the query context takes precedence.
Review Comment:
Thanks for calling it out, updating it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] docs: 38.0.0 release notes (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #19945:
URL: https://github.com/apache/druid/pull/19945#discussion_r4155037065
##########
docs/release-info/release-notes.md:
##########
@@ -57,6 +57,80 @@ For tips about how to write a good release note, see [Release notes](https://git
This section contains important information about new and existing features.
+#### Java
+
+Druid now supports Java 25. While Java 21 is still supported, we recommend you upgrade to Java 25.
+
+Support for Java 17 has been dropped.
+
+[#19304](https://github.com/apache/druid/pull/19304) [#19336](https://github.com/apache/druid/pull/19336)
+
+#### Historical tier aliases
+
+You can map a virtual tier name to a number of real Historical tiers with the `historicalTierAliases` Coordinator dynamic configuration. This creates a single identifier for a group of Historical tiers. When a load/drop rule references the alias, the Coordinator replaces it with the actual tiers. For example, if you map the Historical tiers `hot_1` and `hot_2` to the alias `hot`, the rule `{"hot": 2}` loads 2 replicas of each onto `hot_1` and `hot_2`.
+
+[#19204](https://github.com/apache/druid/pull/19204) [#19667](https://github.com/apache/druid/pull/19667)
+
+#### New load rule types
+
+Adds a new family of retention rules, `loadPartialByPeriod`, `loadPartialByInterval`, `loadPartialForever`, laying the groundwork for partial loading of version 10 segment projections on Historicals.
Review Comment:
I think removing it for now would make more sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] docs: 38.0.0 release notes (druid)
Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #19945:
URL: https://github.com/apache/druid/pull/19945#discussion_r4152518288
##########
docs/release-info/release-notes.md:
##########
@@ -57,6 +57,80 @@ For tips about how to write a good release note, see [Release notes](https://git
This section contains important information about new and existing features.
+#### Java
+
+Druid now supports Java 25. While Java 21 is still supported, we recommend you upgrade to Java 25.
+
+Support for Java 17 has been dropped.
+
+[#19304](https://github.com/apache/druid/pull/19304) [#19336](https://github.com/apache/druid/pull/19336)
+
+#### Historical tier aliases
+
+You can map a virtual tier name to a number of real Historical tiers with the `historicalTierAliases` Coordinator dynamic configuration. This creates a single identifier for a group of Historical tiers. When a load/drop rule references the alias, the Coordinator replaces it with the actual tiers. For example, if you map the Historical tiers `hot_1` and `hot_2` to the alias `hot`, the rule `{"hot": 2}` loads 2 replicas of each onto `hot_1` and `hot_2`.
+
+[#19204](https://github.com/apache/druid/pull/19204) [#19667](https://github.com/apache/druid/pull/19667)
+
+#### New load rule types
+
+Adds a new family of retention rules, `loadPartialByPeriod`, `loadPartialByInterval`, `loadPartialForever`, laying the groundwork for partial loading of version 10 segment projections on Historicals.
+
+[#19374](https://github.com/apache/druid/pull/19374)
+
+#### Realtime segments mode query context
+
+The `realtimeSegmentsOnly` query context parameter has been deprecated and replaced with `realtimeSegmentsMode`.
+
+You can set `realtimeSegmentsMode` to one of the following:
+
+- `include` (default): query all segments, including realtime segments
+- `exclude`: skip realtime segments for the query
+- `exclusive`: query only realtime segments
+
+This is useful when performing things like blue/green deployments and you only want to query new Historical replica autoscaling groups and not touch any "live" nodes (neither realtime nor historical).
+
+[#19486](https://github.com/apache/druid/pull/19486)
+
+#### Faster segment metadata search
+
+You can set `druid.segment.timeline.fastIntervalSearch` to `true` so that Druid uses an index based on interval trees to store that metadata in memory for faster identification and retrieval.
+
+This feature is off by default.
+
+[#19138](https://github.com/apache/druid/issues/19138) [#19850](https://github.com/apache/druid/issues/19850)
+
+#### Fetching files in the background
+
+The MSQ task engine now supports workers fetching input files from cloud storage asynchronously. This overlaps downloading with processing, which generally improves throughput when reading larger numbers of files. Otherwise, workers stream each file directly from cloud storage while processing it.
+
+This feature is on by default and is controlled by the `backgroundFetchExternalFiles` setting.
+
+[#19539](https://github.com/apache/druid/pull/19539)
+
+#### Segment prefetching for Dart
+
+Dart now supports the runtime property `druid.msq.dart.worker.segmentLoadAheadCount`, which controls the number of segments that Dart prefetches. If set greater than 0 for a worker, this setting becomes the default `segmentLoadAheadCount` value for the worker. If a query includes the `segmentLoadAheadCount` query context parameter, the query context takes precedence.
Review Comment:
this is only really applicable to virtual storage mode historicals i think, though i suppose the docs for it don't mention that either
##########
docs/release-info/release-notes.md:
##########
@@ -57,6 +57,80 @@ For tips about how to write a good release note, see [Release notes](https://git
This section contains important information about new and existing features.
+#### Java
+
+Druid now supports Java 25. While Java 21 is still supported, we recommend you upgrade to Java 25.
+
+Support for Java 17 has been dropped.
+
+[#19304](https://github.com/apache/druid/pull/19304) [#19336](https://github.com/apache/druid/pull/19336)
+
+#### Historical tier aliases
+
+You can map a virtual tier name to a number of real Historical tiers with the `historicalTierAliases` Coordinator dynamic configuration. This creates a single identifier for a group of Historical tiers. When a load/drop rule references the alias, the Coordinator replaces it with the actual tiers. For example, if you map the Historical tiers `hot_1` and `hot_2` to the alias `hot`, the rule `{"hot": 2}` loads 2 replicas of each onto `hot_1` and `hot_2`.
+
+[#19204](https://github.com/apache/druid/pull/19204) [#19667](https://github.com/apache/druid/pull/19667)
+
+#### New load rule types
+
+Adds a new family of retention rules, `loadPartialByPeriod`, `loadPartialByInterval`, `loadPartialForever`, laying the groundwork for partial loading of version 10 segment projections on Historicals.
Review Comment:
partial load rules allow virtual storage mode historicals (also not documented i think, druid.segmentCache.virtualStorage and druid.segmentCache.virtualStoragePartialDownloadsEnabled both true) to eagerly load "parts" of v10 segments that are stored in deep storage unzipped (druid.storage.zip=false but only s3 supports that). There are matchers that must be set on the partial rule that can select the parts of the v10 segments (v10 segments are internally organized into 'bundles' which are sort of logical containers that hold all of the columns of a projection, where projection could be the base table (or individual cluster groups for clustered segments which when concatenated together form the base table projection), or individual aggregate projections).
I'm hesitant to call this stuff out since like the other partial load stuff it its going to be pretty tough to use without docs.
I wonder if we should consolidate all of this stuff in an 'experimental virtual storage mode improvements' section (partial load on demand and the partial load rules built on top of it) or just remove it as suggested in other thread
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] docs: 38.0.0 release notes (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz merged PR #19945:
URL: https://github.com/apache/druid/pull/19945
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] docs: 38.0.0 release notes (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #19945:
URL: https://github.com/apache/druid/pull/19945#discussion_r4154581598
##########
docs/release-info/release-notes.md:
##########
@@ -57,6 +57,75 @@ For tips about how to write a good release note, see [Release notes](https://git
This section contains important information about new and existing features.
+#### Java
+
+Druid now supports Java 25. While Java 21 is still supported, we recommend you upgrade to Java 25.
+
+Support for Java 17 has been dropped.
+
+[#19304](https://github.com/apache/druid/pull/19304) [#19336](https://github.com/apache/druid/pull/19336)
+
+#### Historical tier aliases
+
+You can map a virtual tier name to a number of real Historical tiers with the `historicalTierAliases` Coordinator dynamic configuration. This creates a single identifier for a group of Historical tiers. When a load/drop rule references the alias, the Coordinator replaces it with the actual tiers. For example, if you map the Historical tiers `hot_1` and `hot_2` to the alias `hot`, the rule `{"hot": 2}` loads 2 replicas of each onto `hot_1` and `hot_2`.
+
+[#19204](https://github.com/apache/druid/pull/19204) [#19667](https://github.com/apache/druid/pull/19667)
+
+#### New load rule types
+
+Adds a new family of retention rules, `loadPartialByPeriod`, `loadPartialByInterval`, `loadPartialForever`, laying the groundwork for partial loading of version 10 segment projections on Historicals.
+
+[#19374](https://github.com/apache/druid/pull/19374)
+
+#### Realtime segments mode query context
+
+The `realtimeSegmentsOnly` query context parameter has been deprecated and replaced with `realtimeSegmentsMode`.
+
+You can set `realtimeSegmentsMode` to one of the following:
+
+- `include` (default): query all segments, including realtime segments
+- `exclude`: skip realtime segments for the query
+- `exclusive`: query only realtime segments
+
+This is useful when performing things like blue/green deployments and you only want to query new Historical replica autoscaling groups and not touch any "live" nodes (neither realtime nor historical).
+
+[#19486](https://github.com/apache/druid/pull/19486)
+
+#### Faster segment metadata search
+
+You can set `druid.segment.timeline.fastIntervalSearch` to `true` so that Druid uses an index based on interval trees to store that metadata in memory for faster identification and retrieval.
+
+This feature is off by default.
+
+[#19138](https://github.com/apache/druid/issues/19138) [#19850](https://github.com/apache/druid/issues/19850)
+
+#### Fetching files in the background
+
+The MSQ task engine now supports workers fetching input files from cloud storage asynchronously. This overlaps downloading with processing, which generally improves throughput when reading larger numbers of files. Otherwise, workers stream each file directly from cloud storage while processing it.
+
+This feature is on by default and is controlled by the `backgroundFetchExternalFiles` setting.
+
+[#19539](https://github.com/apache/druid/pull/19539)
+
+#### Segment prefetching for Dart
+
+Dart now supports the runtime property `druid.msq.dart.worker.segmentLoadAheadCount`, which controls the number of segments that Dart prefetches. If set greater than 0 for a worker, this setting becomes the default `segmentLoadAheadCount` value for the worker. If a query includes the `segmentLoadAheadCount` query context parameter, the query context takes precedence.
+
+[#19559](https://github.com/apache/druid/pull/19559)
+
+#### Improved Convert to SQL in web console
+
+The web console now supports converting streaming supervisors to SQL-based ingestion queries. This makes it easier for you to run a streaming task as a one-time batch ingestion.
+
+Select the **Convert supervisor to SQL** option from the **...** menu in the Query view. You can select from existing supervisors or provide JSON for a new one.
+
+[#19547](https://github.com/apache/druid/pull/19547)
+
+#### Deprecate legacy zstd frames
+
+Druid 38 will be the last major Druid release that supports reading from legacy v0.4-v0.7 zstd frames. Support will be removed for these frames in future releases for both batch and streaming ingestion.
+
+The primary reason for this deprecation is that the underlying library zstd-jni removes decoding for legacy frames in version 1.5.7-13 onwards.
Review Comment:
```suggestion
#### Deprecate legacy ZSTD frames
Druid 38 will be the last major Druid release that supports reading from legacy ZSTD (version 0.4 to 0.7) frames. Support will be removed for these frames in future releases for both batch and streaming ingestion.
The primary reason for this deprecation is that the underlying library `zstd-jni` removes decoding for legacy frames in version `1.5.7-13` onwards.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] docs: 38.0.0 release notes (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #19945:
URL: https://github.com/apache/druid/pull/19945#discussion_r4155074016
##########
docs/release-info/release-notes.md:
##########
@@ -65,42 +135,389 @@ This section contains detailed release notes separated by areas.
#### Other web console improvements
+- Added the following status details to the **Services** view for Historical services: cloning from another Historical, in turbo loading mode, in decommissioning mode [#19253](https://github.com/apache/druid/pull/19253)
+- Added support for resetting a supervisor to the latest offsets and backfilling [#19533](https://github.com/apache/druid/pull/19533)
+- Improved handling for new tabs [#19483](https://github.com/apache/druid/pull/19483)
+- Improved the Home view's **Services** card. It now reports Overlord, Coordinator, Router, Broker, and Indexer counts on clusters where the web console talks to the Coordinator without SQL access [#19481](https://github.com/apache/druid/pull/19481)
+
### Ingestion
+* You can now use the expression aggregator at ingestion time for expressions that produce a LONG or DOUBLE for both fold and combine expressions [#19508](https://github.com/apache/druid/pull/19508)
+* Added `now()` expression function that returns the current system timestamp in milliseconds since epoch. Useful at ingestion time for troubleshooting pipeline delays (e.g., `now() - __time`). Note: `now()` is non-deterministic as it evaluates for every row, so it can break idempotency. This can be added to any column besides `__time` [#19386](https://github.com/apache/druid/pull/19386)
+* Improved resiliency when ingesting from S3. Druid now retries on `SSLException` and transient credential errors instead of failing [#19617](https://github.com/apache/druid/pull/19617) [#19558](https://github.com/apache/druid/pull/19558)
+* Improved S3 performance [#19394](https://github.com/apache/druid/pull/19394)
+* Updated the default S3 connection pool size so that it's computed based on the number of available processors [#19536](https://github.com/apache/druid/pull/19536)
+
#### SQL-based ingestion
+##### EXTERN for S3 now supports role ARN
+
+You can now include the role ARN when running an INSERT INTO EXTERN query. For example:
+
+```
+INSERT INTO
+EXTERN(
+ s3(bucket => 'test2', prefix => 'export', assumeRoleArn => 'arn:aws:iam::00000:role/test-20260520'))
+AS CSV
+SELECT ...
+```
+
+[#19317](https://github.com/apache/druid/pull/19317)
+
##### Other SQL-based ingestion improvements
+- Added storage counters for the amount of bytes and files written to local and durable storage as well as the state of the local `ByteTracker`. Per-worker storage counters have also been added to the web console [#19316](https://github.com/apache/druid/pull/19316)
+
#### Streaming ingestion
+##### Scaling cool down
+
+You can now configure different cool down durations for scaling up and scaling down streaming task autoscalers.
+
+[#19286](https://github.com/apache/druid/pull/19286)
+
+##### Improved supervisor restarts
+
+Supervisors no longer restart for all changes. Based on the type of change, one of the following can occur:
+
+- Druid persists the updated spec without a restart.
+- Druid restarts the supervisor, but running tasks aren't impacted.
+- Druid restarts the supervisor, terminates the current tasks and launches fresh tasks (default behavior in older Druid versions).
+
+For example, cosmetic changes to a supervisor spec no longer trigger a restart.
+
+[#19700](https://github.com/apache/druid/pull/19700) [#19720](https://github.com/apache/druid/pull/19720)
+
+Additionally, the algorithm for determining a change in the spec has been improved. For example, changes to `ioConfig.taskCount` don't trigger a supervisor restart if auto-scaling is enabled.
+
+[#19541](https://github.com/apache/druid/pull/19541)
+
+###### Latest offset and backfill
+
+For Kafka and RabbitMQ, you can now reset a supervisor to the latest offset and start a new bounded backfill supervisor to ingest data from the skipped range. This is a useful feature for operating Druid clusters where the most recent data is the most important, like alerting scenarios.
+
+Note the following requirements:
+
+- The supervisor's `useEarliestSequenceNumber` property must be `false`.
+- The supervisor context must have `useConcurrentLocks` set to `true` to allow the backfill supervisor's tasks to write concurrently with the main supervisor's tasks.
+- The supervisor must be in a `RUNNING` state.
+
+Use `POST` `/druid/indexer/v1/supervisor/{supervisorId}/resetToLatestAndBackfill` or the web console to perform this action.
+
+[#19477](https://github.com/apache/druid/pull/19477)
+
+##### Prunable shard specs for streaming published segments
+
+Kafka ingestion can now publish segments that the Broker prunes at query time without waiting for compaction. Set `tuningConfig.streamingPartitionsSpec.partitionDimensions` to a list of low-to-medium cardinality dimensions; each task records
+the distinct values it observes per dimension and stamps them onto a new `dim_value_set` shard spec. Queries that filter on a declared dimension then skip segments whose values can't match.
+
+The feature is opt-in, Kafka-only, and disabled by
+default; when unset, behavior is unchanged.
+
+`dim_value_set` is a new core shard spec type with no fallback, so it is not forward-compatible. Upgrade all services before enabling `streamingPartitionsSpec`. Once `dim_value_set` segments are published, downgrade is unsupported
+until they are compacted away or `streamingPartitionsSpec` is removed.
+
+Added `maxValuesPerDimension` (optional) to `streamingPartitionsSpec`.
+
+[#19571](https://github.com/apache/druid/pull/19571) [#19596](https://github.com/apache/druid/pull/19596)
+
##### Other streaming ingestion improvements
+- Added a property called `boundedStreamConfig` to the `SeekableStreamSupervisorIOConfig`, which allows operators to spin up a Supervisor that consumes only a specified offset range [#19372](https://github.com/apache/druid/pull/19372)
+- Improved the cost-based autoscaler for better throughput [#19646](https://github.com/apache/druid/pull/19646)
+
### Querying
+#### New query laning strategy
+
+The `weighted` query laning strategy scores queries by how many thresholds they breach (segment count, interval duration, data age, segment range) and assigns them to configurable graduated lanes with different capacity limits, providing more nuanced lane assignment than the existing binary high/low strategy.
+
+The weighted query laning strategy supports optional per-threshold cost weights:
+
+- `periodWeight`
+- `durationWeight`
+- `segmentCountWeight`
+- `segmentRangeWeight`
+
+These threshold weights default to 1.
+
+[#19225](https://github.com/apache/druid/pull/19225) [#19665](https://github.com/apache/druid/pull/19665) [#19696](https://github.com/apache/druid/pull/19696)
+
#### Other querying improvements
+- Added nullable `minTime/maxTime` Long fields to `ProjectionMetadata` [#19398](https://github.com/apache/druid/pull/19398)
+- Added `getDimensionRangeSet` support to `LikeDimFilter` for equality and prefix cases [#19524](https://github.com/apache/druid/pull/19524)
+- Added support for aggregate projections with clustered segments [#19599](https://github.com/apache/druid/pull/19599)
+- Optimized performance of aggregators for groupBy queries [#19423](https://github.com/apache/druid/pull/19423)
+
### Cluster management
+#### Improved `diskNormalized` balancer strategy
+
+The `diskNormalized` strategy is now more tunable. The `utilizationThreshold` configuration works more intuitively: increasing the threshold increases the "tolerance" of the strategy while decreasing the threshold pushes nodes' disk utilization closer together.
+
+[#19663](https://github.com/apache/druid/pull/19663)
+
+#### MiddleManager and Indexer restarts
+
+MiddleManagers and Indexers now persist their enabled or disabled state across restarts. If you prefer the old behavior, where the server re-enables itself after a restart, set `druid.worker.startAlwaysEnabled = true`.
+
+[#19373](https://github.com/apache/druid/pull/19373)
+
+#### Kafka idle signal
Review Comment:
```suggestion
#### Use processing rate for cost-based auto-scaler
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] docs: 38.0.0 release notes (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #19945:
URL: https://github.com/apache/druid/pull/19945#discussion_r4153383954
##########
docs/release-info/release-notes.md:
##########
@@ -57,6 +57,85 @@ For tips about how to write a good release note, see [Release notes](https://git
This section contains important information about new and existing features.
+#### Java
+
+Druid now supports Java 25. While Java 21 is still supported, we recommend you upgrade to Java 25.
+
+Support for Java 17 has been dropped.
+
+[#19304](https://github.com/apache/druid/pull/19304) [#19336](https://github.com/apache/druid/pull/19336)
+
+#### Historical tier aliases
+
+You can map a virtual tier name to a number of real Historical tiers with the `historicalTierAliases` Coordinator dynamic configuration. This creates a single identifier for a group of Historical tiers. When a load/drop rule references the alias, the Coordinator replaces it with the actual tiers. For example, if you map the Historical tiers `hot_1` and `hot_2` to the alias `hot`, the rule `{"hot": 2}` loads 2 replicas of each onto `hot_1` and `hot_2`.
+
+[#19204](https://github.com/apache/druid/pull/19204) [#19667](https://github.com/apache/druid/pull/19667)
+
+#### New load rule types
+
+Adds a new family of retention rules, `loadPartialByPeriod`, `loadPartialByInterval`, `loadPartialForever`, laying the groundwork for partial loading of version 10 segment projections on Historicals.
+
+[#19374](https://github.com/apache/druid/pull/19374)
+
+#### Realtime segments mode query context
+
+The `realtimeSegmentsOnly` query context parameter has been deprecated and replaced with `realtimeSegmentsMode`.
+
+You can set `realtimeSegmentsMode` to one of the following:
+
+- `include` (default): query all segments, including realtime segments
+- `exclude`: skip realtime segments for the query
+- `exclusive`: query only realtime segments
+
+This is useful when performing things like blue/green deployments and you only want to query new Historical replica autoscaling groups and not touch any "live" nodes (neither realtime nor historical).
+
+[#19486](https://github.com/apache/druid/pull/19486)
+
+#### Faster segment metadata search
+
+You can set `druid.segment.timeline.fastIntervalSearch` to `true` so that Druid uses an index based on interval trees to store that metadata in memory for faster identification and retrieval.
+
+This feature is off by default.
+
+[#19138](https://github.com/apache/druid/issues/19138) [#19850](https://github.com/apache/druid/issues/19850)
+
+#### Fetching files in the background
+
+The MSQ task engine now supports workers fetching input files from cloud storage asynchronously. This overlaps downloading with processing, which generally improves throughput when reading larger numbers of files. Otherwise, workers stream each file directly from cloud storage while processing it.
+
+This feature is on by default and is controlled by the `backgroundFetchExternalFiles` setting.
+
+[#19539](https://github.com/apache/druid/pull/19539)
+
+#### Segment prefetching for Dart
+
+Dart now supports the runtime property `druid.msq.dart.worker.segmentLoadAheadCount`, which controls the number of segments that Dart prefetches. If set greater than 0 for a worker, this setting becomes the default `segmentLoadAheadCount` value for the worker. If a query includes the `segmentLoadAheadCount` query context parameter, the query context takes precedence.
+
+[#19559](https://github.com/apache/druid/pull/19559)
+
+#### Partial segment loading
+
+[#19620](https://github.com/apache/druid/pull/19620)
+[#19535](https://github.com/apache/druid/pull/19535)
+
+#### Clustered segments
+
+[#19579](https://github.com/apache/druid/pull/19579)
+[#19597](https://github.com/apache/druid/pull/19597) [#19460](https://github.com/apache/druid/pull/19460)
+
Review Comment:
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] docs: 38.0.0 release notes (druid)
Posted by "FrankChen021 (via GitHub)" <gi...@apache.org>.
FrankChen021 commented on PR #19945:
URL: https://github.com/apache/druid/pull/19945#issuecomment-5741646980
as discussed in https://github.com/apache/druid/pull/20236#issuecomment-5646494509
we should call out the deprecation of old pre-standard v0.4-v0.7 zstd frames support on both batch and kafka ingestion from this release and the support of these frames will be removed in next release.
See also https://github.com/apache/druid/pull/20110
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] docs: 38.0.0 release notes (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #19945:
URL: https://github.com/apache/druid/pull/19945#discussion_r3998702568
##########
docs/release-info/release-notes.md:
##########
@@ -65,42 +139,389 @@ This section contains detailed release notes separated by areas.
#### Other web console improvements
+- Added the following status details to the **Services** view for Historical services: cloning from another Historical, in turbo loading mode, in decommissioning mode [#19253](https://github.com/apache/druid/pull/19253)
+- Added support for resetting a supervisor to the latest offsets and backfilling [#19533](https://github.com/apache/druid/pull/19533)
+- Improved handling for new tabs [#19483](https://github.com/apache/druid/pull/19483)
+- Improved the Home view's **Services** card. It now reports Overlord, Coordinator, Router, Broker, and Indexer counts on clusters where the web console talks to the Coordinator without SQL access [#19481](https://github.com/apache/druid/pull/19481)
+
### Ingestion
+* You can now use the expression aggregator at ingestion time for expressions that produce a LONG or DOUBLE for both fold and combine expressions [#19508](https://github.com/apache/druid/pull/19508)
+* Added `now()` expression function that returns the current system timestamp in milliseconds since epoch. Useful at ingestion time for troubleshooting pipeline delays (e.g., `now() - __time`). Note: `now()` is non-deterministic as it evaluates for every row, so it can break idempotency. This can be added to any column besides `__time` [#19386](https://github.com/apache/druid/pull/19386)
+* Improved resiliency when ingesting from S3. Druid now retries on `SSLException` and transient credential errors instead of failing [#19617](https://github.com/apache/druid/pull/19617) [#19558](https://github.com/apache/druid/pull/19558)
+* Improved S3 performance [#19394](https://github.com/apache/druid/pull/19394)
+* Updated the default S3 connection pool size so that it's computed based on the number of available processors [#19536](https://github.com/apache/druid/pull/19536)
+
#### SQL-based ingestion
+##### EXTERN for S3 now supports role ARN
+
+You can now include the role ARN when running an INSERT INTO EXTERN query. For example:
+
+```
+INSERT INTO
+EXTERN(
+ s3(bucket => 'test2', prefix => 'export', assumeRoleArn => 'arn:aws:iam::00000:role/test-20260520'))
+AS CSV
+SELECT ...
+```
+
+[#19317](https://github.com/apache/druid/pull/19317)
+
##### Other SQL-based ingestion improvements
+- Added storage counters for the amount of bytes and files written to local and durable storage as well as the state of the local `ByteTracker`. Per-worker storage counters have also been added to the web console [#19316](https://github.com/apache/druid/pull/19316)
+
#### Streaming ingestion
+##### Scaling cool down
+
+You can now configure different cool down durations for scaling up and scaling down streaming task autoscalers.
+
+[#19286](https://github.com/apache/druid/pull/19286)
+
+##### Improved supervisor restarts
+
+Supervisors no longer restart for all changes. Based on the type of change, one of the following can occur:
+
+- Druid persists the updated spec without a restart.
+- Druid restarts the supervisor, but running tasks aren't impacted.
+- Druid restarts the supervisor is restarted and its tasks are terminated, which was the default behavior before this change.
Review Comment:
```suggestion
- Druid restarts the supervisor, terminates the current tasks and launches fresh tasks (default behaviour in older Druid versions).
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] docs: 38.0.0 release notes (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #19945:
URL: https://github.com/apache/druid/pull/19945#discussion_r4155079280
##########
docs/release-info/release-notes.md:
##########
@@ -57,6 +57,76 @@ For tips about how to write a good release note, see [Release notes](https://git
This section contains important information about new and existing features.
+#### Java
+
+Druid now supports Java 25. While Java 21 is still supported, we recommend you upgrade to Java 25.
+
+Support for Java 17 has been dropped.
+
+[#19304](https://github.com/apache/druid/pull/19304) [#19336](https://github.com/apache/druid/pull/19336)
+
+#### Historical tier aliases
+
+You can map a virtual tier name to a number of real Historical tiers with the `historicalTierAliases` Coordinator dynamic configuration. This creates a single identifier for a group of Historical tiers. When a load/drop rule references the alias, the Coordinator replaces it with the actual tiers. For example, if you map the Historical tiers `hot_1` and `hot_2` to the alias `hot`, the rule `{"hot": 2}` loads 2 replicas of each onto `hot_1` and `hot_2`.
+
+[#19204](https://github.com/apache/druid/pull/19204) [#19667](https://github.com/apache/druid/pull/19667)
+
+#### New load rule types
+
+Adds a new family of retention rules, `loadPartialByPeriod`, `loadPartialByInterval`, `loadPartialForever`, laying the groundwork for partial loading of version 10 segment projections on Historicals.
+
+[#19374](https://github.com/apache/druid/pull/19374)
+
Review Comment:
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] docs: 38.0.0 release notes (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #19945:
URL: https://github.com/apache/druid/pull/19945#discussion_r4154623180
##########
docs/release-info/release-notes.md:
##########
@@ -57,6 +57,75 @@ For tips about how to write a good release note, see [Release notes](https://git
This section contains important information about new and existing features.
+#### Java
+
+Druid now supports Java 25. While Java 21 is still supported, we recommend you upgrade to Java 25.
+
+Support for Java 17 has been dropped.
+
+[#19304](https://github.com/apache/druid/pull/19304) [#19336](https://github.com/apache/druid/pull/19336)
+
+#### Historical tier aliases
+
+You can map a virtual tier name to a number of real Historical tiers with the `historicalTierAliases` Coordinator dynamic configuration. This creates a single identifier for a group of Historical tiers. When a load/drop rule references the alias, the Coordinator replaces it with the actual tiers. For example, if you map the Historical tiers `hot_1` and `hot_2` to the alias `hot`, the rule `{"hot": 2}` loads 2 replicas of each onto `hot_1` and `hot_2`.
+
+[#19204](https://github.com/apache/druid/pull/19204) [#19667](https://github.com/apache/druid/pull/19667)
+
+#### New load rule types
+
+Adds a new family of retention rules, `loadPartialByPeriod`, `loadPartialByInterval`, `loadPartialForever`, laying the groundwork for partial loading of version 10 segment projections on Historicals.
+
+[#19374](https://github.com/apache/druid/pull/19374)
+
+#### Realtime segments mode query context
+
+The `realtimeSegmentsOnly` query context parameter has been deprecated and replaced with `realtimeSegmentsMode`.
+
+You can set `realtimeSegmentsMode` to one of the following:
+
+- `include` (default): query all segments, including realtime segments
+- `exclude`: skip realtime segments for the query
+- `exclusive`: query only realtime segments
+
+This is useful when performing things like blue/green deployments and you only want to query new Historical replica autoscaling groups and not touch any "live" nodes (neither realtime nor historical).
+
+[#19486](https://github.com/apache/druid/pull/19486)
+
+#### Faster segment metadata search
+
+You can set `druid.segment.timeline.fastIntervalSearch` to `true` so that Druid uses an index based on interval trees to store that metadata in memory for faster identification and retrieval.
+
+This feature is off by default.
+
+[#19138](https://github.com/apache/druid/issues/19138) [#19850](https://github.com/apache/druid/issues/19850)
+
+#### Fetching files in the background
+
+The MSQ task engine now supports workers fetching input files from cloud storage asynchronously. This overlaps downloading with processing, which generally improves throughput when reading larger numbers of files. Otherwise, workers stream each file directly from cloud storage while processing it.
+
+This feature is on by default and is controlled by the `backgroundFetchExternalFiles` setting.
+
+[#19539](https://github.com/apache/druid/pull/19539)
+
+#### Segment prefetching for Dart
+
+Dart now supports the runtime property `druid.msq.dart.worker.segmentLoadAheadCount`, which controls the number of segments that Dart prefetches. If set greater than 0 for a worker, this setting becomes the default `segmentLoadAheadCount` value for the worker. If a query includes the `segmentLoadAheadCount` query context parameter, the query context takes precedence.
+
+[#19559](https://github.com/apache/druid/pull/19559)
+
+#### Improved Convert to SQL in web console
+
+The web console now supports converting streaming supervisors to SQL-based ingestion queries. This makes it easier for you to run a streaming task as a one-time batch ingestion.
+
+Select the **Convert supervisor to SQL** option from the **...** menu in the Query view. You can select from existing supervisors or provide JSON for a new one.
+
+[#19547](https://github.com/apache/druid/pull/19547)
+
+#### Deprecate legacy ZSTD frames
+
+Druid 38 will be the last major Druid release that supports reading from legacy ZSTD (version 0.4 to 0.7) frames. Support will be removed for these frames in future releases for both batch and streaming ingestion.
+
+The primary reason for this deprecation is that the underlying library `zstd-jni` removes decoding for legacy frames in version `1.5.7-13` onwards.
Review Comment:
```suggestion
The primary reason for this deprecation is that the underlying library `zstd-jni` removes decoding for legacy frames in version `1.5.7-13` onward.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] docs: 38.0.0 release notes (druid)
Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #19945:
URL: https://github.com/apache/druid/pull/19945#discussion_r4152439380
##########
docs/release-info/release-notes.md:
##########
@@ -57,6 +57,80 @@ For tips about how to write a good release note, see [Release notes](https://git
This section contains important information about new and existing features.
+#### Java
+
+Druid now supports Java 25. While Java 21 is still supported, we recommend you upgrade to Java 25.
+
+Support for Java 17 has been dropped.
+
+[#19304](https://github.com/apache/druid/pull/19304) [#19336](https://github.com/apache/druid/pull/19336)
+
+#### Historical tier aliases
+
+You can map a virtual tier name to a number of real Historical tiers with the `historicalTierAliases` Coordinator dynamic configuration. This creates a single identifier for a group of Historical tiers. When a load/drop rule references the alias, the Coordinator replaces it with the actual tiers. For example, if you map the Historical tiers `hot_1` and `hot_2` to the alias `hot`, the rule `{"hot": 2}` loads 2 replicas of each onto `hot_1` and `hot_2`.
+
+[#19204](https://github.com/apache/druid/pull/19204) [#19667](https://github.com/apache/druid/pull/19667)
+
+#### New load rule types
+
+Adds a new family of retention rules, `loadPartialByPeriod`, `loadPartialByInterval`, `loadPartialForever`, laying the groundwork for partial loading of version 10 segment projections on Historicals.
+
+[#19374](https://github.com/apache/druid/pull/19374)
+
+#### Realtime segments mode query context
+
+The `realtimeSegmentsOnly` query context parameter has been deprecated and replaced with `realtimeSegmentsMode`.
+
+You can set `realtimeSegmentsMode` to one of the following:
+
+- `include` (default): query all segments, including realtime segments
+- `exclude`: skip realtime segments for the query
+- `exclusive`: query only realtime segments
+
+This is useful when performing things like blue/green deployments and you only want to query new Historical replica autoscaling groups and not touch any "live" nodes (neither realtime nor historical).
+
+[#19486](https://github.com/apache/druid/pull/19486)
+
+#### Faster segment metadata search
+
+You can set `druid.segment.timeline.fastIntervalSearch` to `true` so that Druid uses an index based on interval trees to store that metadata in memory for faster identification and retrieval.
+
+This feature is off by default.
+
+[#19138](https://github.com/apache/druid/issues/19138) [#19850](https://github.com/apache/druid/issues/19850)
+
+#### Fetching files in the background
+
+The MSQ task engine now supports workers fetching input files from cloud storage asynchronously. This overlaps downloading with processing, which generally improves throughput when reading larger numbers of files. Otherwise, workers stream each file directly from cloud storage while processing it.
+
+This feature is on by default and is controlled by the `backgroundFetchExternalFiles` setting.
+
+[#19539](https://github.com/apache/druid/pull/19539)
+
+#### Segment prefetching for Dart
+
+Dart now supports the runtime property `druid.msq.dart.worker.segmentLoadAheadCount`, which controls the number of segments that Dart prefetches. If set greater than 0 for a worker, this setting becomes the default `segmentLoadAheadCount` value for the worker. If a query includes the `segmentLoadAheadCount` query context parameter, the query context takes precedence.
+
+[#19559](https://github.com/apache/druid/pull/19559)
+
+#### Partial segment loading
+
+[#19620](https://github.com/apache/druid/pull/19620)
+[#19535](https://github.com/apache/druid/pull/19535)
+
+#### Clustered segments
Review Comment:
its probably worth mentioning that it only supports v10 segments, which are not really documented either. I think the tricky part with announcing some of this stuff is that probably the only way to use them is to look at PRs or look at the code since none of it is documented yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org