You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "techdocsmith (via GitHub)" <gi...@apache.org> on 2026/04/10 22:29:26 UTC
Re: [PR] doc: druid 37 release notes (druid)
techdocsmith commented on code in PR #19287:
URL: https://github.com/apache/druid/pull/19287#discussion_r3066745202
##########
docs/release-info/release-notes.md:
##########
@@ -57,50 +57,551 @@ For tips about how to write a good release note, see [Release notes](https://git
This section contains important information about new and existing features.
+### Hadoop-based ingestion
+
+Support for Hadoop-based ingestion has been removed. The feature was deprecated in Druid 34.
+
+Use one of Druid's other supported ingestion methods, such as SQL-based ingestion or MiddleManager-less ingestion using Kubernetes.
+
+[#19109](https://github.com/apache/druid/pull/19109)
+
+### Query blocklist
+
+You can now create a query blocklist for dynamically blocking queries without restarting your deployment. You can block queries by datasource, query type, or query context using the new `/druid/coordinator/v1/config/broker` API. Block rules use `AND` logic, which means all criteria must match. Druid stores the rules in the metadata database.
+
+The following example blocks all groupBy queries on the `wikipedia` datasource with a query context parameter of `priority` equal to `0`:
+
+```
+POST /druid/coordinator/v1/config/broker
+ {
+ "queryBlocklist": [
+ {
+ "ruleName": "block-wikipedia-groupbys",
+ "dataSources": ["wikipedia"],
+ "queryTypes": ["groupBy"],
+ "contextMatches": {"priority": "0"}
+ }
+ ]
+ }
+```
+
+[#19011](https://github.com/apache/druid/pull/19011)
+
+### Minor compaction for Overlord-based compaction (experimental)
+
+You can now configure minor compaction. With minor compaction, Druid only compacts newly ingested segments while upgrading existing compacted segments. When Druid upgrades segments, it updates the metadata instead of using resources to compact it again. You can use the native compaction engine or the MSQ task engine.
+
+Use the `mostFragmentedFirst` compaction policy and set either a percentage of rows-based or byte-based threshold for minor compaction.
+
+[#19059](https://github.com/apache/druid/pull/19059) [#19205](https://github.com/apache/druid/pull/19205) [#19016](https://github.com/apache/druid/pull/19016)
+
+### Thrift input format
+
+As part of the Thrift contributor extension, Druid now supports Thrift-encoded data for Kafka and Kinesis streaming ingestion.
+
+[#19111](https://github.com/apache/druid/pull/19111)
+
+To use this feature, you must add `druid-thrift-extensions` to your extension load list.
+
+### Kubernetes-based task management
+
+This extension is now generally available.
+
+[#19128](https://github.com/apache/druid/pull/19128)
+
+### Tombstones
+
+Tombstones for JSON-based native batch ingestion (the `dropExisting` flag for `ioConfig`) are now generally available.
+
+[#19128](https://github.com/apache/druid/pull/19128)
+
+### Dynamic default query context
+
+You can now add default query context parameters as a dynamic configuration to the Broker. This allows you to override static defaults set in your runtime properties without restarting your deployment or having to update multiple queries individually. Druid resolves which query context parameters to use based on the following priority:
Review Comment:
```suggestion
You can now add default query context parameters as a dynamic configuration to the Broker. This allows you to override static defaults set in your runtime properties without restarting your deployment or having to update multiple queries individually. Druid applies query context parameters based on the following priority:
```
##########
docs/release-info/release-notes.md:
##########
@@ -57,50 +57,551 @@ For tips about how to write a good release note, see [Release notes](https://git
This section contains important information about new and existing features.
+### Hadoop-based ingestion
+
+Support for Hadoop-based ingestion has been removed. The feature was deprecated in Druid 34.
+
+Use one of Druid's other supported ingestion methods, such as SQL-based ingestion or MiddleManager-less ingestion using Kubernetes.
+
+[#19109](https://github.com/apache/druid/pull/19109)
+
+### Query blocklist
+
+You can now create a query blocklist for dynamically blocking queries without restarting your deployment. You can block queries by datasource, query type, or query context using the new `/druid/coordinator/v1/config/broker` API. Block rules use `AND` logic, which means all criteria must match. Druid stores the rules in the metadata database.
+
+The following example blocks all groupBy queries on the `wikipedia` datasource with a query context parameter of `priority` equal to `0`:
+
+```
+POST /druid/coordinator/v1/config/broker
+ {
+ "queryBlocklist": [
+ {
+ "ruleName": "block-wikipedia-groupbys",
+ "dataSources": ["wikipedia"],
+ "queryTypes": ["groupBy"],
+ "contextMatches": {"priority": "0"}
+ }
+ ]
+ }
+```
+
+[#19011](https://github.com/apache/druid/pull/19011)
+
+### Minor compaction for Overlord-based compaction (experimental)
+
+You can now configure minor compaction. With minor compaction, Druid only compacts newly ingested segments while upgrading existing compacted segments. When Druid upgrades segments, it updates the metadata instead of using resources to compact it again. You can use the native compaction engine or the MSQ task engine.
+
+Use the `mostFragmentedFirst` compaction policy and set either a percentage of rows-based or byte-based threshold for minor compaction.
+
+[#19059](https://github.com/apache/druid/pull/19059) [#19205](https://github.com/apache/druid/pull/19205) [#19016](https://github.com/apache/druid/pull/19016)
+
+### Thrift input format
+
+As part of the Thrift contributor extension, Druid now supports Thrift-encoded data for Kafka and Kinesis streaming ingestion.
+
+[#19111](https://github.com/apache/druid/pull/19111)
+
+To use this feature, you must add `druid-thrift-extensions` to your extension load list.
+
+### Kubernetes-based task management
+
+This extension is now generally available.
+
+[#19128](https://github.com/apache/druid/pull/19128)
+
+### Tombstones
+
+Tombstones for JSON-based native batch ingestion (the `dropExisting` flag for `ioConfig`) are now generally available.
+
+[#19128](https://github.com/apache/druid/pull/19128)
+
+### Dynamic default query context
+
+You can now add default query context parameters as a dynamic configuration to the Broker. This allows you to override static defaults set in your runtime properties without restarting your deployment or having to update multiple queries individually. Druid resolves which query context parameters to use based on the following priority:
+
+1. The query context included with the query
+1. The query context set as a dynamic configuration on the Broker
+1. The query context parameters set in the runtime properties
+1. The defaults that ship with Druid
+
+Note that like other Broker dynamic configuration, this is best-effort. Settings may not be applied in certain
+cases, such as when a Broker has recently started and hasn't received the config yet, or if the
+Broker can't contact the Coordinator. If a query context parameter is critical for all your queries, set it in the runtime properties.
+
+[#19146](https://github.com/apache/druid/pull/19146)
+
+### `sys.queries` table (experimental)
+
+The new system queries table provides information about currently running and recently completed queries that use the Dart engine. This table is off by default. To enable the table, set the following config:
Review Comment:
```suggestion
The new system queries table provides information about currently running and recently completed queries that use the Dart engine. This table is off by default. To enable the table, set the following:
```
##########
docs/release-info/release-notes.md:
##########
@@ -57,50 +57,551 @@ For tips about how to write a good release note, see [Release notes](https://git
This section contains important information about new and existing features.
+### Hadoop-based ingestion
+
+Support for Hadoop-based ingestion has been removed. The feature was deprecated in Druid 34.
+
+Use one of Druid's other supported ingestion methods, such as SQL-based ingestion or MiddleManager-less ingestion using Kubernetes.
+
+[#19109](https://github.com/apache/druid/pull/19109)
+
+### Query blocklist
+
+You can now create a query blocklist for dynamically blocking queries without restarting your deployment. You can block queries by datasource, query type, or query context using the new `/druid/coordinator/v1/config/broker` API. Block rules use `AND` logic, which means all criteria must match. Druid stores the rules in the metadata database.
+
+The following example blocks all groupBy queries on the `wikipedia` datasource with a query context parameter of `priority` equal to `0`:
+
+```
+POST /druid/coordinator/v1/config/broker
+ {
+ "queryBlocklist": [
+ {
+ "ruleName": "block-wikipedia-groupbys",
+ "dataSources": ["wikipedia"],
+ "queryTypes": ["groupBy"],
+ "contextMatches": {"priority": "0"}
+ }
+ ]
+ }
+```
+
+[#19011](https://github.com/apache/druid/pull/19011)
+
+### Minor compaction for Overlord-based compaction (experimental)
+
+You can now configure minor compaction. With minor compaction, Druid only compacts newly ingested segments while upgrading existing compacted segments. When Druid upgrades segments, it updates the metadata instead of using resources to compact it again. You can use the native compaction engine or the MSQ task engine.
+
+Use the `mostFragmentedFirst` compaction policy and set either a percentage of rows-based or byte-based threshold for minor compaction.
+
+[#19059](https://github.com/apache/druid/pull/19059) [#19205](https://github.com/apache/druid/pull/19205) [#19016](https://github.com/apache/druid/pull/19016)
+
+### Thrift input format
+
+As part of the Thrift contributor extension, Druid now supports Thrift-encoded data for Kafka and Kinesis streaming ingestion.
+
+[#19111](https://github.com/apache/druid/pull/19111)
+
+To use this feature, you must add `druid-thrift-extensions` to your extension load list.
+
+### Kubernetes-based task management
+
+This extension is now generally available.
+
+[#19128](https://github.com/apache/druid/pull/19128)
+
+### Tombstones
+
+Tombstones for JSON-based native batch ingestion (the `dropExisting` flag for `ioConfig`) are now generally available.
+
+[#19128](https://github.com/apache/druid/pull/19128)
+
+### Dynamic default query context
+
+You can now add default query context parameters as a dynamic configuration to the Broker. This allows you to override static defaults set in your runtime properties without restarting your deployment or having to update multiple queries individually. Druid resolves which query context parameters to use based on the following priority:
+
+1. The query context included with the query
+1. The query context set as a dynamic configuration on the Broker
+1. The query context parameters set in the runtime properties
+1. The defaults that ship with Druid
+
+Note that like other Broker dynamic configuration, this is best-effort. Settings may not be applied in certain
+cases, such as when a Broker has recently started and hasn't received the config yet, or if the
+Broker can't contact the Coordinator. If a query context parameter is critical for all your queries, set it in the runtime properties.
+
+[#19146](https://github.com/apache/druid/pull/19146)
+
+### `sys.queries` table (experimental)
+
+The new system queries table provides information about currently running and recently completed queries that use the Dart engine. This table is off by default. To enable the table, set the following config:
+
+```
+druid.sql.planner.enableSysQueriesTable = true
+```
+
+As part of this change, the `/druid/v2/sql/queries` API now supports an `includeComplete` parameter that shows recently completed queries.
+
+[#18923](https://github.com/apache/druid/pull/18923)
+
+### Auto-compaction with compaction supervisors
+
+Auto-compaction using compaction supervisors has been improved and is now generally available.
+
+As part of the improvement, instead of individual compacted segments having to store their full compaction state in the metadata store, states are now stored in a central location, a new `indexingStates` table, in the metadata store. Individual segments only need to store a unique reference (`indexing_state_fingerprint`) to their full compaction state.
+
+Since many segments in a single datasource share the same underlying compaction state, this greatly reduces metadata storage requirements for automatic compaction.
+
+For backwards compatibility, detailed compaction state will continue to be persisted in each segment for now. This will be removed in a future release.
Review Comment:
```suggestion
For backwards compatibility, Druid continues to persist the detailed compaction state in each segment. This functionality will be removed in a future release.
```
##########
docs/release-info/release-notes.md:
##########
@@ -57,50 +57,551 @@ For tips about how to write a good release note, see [Release notes](https://git
This section contains important information about new and existing features.
+### Hadoop-based ingestion
+
+Support for Hadoop-based ingestion has been removed. The feature was deprecated in Druid 34.
+
+Use one of Druid's other supported ingestion methods, such as SQL-based ingestion or MiddleManager-less ingestion using Kubernetes.
+
+[#19109](https://github.com/apache/druid/pull/19109)
+
+### Query blocklist
+
+You can now create a query blocklist for dynamically blocking queries without restarting your deployment. You can block queries by datasource, query type, or query context using the new `/druid/coordinator/v1/config/broker` API. Block rules use `AND` logic, which means all criteria must match. Druid stores the rules in the metadata database.
+
+The following example blocks all groupBy queries on the `wikipedia` datasource with a query context parameter of `priority` equal to `0`:
+
+```
+POST /druid/coordinator/v1/config/broker
+ {
+ "queryBlocklist": [
+ {
+ "ruleName": "block-wikipedia-groupbys",
+ "dataSources": ["wikipedia"],
+ "queryTypes": ["groupBy"],
+ "contextMatches": {"priority": "0"}
+ }
+ ]
+ }
+```
+
+[#19011](https://github.com/apache/druid/pull/19011)
+
+### Minor compaction for Overlord-based compaction (experimental)
+
+You can now configure minor compaction. With minor compaction, Druid only compacts newly ingested segments while upgrading existing compacted segments. When Druid upgrades segments, it updates the metadata instead of using resources to compact it again. You can use the native compaction engine or the MSQ task engine.
+
+Use the `mostFragmentedFirst` compaction policy and set either a percentage of rows-based or byte-based threshold for minor compaction.
+
+[#19059](https://github.com/apache/druid/pull/19059) [#19205](https://github.com/apache/druid/pull/19205) [#19016](https://github.com/apache/druid/pull/19016)
+
+### Thrift input format
+
+As part of the Thrift contributor extension, Druid now supports Thrift-encoded data for Kafka and Kinesis streaming ingestion.
+
+[#19111](https://github.com/apache/druid/pull/19111)
+
+To use this feature, you must add `druid-thrift-extensions` to your extension load list.
+
+### Kubernetes-based task management
+
+This extension is now generally available.
+
+[#19128](https://github.com/apache/druid/pull/19128)
+
+### Tombstones
+
+Tombstones for JSON-based native batch ingestion (the `dropExisting` flag for `ioConfig`) are now generally available.
+
+[#19128](https://github.com/apache/druid/pull/19128)
+
+### Dynamic default query context
+
+You can now add default query context parameters as a dynamic configuration to the Broker. This allows you to override static defaults set in your runtime properties without restarting your deployment or having to update multiple queries individually. Druid resolves which query context parameters to use based on the following priority:
+
+1. The query context included with the query
+1. The query context set as a dynamic configuration on the Broker
+1. The query context parameters set in the runtime properties
+1. The defaults that ship with Druid
+
+Note that like other Broker dynamic configuration, this is best-effort. Settings may not be applied in certain
+cases, such as when a Broker has recently started and hasn't received the config yet, or if the
Review Comment:
```suggestion
cases, such as when a Broker has recently started and hasn't received the configuration yet, or if the
```
##########
docs/release-info/release-notes.md:
##########
@@ -57,50 +57,551 @@ For tips about how to write a good release note, see [Release notes](https://git
This section contains important information about new and existing features.
+### Hadoop-based ingestion
+
+Support for Hadoop-based ingestion has been removed. The feature was deprecated in Druid 34.
+
+Use one of Druid's other supported ingestion methods, such as SQL-based ingestion or MiddleManager-less ingestion using Kubernetes.
+
+[#19109](https://github.com/apache/druid/pull/19109)
+
+### Query blocklist
+
+You can now create a query blocklist for dynamically blocking queries without restarting your deployment. You can block queries by datasource, query type, or query context using the new `/druid/coordinator/v1/config/broker` API. Block rules use `AND` logic, which means all criteria must match. Druid stores the rules in the metadata database.
+
+The following example blocks all groupBy queries on the `wikipedia` datasource with a query context parameter of `priority` equal to `0`:
+
+```
+POST /druid/coordinator/v1/config/broker
+ {
+ "queryBlocklist": [
+ {
+ "ruleName": "block-wikipedia-groupbys",
+ "dataSources": ["wikipedia"],
+ "queryTypes": ["groupBy"],
+ "contextMatches": {"priority": "0"}
+ }
+ ]
+ }
+```
+
+[#19011](https://github.com/apache/druid/pull/19011)
+
+### Minor compaction for Overlord-based compaction (experimental)
+
+You can now configure minor compaction. With minor compaction, Druid only compacts newly ingested segments while upgrading existing compacted segments. When Druid upgrades segments, it updates the metadata instead of using resources to compact it again. You can use the native compaction engine or the MSQ task engine.
+
+Use the `mostFragmentedFirst` compaction policy and set either a percentage of rows-based or byte-based threshold for minor compaction.
+
+[#19059](https://github.com/apache/druid/pull/19059) [#19205](https://github.com/apache/druid/pull/19205) [#19016](https://github.com/apache/druid/pull/19016)
+
+### Thrift input format
+
+As part of the Thrift contributor extension, Druid now supports Thrift-encoded data for Kafka and Kinesis streaming ingestion.
+
+[#19111](https://github.com/apache/druid/pull/19111)
+
+To use this feature, you must add `druid-thrift-extensions` to your extension load list.
+
+### Kubernetes-based task management
+
+This extension is now generally available.
+
+[#19128](https://github.com/apache/druid/pull/19128)
+
+### Tombstones
+
+Tombstones for JSON-based native batch ingestion (the `dropExisting` flag for `ioConfig`) are now generally available.
+
+[#19128](https://github.com/apache/druid/pull/19128)
+
+### Dynamic default query context
+
+You can now add default query context parameters as a dynamic configuration to the Broker. This allows you to override static defaults set in your runtime properties without restarting your deployment or having to update multiple queries individually. Druid resolves which query context parameters to use based on the following priority:
+
+1. The query context included with the query
+1. The query context set as a dynamic configuration on the Broker
+1. The query context parameters set in the runtime properties
+1. The defaults that ship with Druid
+
+Note that like other Broker dynamic configuration, this is best-effort. Settings may not be applied in certain
+cases, such as when a Broker has recently started and hasn't received the config yet, or if the
+Broker can't contact the Coordinator. If a query context parameter is critical for all your queries, set it in the runtime properties.
+
+[#19146](https://github.com/apache/druid/pull/19146)
+
+### `sys.queries` table (experimental)
+
+The new system queries table provides information about currently running and recently completed queries that use the Dart engine. This table is off by default. To enable the table, set the following config:
+
+```
+druid.sql.planner.enableSysQueriesTable = true
+```
+
+As part of this change, the `/druid/v2/sql/queries` API now supports an `includeComplete` parameter that shows recently completed queries.
+
+[#18923](https://github.com/apache/druid/pull/18923)
+
+### Auto-compaction with compaction supervisors
+
+Auto-compaction using compaction supervisors has been improved and is now generally available.
+
+As part of the improvement, instead of individual compacted segments having to store their full compaction state in the metadata store, states are now stored in a central location, a new `indexingStates` table, in the metadata store. Individual segments only need to store a unique reference (`indexing_state_fingerprint`) to their full compaction state.
+
+Since many segments in a single datasource share the same underlying compaction state, this greatly reduces metadata storage requirements for automatic compaction.
+
+For backwards compatibility, detailed compaction state will continue to be persisted in each segment for now. This will be removed in a future release.
+
+You can stop storing detailed compaction state by setting `storeCompactionStatePerSegment` to `false` in the cluster compaction config. If you turn it off and need to downgrade, Druid needs to re-compact any segments that have been compacted since you changed the config.
+
+This change has upgrade impacts for metadata storage and metadata caching. For more information, see the [Metadata storage for auto-compaction with compaction supervisors](#metadata-storage-for-auto-compaction-with-compaction-supervisors) upgrade note.
+
+[#19113](https://github.com/apache/druid/pull/19113) [#18844](https://github.com/apache/druid/pull/18844)
+
+### Broker tier selection for realtime servers
+
+Added `druid.broker.realtime.select.tier` and `druid.broker.realtime.balancer.type` on the Brokers to optionally override the Broker’s tier selection and balancer strategies for realtime servers. If these properties are not set (the default), realtime servers continue to use the existing `druid.broker.select` and `druid.broker.balancer` configurations that apply to both historical and realtime servers.
+
+[#19062](https://github.com/apache/druid/pull/19062)
+
+### Manual Broker routing in the web console
+
+You can now configure which Broker the Router uses for queries issued from the web console. You may want to do this if there are Brokers that don't have visibility into certain data tiers, and you know you're querying data available only on a certain tier.
+
+To specify a Broker, add the following config to `web-console/console-config.js`:
+
+```js
+consoleBrokerService: 'druid/BROKER_NAME'
+```
+
+[#19069](https://github.com/apache/druid/pull/19069)
+
+### Consul extension
+
+The contributor extension `druid-consul-extensions` lets Druid clusters use Consul for service discovery and
+Coordinator/Overlord leader election instead of ZooKeeper. The extension supports ACLs, TLS/mTLS, and metrics.
+
+Before you switch to Consul, you need to set
+`druid.serverview.type=http` and `druid.indexer.runner.type=httpRemote` cluster wide.
+
+[#18843](https://github.com/apache/druid/pull/18843)
+
## Functional area and related changes
This section contains detailed release notes separated by areas.
### Web console
+#### Changed storage column displays
+
+- Improved the compaction config view to
+- Renamed **Current size** to **Assigned size**.
+- Renamed **Max size** to **Effective size**. It now displays the smaller value between `max_size` and `storage_size`. The max size is still shown as a tooltip.
+- Changed usage calculation to use `effective_size`
+
+[#19007](https://github.com/apache/druid/pull/19007)
+
#### Other web console improvements
+- Added `workerDesc` to `WorkerStats`, which makes it easier to identify where a worker is running [#19171](https://github.com/apache/druid/pull/19171)
+- Added the Dart unique execution ID (`dartQueryId`) and the `sqlQueryId` to the **Details** pane in the web console [#19185](https://github.com/apache/druid/pull/19185)
+- Added support for showing completed Dart queries in the web console [#18940](https://github.com/apache/druid/pull/18940)
+- Added a detail dialog to the **Services** page [#18960](https://github.com/apache/druid/pull/18960)
+- Added icons to indicate when data is loaded into virtual storage, including a tooltip that shows all the counters for the data [#19010](https://github.com/apache/druid/pull/19010)
+- Added support for Dart reports [#18897](https://github.com/apache/druid/pull/18897)
+- Changed what's considered an active worker: any nonzero rows, files, bytes, frames, or wall time is enough to consider a worker active [#19183](https://github.com/apache/druid/pull/19183)
+- Changed the **Cancel query** option to show only if a query is in an accepted or running state [#19182](https://github.com/apache/druid/pull/19182)
+- Changed the ordering of the current Dart queries panel to show queries in the following order: RUNNING, ACCEPTED, and then COMPLETED. RUNNING and ACCEPTED queries are ordered by the most recent first (based on timestamp). COMPLETED queries are sorted by finish time [#19237](https://github.com/apache/druid/pull/19237)
+
### Ingestion
#### SQL-based ingestion
##### Other SQL-based ingestion improvements
+- Added support for virtual storage fabric when performing SQL-based ingestion [#18873](https://github.com/apache/druid/pull/18873)
+- Added support for `StorageMonitor` so that MSQ task engine tasks always emit `taskId` and `groupId` [#19048](https://github.com/apache/druid/pull/19048)
+- Improved worker cancellation [#18931](https://github.com/apache/druid/pull/18931)
+- Improved exception handling [#19234](https://github.com/apache/druid/pull/19234)
+
#### Streaming ingestion
+##### Changed how tasks get launched for autoscaling
+
+The behavior of `taskCountMin` and `taskCountStart` has been changed for autoscaling. Druid now computes the initial number of tasks to launch by checking the configs in the following order: `taskCountStart` (optional), then `taskCount` (in `ioConfig`), then `taskCountMin`.
+
+[#19091](https://github.com/apache/druid/pull/19091)
+
+##### Improved query isolation
+
+Added `serverPriorityToReplicas` parameter to the streaming supervisor specs (Kafka, Kinesis, and Rabbit). This allows operators to distribute task replicas across different server priorities for realtime indexing tasks. Similar to Historical tiering, this enables query isolation for mixed workload scenarios on the Peons, allowing some task replicas to handle queries of specific priorities.
+
+[#19040](https://github.com/apache/druid/pull/19040)
+
##### Other streaming ingestion improvements
+- Improved cost-based autoscaler performance in high lag scenarios [#19045](https://github.com/apache/druid/pull/19045)
+- Improved the performance of realtime task scheduling by ordering schedule requests by priority on the `TaskQueue` [#19203](https://github.com/apache/druid/pull/19203)
+
+
+#### Ingestion improvements
+
+- Sped up task scheduling on the Overlord [#19199](https://github.com/apache/druid/pull/19199)
+
### Querying
+#### groupBy query configuration
+
+Added a new groupBy query configuration property `druid.query.groupBy.maxSpillFileCount` to limit the maximum number of spill files created per query. When the limit is exceeded, the query fails with a clear error message instead of causing Historical nodes to run out of memory during spill file merging. The limit can also be overridden per query using the query context parameter `maxSpillFileCount`.
+
+[#19141](https://github.com/apache/druid/pull/19141)
+
+#### Improved handling of nested aggregates
+
+Druid can now merge two Aggregates with a Projection between them. For example, the following query:
+
+```sql
+SELECT
+ hr,
+ UPPER(t1.x) x,
+ SUM(t1.cnt) cnt,
+ MIN(t1.mn) mn,
+ MAX(t1.mx) mx
+FROM (
+ SELECT
+ floor(__time to hour) hr,
+ dim2 x,
+ COUNT(*) cnt,
+ MIN(m1 * 5) mn,
+ MAX(m1 + m2) mx
+ FROM druid.foo
+ WHERE dim2 IN ('abc', 'def', 'a', 'b', '')
+ GROUP BY 1, 2
+) t1
+WHERE t1.x IN ('abc', 'foo', 'bar', 'a', '')
+GROUP BY 1, 2
+```
+
+can be simplified to the following:
+
+```sql
+SELECT
+ FLOOR(__time TO hour) hr,
+ UPPER(dim2) x,
+ COUNT(*) cnt,
+ MIN(m1 * 5) mn,
+ MAX(m1 + m2) mx
+FROM druid.foo
+WHERE dim2 IN ('abc', 'a', '')
+GROUP BY 1, 2
+```
+
+[#18498](https://github.com/apache/druid/pull/18498)
+
#### Other querying improvements
+- Added `durationMs` to Dart query reports [#19169](https://github.com/apache/druid/pull/19169)
+- Improved error handling so that row signature column order is preserved when column analysis encounters an error [#19162](https://github.com/apache/druid/pull/19162)
+- Improved GROUP BY performance [#18952](https://github.com/apache/druid/pull/18952)
+- Improved expression filters to take advantage of specialized virtual columns when possible, resulting in better performance for the query [#18965](https://github.com/apache/druid/pull/18965)
+
### Cluster management
+#### New Broker tier selection strategies
+
+Operators can now configure two new Broker `TierSelectorStrategy` implementations:
+
+- `strict` - Only selects servers whose priorities match the configured list. Example configuration: `druid.broker.select.tier=strict` and `druid.broker.select.tier.strict.priorities=[1]`.
+- `pooled` - Pools servers across the configured priorities and selects among them, allowing queries to utilize multiple priority tiers for improved availability. Example configuration: `druid.broker.select.tier=pooled` and `druid.broker.select.tier.pooled.priorities=[2,1]`.
+
+These strategies can also be configured for realtime servers using `druid.broker.realtime.select.tier`.
+
+[#19094](https://github.com/apache/druid/pull/19094)
+
+#### Druid operator
+
+The Druid operator now resides in its own repository: [`apache/druid-operator`](https://github.com/apache/druid-operator).
+
+[#19156](https://github.com/apache/druid/pull/19156)
+
+#### Cost-based autoscaler algorithm
+
+The algorithm for cost-based autoscaling has been changed:
+
+- Scale up more aggressively when per-partition lag is meaningful
+- Relax the partitions-per-task increase limit based on lag severity and headroom
+- Keep behavior conservative near `taskCountMax` and avoid negative headroom effects
+
+[#18936](https://github.com/apache/druid/pull/18936)
+
#### Other cluster management improvements
+- Added a `ReadOnly` authorizer that allows all READ operations but denies any other operation, such as WRITE [#19243](https://github.com/apache/druid/pull/19243)
+- Added `/status/ready` endpoint for service health so that external load balancers can handle a graceful shutdown better [#19148](https://github.com/apache/druid/pull/19148)
+- Added a configurable option to scale-down during task run time for cost-based autoscaler [#18958](https://github.com/apache/druid/pull/18958)
+- Added `storage_size` to `sys.servers` to facilitate retrieving disk cache size for Historicals when using the virtual storage fabric [#18979](https://github.com/apache/druid/pull/18979)
+- Added a log for new task count computation for the cost-based auto scaler [#18929](https://github.com/apache/druid/pull/18929)
+- Changed how the scaling is calculated from a square root-based scaling formula to a logarithmic formula that provides better emergency recovery at low task counts and millions of lag [#18976](https://github.com/apache/druid/pull/18976)
+- Improved the load speed of cached segments during Historical startup [#18489](https://github.com/apache/druid/pull/18489)
+- Improved Broker startup time by parallelizing buffer initialization [#19025](https://github.com/apache/druid/pull/19025)
+- Improved the stack trace for MSQ task engine worker failures so that they're preserved [#19049](https://github.com/apache/druid/pull/19049)
+- Improved the performance of the cost-based autoscaler during loaded lag conditions [#18991](https://github.com/apache/druid/pull/18991)
+
### Data management
+#### Durable storage cleaner
+
+The durable storage cleaner now supports configurable time-based retention for MSQ query results. Previously, query results were retained for all known tasks, which was unreliable for completed tasks. With this change, query results are retained for a configurable time period based on the task creation time.
+
+The new configuration property `druid.msq.intermediate.storage.cleaner.durationToRetain` controls the retention period for query results. The default retention period is 6 hours.
+
+[#19074](https://github.com/apache/druid/pull/19074)
+
#### Other data management improvements
+- Added the `druid.storage.transfer.asyncHttpClientType` config that specifies which async HTTP client to use for S3 transfers: `crt` for Amazon CRT or `netty` for Netty NIO [#19249](https://github.com/apache/druid/pull/19249)
+- Added a mechanism to automatically clean up intermediary files on HDFS storage [#19187](https://github.com/apache/druid/pull/19187)
+- Changed the default value for `druid.indexing.formats.maxStringLength` to null from 0 [#19198](https://github.com/apache/druid/pull/19198)
+
### Metrics and monitoring
+#### Version field
+
+All Druid metrics now include `version` and `buildRevision` fields to help you identify which version of Druid is emitting a metric. You can use this information to verify that all nodes in a cluster are running the intended revision during rolling deployments.
+
+[#19123](https://github.com/apache/druid/pull/19123)
+
+#### Monitoring supervisor state
+
+Added a new `supervisor/count` metric when `SupervisorStatsMonitor` is enabled in `druid.monitoring.monitors`. The metric reports each supervisor’s state, such as `RUNNING` or `SUSPENDED`, for Prometheus, StatsD, and other metric systems.
+
+[#19114](https://github.com/apache/druid/pull/19114)
+
+#### Improved `groupBy` metrics
+
+`GroupByStatsMonitor` now provides the following metrics:
+
+- `mergeBuffer/bytesUsed`
+- `mergeBuffer/maxAcquisitionTimeNs`
+- `groupBy/maxSpilledBytes`
+- `groupBy/maxMergeDictionarySize`
+
+[#18731](https://github.com/apache/druid/pull/18731) [#18970](https://github.com/apache/druid/pull/18970) [#18934](https://github.com/apache/druid/pull/18934)
+
+#### Filtering metrics
+
+Operators can limit which metrics the logging emitter writes by setting `druid.emitter.logging.shouldFilterMetrics=true` and, if desired, `druid.emitter.logging.allowedMetricsPath` to a JSON object file where the keys are metric names. A missing custom file results in a warning and use of the bundled `defaultMetrics.json`. Alerts and other non-metric events are always logged.
+
+[#19030](https://github.com/apache/druid/pull/19030)
+
+#### New Broker metrics
+
+Added `segment/schemaCache/rowSignature/changed` and `segment/schemaCache/rowSignature/column/count` metrics to get visibility into when the Broker's segment metadata cache's row signature for each datasource is initialized and updated.
+
+[#18966](https://github.com/apache/druid/pull/18966)
+
+#### Other metrics and monitoring improvements
+
+- Added the following metrics to the default for Prometheus: `mergeBuffer/bytesUsed` and `mergeBuffer/maxBytesUsed` [#19110](https://github.com/apache/druid/pull/19110)
+- Added compaction mode to what gets emitted for the `compact/task/count` metric [#19151](https://github.com/apache/druid/pull/19151)
+- Added support for logging and emitting SQL dynamic parameter values [#19067](https://github.com/apache/druid/pull/19067)
+- Added `ingest/rows/published`, which gets emitted from all task types to denote the total row count of successfully published segments [#19177](https://github.com/apache/druid/pull/19177)
+- Added `queries` and `totalQueries` counters, which reflect queries made to realtime servers to get realtime data
Review Comment:
```suggestion
- Added `queries` and `totalQueries` counters, which reflect queries made to realtime servers to retrieve realtime data
```
##########
docs/release-info/release-notes.md:
##########
@@ -57,50 +57,551 @@ For tips about how to write a good release note, see [Release notes](https://git
This section contains important information about new and existing features.
+### Hadoop-based ingestion
+
+Support for Hadoop-based ingestion has been removed. The feature was deprecated in Druid 34.
+
+Use one of Druid's other supported ingestion methods, such as SQL-based ingestion or MiddleManager-less ingestion using Kubernetes.
+
+[#19109](https://github.com/apache/druid/pull/19109)
+
+### Query blocklist
+
+You can now create a query blocklist for dynamically blocking queries without restarting your deployment. You can block queries by datasource, query type, or query context using the new `/druid/coordinator/v1/config/broker` API. Block rules use `AND` logic, which means all criteria must match. Druid stores the rules in the metadata database.
+
+The following example blocks all groupBy queries on the `wikipedia` datasource with a query context parameter of `priority` equal to `0`:
+
+```
+POST /druid/coordinator/v1/config/broker
+ {
+ "queryBlocklist": [
+ {
+ "ruleName": "block-wikipedia-groupbys",
+ "dataSources": ["wikipedia"],
+ "queryTypes": ["groupBy"],
+ "contextMatches": {"priority": "0"}
+ }
+ ]
+ }
+```
+
+[#19011](https://github.com/apache/druid/pull/19011)
+
+### Minor compaction for Overlord-based compaction (experimental)
+
+You can now configure minor compaction. With minor compaction, Druid only compacts newly ingested segments while upgrading existing compacted segments. When Druid upgrades segments, it updates the metadata instead of using resources to compact it again. You can use the native compaction engine or the MSQ task engine.
+
+Use the `mostFragmentedFirst` compaction policy and set either a percentage of rows-based or byte-based threshold for minor compaction.
+
+[#19059](https://github.com/apache/druid/pull/19059) [#19205](https://github.com/apache/druid/pull/19205) [#19016](https://github.com/apache/druid/pull/19016)
+
+### Thrift input format
+
+As part of the Thrift contributor extension, Druid now supports Thrift-encoded data for Kafka and Kinesis streaming ingestion.
+
+[#19111](https://github.com/apache/druid/pull/19111)
+
+To use this feature, you must add `druid-thrift-extensions` to your extension load list.
+
+### Kubernetes-based task management
+
+This extension is now generally available.
+
+[#19128](https://github.com/apache/druid/pull/19128)
+
+### Tombstones
+
+Tombstones for JSON-based native batch ingestion (the `dropExisting` flag for `ioConfig`) are now generally available.
+
+[#19128](https://github.com/apache/druid/pull/19128)
+
+### Dynamic default query context
+
+You can now add default query context parameters as a dynamic configuration to the Broker. This allows you to override static defaults set in your runtime properties without restarting your deployment or having to update multiple queries individually. Druid resolves which query context parameters to use based on the following priority:
+
+1. The query context included with the query
+1. The query context set as a dynamic configuration on the Broker
+1. The query context parameters set in the runtime properties
+1. The defaults that ship with Druid
+
+Note that like other Broker dynamic configuration, this is best-effort. Settings may not be applied in certain
+cases, such as when a Broker has recently started and hasn't received the config yet, or if the
+Broker can't contact the Coordinator. If a query context parameter is critical for all your queries, set it in the runtime properties.
+
+[#19146](https://github.com/apache/druid/pull/19146)
+
+### `sys.queries` table (experimental)
+
+The new system queries table provides information about currently running and recently completed queries that use the Dart engine. This table is off by default. To enable the table, set the following config:
+
+```
+druid.sql.planner.enableSysQueriesTable = true
+```
+
+As part of this change, the `/druid/v2/sql/queries` API now supports an `includeComplete` parameter that shows recently completed queries.
+
+[#18923](https://github.com/apache/druid/pull/18923)
+
+### Auto-compaction with compaction supervisors
+
+Auto-compaction using compaction supervisors has been improved and is now generally available.
+
+As part of the improvement, instead of individual compacted segments having to store their full compaction state in the metadata store, states are now stored in a central location, a new `indexingStates` table, in the metadata store. Individual segments only need to store a unique reference (`indexing_state_fingerprint`) to their full compaction state.
+
+Since many segments in a single datasource share the same underlying compaction state, this greatly reduces metadata storage requirements for automatic compaction.
+
+For backwards compatibility, detailed compaction state will continue to be persisted in each segment for now. This will be removed in a future release.
+
+You can stop storing detailed compaction state by setting `storeCompactionStatePerSegment` to `false` in the cluster compaction config. If you turn it off and need to downgrade, Druid needs to re-compact any segments that have been compacted since you changed the config.
+
+This change has upgrade impacts for metadata storage and metadata caching. For more information, see the [Metadata storage for auto-compaction with compaction supervisors](#metadata-storage-for-auto-compaction-with-compaction-supervisors) upgrade note.
+
+[#19113](https://github.com/apache/druid/pull/19113) [#18844](https://github.com/apache/druid/pull/18844)
+
+### Broker tier selection for realtime servers
+
+Added `druid.broker.realtime.select.tier` and `druid.broker.realtime.balancer.type` on the Brokers to optionally override the Broker’s tier selection and balancer strategies for realtime servers. If these properties are not set (the default), realtime servers continue to use the existing `druid.broker.select` and `druid.broker.balancer` configurations that apply to both historical and realtime servers.
+
+[#19062](https://github.com/apache/druid/pull/19062)
+
+### Manual Broker routing in the web console
+
+You can now configure which Broker the Router uses for queries issued from the web console. You may want to do this if there are Brokers that don't have visibility into certain data tiers, and you know you're querying data available only on a certain tier.
+
+To specify a Broker, add the following config to `web-console/console-config.js`:
+
+```js
+consoleBrokerService: 'druid/BROKER_NAME'
+```
+
+[#19069](https://github.com/apache/druid/pull/19069)
+
+### Consul extension
+
+The contributor extension `druid-consul-extensions` lets Druid clusters use Consul for service discovery and
+Coordinator/Overlord leader election instead of ZooKeeper. The extension supports ACLs, TLS/mTLS, and metrics.
+
+Before you switch to Consul, you need to set
+`druid.serverview.type=http` and `druid.indexer.runner.type=httpRemote` cluster wide.
+
+[#18843](https://github.com/apache/druid/pull/18843)
+
## Functional area and related changes
This section contains detailed release notes separated by areas.
### Web console
+#### Changed storage column displays
+
+- Improved the compaction config view to
+- Renamed **Current size** to **Assigned size**.
+- Renamed **Max size** to **Effective size**. It now displays the smaller value between `max_size` and `storage_size`. The max size is still shown as a tooltip.
+- Changed usage calculation to use `effective_size`
+
+[#19007](https://github.com/apache/druid/pull/19007)
+
#### Other web console improvements
+- Added `workerDesc` to `WorkerStats`, which makes it easier to identify where a worker is running [#19171](https://github.com/apache/druid/pull/19171)
+- Added the Dart unique execution ID (`dartQueryId`) and the `sqlQueryId` to the **Details** pane in the web console [#19185](https://github.com/apache/druid/pull/19185)
+- Added support for showing completed Dart queries in the web console [#18940](https://github.com/apache/druid/pull/18940)
+- Added a detail dialog to the **Services** page [#18960](https://github.com/apache/druid/pull/18960)
+- Added icons to indicate when data is loaded into virtual storage, including a tooltip that shows all the counters for the data [#19010](https://github.com/apache/druid/pull/19010)
+- Added support for Dart reports [#18897](https://github.com/apache/druid/pull/18897)
+- Changed what's considered an active worker: any nonzero rows, files, bytes, frames, or wall time is enough to consider a worker active [#19183](https://github.com/apache/druid/pull/19183)
Review Comment:
```suggestion
- Changed the criteria for active workers: any nonzero rows, files, bytes, frames, or wall time is enough to consider a worker active [#19183](https://github.com/apache/druid/pull/19183)
```
##########
docs/release-info/release-notes.md:
##########
@@ -57,50 +57,551 @@ For tips about how to write a good release note, see [Release notes](https://git
This section contains important information about new and existing features.
+### Hadoop-based ingestion
+
+Support for Hadoop-based ingestion has been removed. The feature was deprecated in Druid 34.
+
+Use one of Druid's other supported ingestion methods, such as SQL-based ingestion or MiddleManager-less ingestion using Kubernetes.
+
+[#19109](https://github.com/apache/druid/pull/19109)
+
+### Query blocklist
+
+You can now create a query blocklist for dynamically blocking queries without restarting your deployment. You can block queries by datasource, query type, or query context using the new `/druid/coordinator/v1/config/broker` API. Block rules use `AND` logic, which means all criteria must match. Druid stores the rules in the metadata database.
+
+The following example blocks all groupBy queries on the `wikipedia` datasource with a query context parameter of `priority` equal to `0`:
+
+```
+POST /druid/coordinator/v1/config/broker
+ {
+ "queryBlocklist": [
+ {
+ "ruleName": "block-wikipedia-groupbys",
+ "dataSources": ["wikipedia"],
+ "queryTypes": ["groupBy"],
+ "contextMatches": {"priority": "0"}
+ }
+ ]
+ }
+```
+
+[#19011](https://github.com/apache/druid/pull/19011)
+
+### Minor compaction for Overlord-based compaction (experimental)
+
+You can now configure minor compaction. With minor compaction, Druid only compacts newly ingested segments while upgrading existing compacted segments. When Druid upgrades segments, it updates the metadata instead of using resources to compact it again. You can use the native compaction engine or the MSQ task engine.
+
+Use the `mostFragmentedFirst` compaction policy and set either a percentage of rows-based or byte-based threshold for minor compaction.
+
+[#19059](https://github.com/apache/druid/pull/19059) [#19205](https://github.com/apache/druid/pull/19205) [#19016](https://github.com/apache/druid/pull/19016)
+
+### Thrift input format
+
+As part of the Thrift contributor extension, Druid now supports Thrift-encoded data for Kafka and Kinesis streaming ingestion.
+
+[#19111](https://github.com/apache/druid/pull/19111)
+
+To use this feature, you must add `druid-thrift-extensions` to your extension load list.
+
+### Kubernetes-based task management
+
+This extension is now generally available.
+
+[#19128](https://github.com/apache/druid/pull/19128)
+
+### Tombstones
+
+Tombstones for JSON-based native batch ingestion (the `dropExisting` flag for `ioConfig`) are now generally available.
+
+[#19128](https://github.com/apache/druid/pull/19128)
+
+### Dynamic default query context
+
+You can now add default query context parameters as a dynamic configuration to the Broker. This allows you to override static defaults set in your runtime properties without restarting your deployment or having to update multiple queries individually. Druid resolves which query context parameters to use based on the following priority:
+
+1. The query context included with the query
+1. The query context set as a dynamic configuration on the Broker
+1. The query context parameters set in the runtime properties
+1. The defaults that ship with Druid
+
+Note that like other Broker dynamic configuration, this is best-effort. Settings may not be applied in certain
+cases, such as when a Broker has recently started and hasn't received the config yet, or if the
+Broker can't contact the Coordinator. If a query context parameter is critical for all your queries, set it in the runtime properties.
+
+[#19146](https://github.com/apache/druid/pull/19146)
+
+### `sys.queries` table (experimental)
+
+The new system queries table provides information about currently running and recently completed queries that use the Dart engine. This table is off by default. To enable the table, set the following config:
+
+```
+druid.sql.planner.enableSysQueriesTable = true
+```
+
+As part of this change, the `/druid/v2/sql/queries` API now supports an `includeComplete` parameter that shows recently completed queries.
+
+[#18923](https://github.com/apache/druid/pull/18923)
+
+### Auto-compaction with compaction supervisors
+
+Auto-compaction using compaction supervisors has been improved and is now generally available.
+
+As part of the improvement, instead of individual compacted segments having to store their full compaction state in the metadata store, states are now stored in a central location, a new `indexingStates` table, in the metadata store. Individual segments only need to store a unique reference (`indexing_state_fingerprint`) to their full compaction state.
+
+Since many segments in a single datasource share the same underlying compaction state, this greatly reduces metadata storage requirements for automatic compaction.
+
+For backwards compatibility, detailed compaction state will continue to be persisted in each segment for now. This will be removed in a future release.
+
+You can stop storing detailed compaction state by setting `storeCompactionStatePerSegment` to `false` in the cluster compaction config. If you turn it off and need to downgrade, Druid needs to re-compact any segments that have been compacted since you changed the config.
+
+This change has upgrade impacts for metadata storage and metadata caching. For more information, see the [Metadata storage for auto-compaction with compaction supervisors](#metadata-storage-for-auto-compaction-with-compaction-supervisors) upgrade note.
+
+[#19113](https://github.com/apache/druid/pull/19113) [#18844](https://github.com/apache/druid/pull/18844)
+
+### Broker tier selection for realtime servers
+
+Added `druid.broker.realtime.select.tier` and `druid.broker.realtime.balancer.type` on the Brokers to optionally override the Broker’s tier selection and balancer strategies for realtime servers. If these properties are not set (the default), realtime servers continue to use the existing `druid.broker.select` and `druid.broker.balancer` configurations that apply to both historical and realtime servers.
+
+[#19062](https://github.com/apache/druid/pull/19062)
+
+### Manual Broker routing in the web console
+
+You can now configure which Broker the Router uses for queries issued from the web console. You may want to do this if there are Brokers that don't have visibility into certain data tiers, and you know you're querying data available only on a certain tier.
+
+To specify a Broker, add the following config to `web-console/console-config.js`:
+
+```js
+consoleBrokerService: 'druid/BROKER_NAME'
+```
+
+[#19069](https://github.com/apache/druid/pull/19069)
+
+### Consul extension
+
+The contributor extension `druid-consul-extensions` lets Druid clusters use Consul for service discovery and
+Coordinator/Overlord leader election instead of ZooKeeper. The extension supports ACLs, TLS/mTLS, and metrics.
+
+Before you switch to Consul, you need to set
+`druid.serverview.type=http` and `druid.indexer.runner.type=httpRemote` cluster wide.
+
+[#18843](https://github.com/apache/druid/pull/18843)
+
## Functional area and related changes
This section contains detailed release notes separated by areas.
### Web console
+#### Changed storage column displays
+
+- Improved the compaction config view to
+- Renamed **Current size** to **Assigned size**.
+- Renamed **Max size** to **Effective size**. It now displays the smaller value between `max_size` and `storage_size`. The max size is still shown as a tooltip.
+- Changed usage calculation to use `effective_size`
+
+[#19007](https://github.com/apache/druid/pull/19007)
+
#### Other web console improvements
+- Added `workerDesc` to `WorkerStats`, which makes it easier to identify where a worker is running [#19171](https://github.com/apache/druid/pull/19171)
+- Added the Dart unique execution ID (`dartQueryId`) and the `sqlQueryId` to the **Details** pane in the web console [#19185](https://github.com/apache/druid/pull/19185)
+- Added support for showing completed Dart queries in the web console [#18940](https://github.com/apache/druid/pull/18940)
+- Added a detail dialog to the **Services** page [#18960](https://github.com/apache/druid/pull/18960)
+- Added icons to indicate when data is loaded into virtual storage, including a tooltip that shows all the counters for the data [#19010](https://github.com/apache/druid/pull/19010)
+- Added support for Dart reports [#18897](https://github.com/apache/druid/pull/18897)
+- Changed what's considered an active worker: any nonzero rows, files, bytes, frames, or wall time is enough to consider a worker active [#19183](https://github.com/apache/druid/pull/19183)
+- Changed the **Cancel query** option to show only if a query is in an accepted or running state [#19182](https://github.com/apache/druid/pull/19182)
+- Changed the ordering of the current Dart queries panel to show queries in the following order: RUNNING, ACCEPTED, and then COMPLETED. RUNNING and ACCEPTED queries are ordered by the most recent first (based on timestamp). COMPLETED queries are sorted by finish time [#19237](https://github.com/apache/druid/pull/19237)
+
### Ingestion
#### SQL-based ingestion
##### Other SQL-based ingestion improvements
+- Added support for virtual storage fabric when performing SQL-based ingestion [#18873](https://github.com/apache/druid/pull/18873)
+- Added support for `StorageMonitor` so that MSQ task engine tasks always emit `taskId` and `groupId` [#19048](https://github.com/apache/druid/pull/19048)
+- Improved worker cancellation [#18931](https://github.com/apache/druid/pull/18931)
+- Improved exception handling [#19234](https://github.com/apache/druid/pull/19234)
+
#### Streaming ingestion
+##### Changed how tasks get launched for autoscaling
+
+The behavior of `taskCountMin` and `taskCountStart` has been changed for autoscaling. Druid now computes the initial number of tasks to launch by checking the configs in the following order: `taskCountStart` (optional), then `taskCount` (in `ioConfig`), then `taskCountMin`.
+
+[#19091](https://github.com/apache/druid/pull/19091)
+
+##### Improved query isolation
+
+Added `serverPriorityToReplicas` parameter to the streaming supervisor specs (Kafka, Kinesis, and Rabbit). This allows operators to distribute task replicas across different server priorities for realtime indexing tasks. Similar to Historical tiering, this enables query isolation for mixed workload scenarios on the Peons, allowing some task replicas to handle queries of specific priorities.
+
+[#19040](https://github.com/apache/druid/pull/19040)
+
##### Other streaming ingestion improvements
+- Improved cost-based autoscaler performance in high lag scenarios [#19045](https://github.com/apache/druid/pull/19045)
+- Improved the performance of realtime task scheduling by ordering schedule requests by priority on the `TaskQueue` [#19203](https://github.com/apache/druid/pull/19203)
+
+
+#### Ingestion improvements
+
+- Sped up task scheduling on the Overlord [#19199](https://github.com/apache/druid/pull/19199)
+
### Querying
+#### groupBy query configuration
+
+Added a new groupBy query configuration property `druid.query.groupBy.maxSpillFileCount` to limit the maximum number of spill files created per query. When the limit is exceeded, the query fails with a clear error message instead of causing Historical nodes to run out of memory during spill file merging. The limit can also be overridden per query using the query context parameter `maxSpillFileCount`.
+
+[#19141](https://github.com/apache/druid/pull/19141)
+
+#### Improved handling of nested aggregates
+
+Druid can now merge two Aggregates with a Projection between them. For example, the following query:
+
+```sql
+SELECT
+ hr,
+ UPPER(t1.x) x,
+ SUM(t1.cnt) cnt,
+ MIN(t1.mn) mn,
+ MAX(t1.mx) mx
+FROM (
+ SELECT
+ floor(__time to hour) hr,
+ dim2 x,
+ COUNT(*) cnt,
+ MIN(m1 * 5) mn,
+ MAX(m1 + m2) mx
+ FROM druid.foo
+ WHERE dim2 IN ('abc', 'def', 'a', 'b', '')
+ GROUP BY 1, 2
+) t1
+WHERE t1.x IN ('abc', 'foo', 'bar', 'a', '')
+GROUP BY 1, 2
+```
+
+can be simplified to the following:
+
+```sql
+SELECT
+ FLOOR(__time TO hour) hr,
+ UPPER(dim2) x,
+ COUNT(*) cnt,
+ MIN(m1 * 5) mn,
+ MAX(m1 + m2) mx
+FROM druid.foo
+WHERE dim2 IN ('abc', 'a', '')
+GROUP BY 1, 2
+```
+
+[#18498](https://github.com/apache/druid/pull/18498)
+
#### Other querying improvements
+- Added `durationMs` to Dart query reports [#19169](https://github.com/apache/druid/pull/19169)
+- Improved error handling so that row signature column order is preserved when column analysis encounters an error [#19162](https://github.com/apache/druid/pull/19162)
+- Improved GROUP BY performance [#18952](https://github.com/apache/druid/pull/18952)
+- Improved expression filters to take advantage of specialized virtual columns when possible, resulting in better performance for the query [#18965](https://github.com/apache/druid/pull/18965)
+
### Cluster management
+#### New Broker tier selection strategies
+
+Operators can now configure two new Broker `TierSelectorStrategy` implementations:
+
+- `strict` - Only selects servers whose priorities match the configured list. Example configuration: `druid.broker.select.tier=strict` and `druid.broker.select.tier.strict.priorities=[1]`.
+- `pooled` - Pools servers across the configured priorities and selects among them, allowing queries to utilize multiple priority tiers for improved availability. Example configuration: `druid.broker.select.tier=pooled` and `druid.broker.select.tier.pooled.priorities=[2,1]`.
+
+These strategies can also be configured for realtime servers using `druid.broker.realtime.select.tier`.
+
+[#19094](https://github.com/apache/druid/pull/19094)
+
+#### Druid operator
+
+The Druid operator now resides in its own repository: [`apache/druid-operator`](https://github.com/apache/druid-operator).
+
+[#19156](https://github.com/apache/druid/pull/19156)
+
+#### Cost-based autoscaler algorithm
+
+The algorithm for cost-based autoscaling has been changed:
+
+- Scale up more aggressively when per-partition lag is meaningful
+- Relax the partitions-per-task increase limit based on lag severity and headroom
+- Keep behavior conservative near `taskCountMax` and avoid negative headroom effects
+
+[#18936](https://github.com/apache/druid/pull/18936)
+
#### Other cluster management improvements
+- Added a `ReadOnly` authorizer that allows all READ operations but denies any other operation, such as WRITE [#19243](https://github.com/apache/druid/pull/19243)
+- Added `/status/ready` endpoint for service health so that external load balancers can handle a graceful shutdown better [#19148](https://github.com/apache/druid/pull/19148)
+- Added a configurable option to scale-down during task run time for cost-based autoscaler [#18958](https://github.com/apache/druid/pull/18958)
+- Added `storage_size` to `sys.servers` to facilitate retrieving disk cache size for Historicals when using the virtual storage fabric [#18979](https://github.com/apache/druid/pull/18979)
+- Added a log for new task count computation for the cost-based auto scaler [#18929](https://github.com/apache/druid/pull/18929)
+- Changed how the scaling is calculated from a square root-based scaling formula to a logarithmic formula that provides better emergency recovery at low task counts and millions of lag [#18976](https://github.com/apache/druid/pull/18976)
+- Improved the load speed of cached segments during Historical startup [#18489](https://github.com/apache/druid/pull/18489)
+- Improved Broker startup time by parallelizing buffer initialization [#19025](https://github.com/apache/druid/pull/19025)
+- Improved the stack trace for MSQ task engine worker failures so that they're preserved [#19049](https://github.com/apache/druid/pull/19049)
+- Improved the performance of the cost-based autoscaler during loaded lag conditions [#18991](https://github.com/apache/druid/pull/18991)
+
### Data management
+#### Durable storage cleaner
+
+The durable storage cleaner now supports configurable time-based retention for MSQ query results. Previously, query results were retained for all known tasks, which was unreliable for completed tasks. With this change, query results are retained for a configurable time period based on the task creation time.
+
+The new configuration property `druid.msq.intermediate.storage.cleaner.durationToRetain` controls the retention period for query results. The default retention period is 6 hours.
+
+[#19074](https://github.com/apache/druid/pull/19074)
+
#### Other data management improvements
+- Added the `druid.storage.transfer.asyncHttpClientType` config that specifies which async HTTP client to use for S3 transfers: `crt` for Amazon CRT or `netty` for Netty NIO [#19249](https://github.com/apache/druid/pull/19249)
+- Added a mechanism to automatically clean up intermediary files on HDFS storage [#19187](https://github.com/apache/druid/pull/19187)
+- Changed the default value for `druid.indexing.formats.maxStringLength` to null from 0 [#19198](https://github.com/apache/druid/pull/19198)
+
### Metrics and monitoring
+#### Version field
+
+All Druid metrics now include `version` and `buildRevision` fields to help you identify which version of Druid is emitting a metric. You can use this information to verify that all nodes in a cluster are running the intended revision during rolling deployments.
+
+[#19123](https://github.com/apache/druid/pull/19123)
+
+#### Monitoring supervisor state
+
+Added a new `supervisor/count` metric when `SupervisorStatsMonitor` is enabled in `druid.monitoring.monitors`. The metric reports each supervisor’s state, such as `RUNNING` or `SUSPENDED`, for Prometheus, StatsD, and other metric systems.
+
+[#19114](https://github.com/apache/druid/pull/19114)
+
+#### Improved `groupBy` metrics
+
+`GroupByStatsMonitor` now provides the following metrics:
+
+- `mergeBuffer/bytesUsed`
+- `mergeBuffer/maxAcquisitionTimeNs`
+- `groupBy/maxSpilledBytes`
+- `groupBy/maxMergeDictionarySize`
+
+[#18731](https://github.com/apache/druid/pull/18731) [#18970](https://github.com/apache/druid/pull/18970) [#18934](https://github.com/apache/druid/pull/18934)
+
+#### Filtering metrics
+
+Operators can limit which metrics the logging emitter writes by setting `druid.emitter.logging.shouldFilterMetrics=true` and, if desired, `druid.emitter.logging.allowedMetricsPath` to a JSON object file where the keys are metric names. A missing custom file results in a warning and use of the bundled `defaultMetrics.json`. Alerts and other non-metric events are always logged.
Review Comment:
```suggestion
Operators can set `druid.emitter.logging.shouldFilterMetrics=true` to limit which metrics the logging emitter writes. Optionally, they can set `druid.emitter.logging.allowedMetricsPath` to a JSON object file where the keys are metric names. A missing custom file results in a warning and use of the bundled `defaultMetrics.json`. Alerts and other non-metric events are always logged.
```
##########
docs/release-info/release-notes.md:
##########
@@ -57,50 +57,551 @@ For tips about how to write a good release note, see [Release notes](https://git
This section contains important information about new and existing features.
+### Hadoop-based ingestion
+
+Support for Hadoop-based ingestion has been removed. The feature was deprecated in Druid 34.
+
+Use one of Druid's other supported ingestion methods, such as SQL-based ingestion or MiddleManager-less ingestion using Kubernetes.
+
+[#19109](https://github.com/apache/druid/pull/19109)
+
+### Query blocklist
+
+You can now create a query blocklist for dynamically blocking queries without restarting your deployment. You can block queries by datasource, query type, or query context using the new `/druid/coordinator/v1/config/broker` API. Block rules use `AND` logic, which means all criteria must match. Druid stores the rules in the metadata database.
+
+The following example blocks all groupBy queries on the `wikipedia` datasource with a query context parameter of `priority` equal to `0`:
+
+```
+POST /druid/coordinator/v1/config/broker
+ {
+ "queryBlocklist": [
+ {
+ "ruleName": "block-wikipedia-groupbys",
+ "dataSources": ["wikipedia"],
+ "queryTypes": ["groupBy"],
+ "contextMatches": {"priority": "0"}
+ }
+ ]
+ }
+```
+
+[#19011](https://github.com/apache/druid/pull/19011)
+
+### Minor compaction for Overlord-based compaction (experimental)
+
+You can now configure minor compaction. With minor compaction, Druid only compacts newly ingested segments while upgrading existing compacted segments. When Druid upgrades segments, it updates the metadata instead of using resources to compact it again. You can use the native compaction engine or the MSQ task engine.
+
+Use the `mostFragmentedFirst` compaction policy and set either a percentage of rows-based or byte-based threshold for minor compaction.
+
+[#19059](https://github.com/apache/druid/pull/19059) [#19205](https://github.com/apache/druid/pull/19205) [#19016](https://github.com/apache/druid/pull/19016)
+
+### Thrift input format
+
+As part of the Thrift contributor extension, Druid now supports Thrift-encoded data for Kafka and Kinesis streaming ingestion.
+
+[#19111](https://github.com/apache/druid/pull/19111)
+
+To use this feature, you must add `druid-thrift-extensions` to your extension load list.
+
+### Kubernetes-based task management
+
+This extension is now generally available.
+
+[#19128](https://github.com/apache/druid/pull/19128)
+
+### Tombstones
+
+Tombstones for JSON-based native batch ingestion (the `dropExisting` flag for `ioConfig`) are now generally available.
+
+[#19128](https://github.com/apache/druid/pull/19128)
+
+### Dynamic default query context
+
+You can now add default query context parameters as a dynamic configuration to the Broker. This allows you to override static defaults set in your runtime properties without restarting your deployment or having to update multiple queries individually. Druid resolves which query context parameters to use based on the following priority:
+
+1. The query context included with the query
+1. The query context set as a dynamic configuration on the Broker
+1. The query context parameters set in the runtime properties
+1. The defaults that ship with Druid
+
+Note that like other Broker dynamic configuration, this is best-effort. Settings may not be applied in certain
+cases, such as when a Broker has recently started and hasn't received the config yet, or if the
+Broker can't contact the Coordinator. If a query context parameter is critical for all your queries, set it in the runtime properties.
+
+[#19146](https://github.com/apache/druid/pull/19146)
+
+### `sys.queries` table (experimental)
+
+The new system queries table provides information about currently running and recently completed queries that use the Dart engine. This table is off by default. To enable the table, set the following config:
+
+```
+druid.sql.planner.enableSysQueriesTable = true
+```
+
+As part of this change, the `/druid/v2/sql/queries` API now supports an `includeComplete` parameter that shows recently completed queries.
+
+[#18923](https://github.com/apache/druid/pull/18923)
+
+### Auto-compaction with compaction supervisors
+
+Auto-compaction using compaction supervisors has been improved and is now generally available.
+
+As part of the improvement, instead of individual compacted segments having to store their full compaction state in the metadata store, states are now stored in a central location, a new `indexingStates` table, in the metadata store. Individual segments only need to store a unique reference (`indexing_state_fingerprint`) to their full compaction state.
+
+Since many segments in a single datasource share the same underlying compaction state, this greatly reduces metadata storage requirements for automatic compaction.
+
+For backwards compatibility, detailed compaction state will continue to be persisted in each segment for now. This will be removed in a future release.
+
+You can stop storing detailed compaction state by setting `storeCompactionStatePerSegment` to `false` in the cluster compaction config. If you turn it off and need to downgrade, Druid needs to re-compact any segments that have been compacted since you changed the config.
+
+This change has upgrade impacts for metadata storage and metadata caching. For more information, see the [Metadata storage for auto-compaction with compaction supervisors](#metadata-storage-for-auto-compaction-with-compaction-supervisors) upgrade note.
+
+[#19113](https://github.com/apache/druid/pull/19113) [#18844](https://github.com/apache/druid/pull/18844)
+
+### Broker tier selection for realtime servers
+
+Added `druid.broker.realtime.select.tier` and `druid.broker.realtime.balancer.type` on the Brokers to optionally override the Broker’s tier selection and balancer strategies for realtime servers. If these properties are not set (the default), realtime servers continue to use the existing `druid.broker.select` and `druid.broker.balancer` configurations that apply to both historical and realtime servers.
+
+[#19062](https://github.com/apache/druid/pull/19062)
+
+### Manual Broker routing in the web console
+
+You can now configure which Broker the Router uses for queries issued from the web console. You may want to do this if there are Brokers that don't have visibility into certain data tiers, and you know you're querying data available only on a certain tier.
+
+To specify a Broker, add the following config to `web-console/console-config.js`:
+
+```js
+consoleBrokerService: 'druid/BROKER_NAME'
+```
+
+[#19069](https://github.com/apache/druid/pull/19069)
+
+### Consul extension
+
+The contributor extension `druid-consul-extensions` lets Druid clusters use Consul for service discovery and
+Coordinator/Overlord leader election instead of ZooKeeper. The extension supports ACLs, TLS/mTLS, and metrics.
+
+Before you switch to Consul, you need to set
+`druid.serverview.type=http` and `druid.indexer.runner.type=httpRemote` cluster wide.
+
+[#18843](https://github.com/apache/druid/pull/18843)
+
## Functional area and related changes
This section contains detailed release notes separated by areas.
### Web console
+#### Changed storage column displays
+
+- Improved the compaction config view to
+- Renamed **Current size** to **Assigned size**.
+- Renamed **Max size** to **Effective size**. It now displays the smaller value between `max_size` and `storage_size`. The max size is still shown as a tooltip.
+- Changed usage calculation to use `effective_size`
+
+[#19007](https://github.com/apache/druid/pull/19007)
+
#### Other web console improvements
+- Added `workerDesc` to `WorkerStats`, which makes it easier to identify where a worker is running [#19171](https://github.com/apache/druid/pull/19171)
+- Added the Dart unique execution ID (`dartQueryId`) and the `sqlQueryId` to the **Details** pane in the web console [#19185](https://github.com/apache/druid/pull/19185)
+- Added support for showing completed Dart queries in the web console [#18940](https://github.com/apache/druid/pull/18940)
+- Added a detail dialog to the **Services** page [#18960](https://github.com/apache/druid/pull/18960)
+- Added icons to indicate when data is loaded into virtual storage, including a tooltip that shows all the counters for the data [#19010](https://github.com/apache/druid/pull/19010)
+- Added support for Dart reports [#18897](https://github.com/apache/druid/pull/18897)
+- Changed what's considered an active worker: any nonzero rows, files, bytes, frames, or wall time is enough to consider a worker active [#19183](https://github.com/apache/druid/pull/19183)
+- Changed the **Cancel query** option to show only if a query is in an accepted or running state [#19182](https://github.com/apache/druid/pull/19182)
+- Changed the ordering of the current Dart queries panel to show queries in the following order: RUNNING, ACCEPTED, and then COMPLETED. RUNNING and ACCEPTED queries are ordered by the most recent first (based on timestamp). COMPLETED queries are sorted by finish time [#19237](https://github.com/apache/druid/pull/19237)
+
### Ingestion
#### SQL-based ingestion
##### Other SQL-based ingestion improvements
+- Added support for virtual storage fabric when performing SQL-based ingestion [#18873](https://github.com/apache/druid/pull/18873)
+- Added support for `StorageMonitor` so that MSQ task engine tasks always emit `taskId` and `groupId` [#19048](https://github.com/apache/druid/pull/19048)
+- Improved worker cancellation [#18931](https://github.com/apache/druid/pull/18931)
+- Improved exception handling [#19234](https://github.com/apache/druid/pull/19234)
+
#### Streaming ingestion
+##### Changed how tasks get launched for autoscaling
+
+The behavior of `taskCountMin` and `taskCountStart` has been changed for autoscaling. Druid now computes the initial number of tasks to launch by checking the configs in the following order: `taskCountStart` (optional), then `taskCount` (in `ioConfig`), then `taskCountMin`.
+
+[#19091](https://github.com/apache/druid/pull/19091)
+
+##### Improved query isolation
+
+Added `serverPriorityToReplicas` parameter to the streaming supervisor specs (Kafka, Kinesis, and Rabbit). This allows operators to distribute task replicas across different server priorities for realtime indexing tasks. Similar to Historical tiering, this enables query isolation for mixed workload scenarios on the Peons, allowing some task replicas to handle queries of specific priorities.
+
+[#19040](https://github.com/apache/druid/pull/19040)
+
##### Other streaming ingestion improvements
+- Improved cost-based autoscaler performance in high lag scenarios [#19045](https://github.com/apache/druid/pull/19045)
+- Improved the performance of realtime task scheduling by ordering schedule requests by priority on the `TaskQueue` [#19203](https://github.com/apache/druid/pull/19203)
+
+
+#### Ingestion improvements
+
+- Sped up task scheduling on the Overlord [#19199](https://github.com/apache/druid/pull/19199)
+
### Querying
+#### groupBy query configuration
+
+Added a new groupBy query configuration property `druid.query.groupBy.maxSpillFileCount` to limit the maximum number of spill files created per query. When the limit is exceeded, the query fails with a clear error message instead of causing Historical nodes to run out of memory during spill file merging. The limit can also be overridden per query using the query context parameter `maxSpillFileCount`.
+
+[#19141](https://github.com/apache/druid/pull/19141)
+
+#### Improved handling of nested aggregates
+
+Druid can now merge two Aggregates with a Projection between them. For example, the following query:
+
+```sql
+SELECT
+ hr,
+ UPPER(t1.x) x,
+ SUM(t1.cnt) cnt,
+ MIN(t1.mn) mn,
+ MAX(t1.mx) mx
+FROM (
+ SELECT
+ floor(__time to hour) hr,
+ dim2 x,
+ COUNT(*) cnt,
+ MIN(m1 * 5) mn,
+ MAX(m1 + m2) mx
+ FROM druid.foo
+ WHERE dim2 IN ('abc', 'def', 'a', 'b', '')
+ GROUP BY 1, 2
+) t1
+WHERE t1.x IN ('abc', 'foo', 'bar', 'a', '')
+GROUP BY 1, 2
+```
+
+can be simplified to the following:
+
+```sql
+SELECT
+ FLOOR(__time TO hour) hr,
+ UPPER(dim2) x,
+ COUNT(*) cnt,
+ MIN(m1 * 5) mn,
+ MAX(m1 + m2) mx
+FROM druid.foo
+WHERE dim2 IN ('abc', 'a', '')
+GROUP BY 1, 2
+```
+
+[#18498](https://github.com/apache/druid/pull/18498)
+
#### Other querying improvements
+- Added `durationMs` to Dart query reports [#19169](https://github.com/apache/druid/pull/19169)
+- Improved error handling so that row signature column order is preserved when column analysis encounters an error [#19162](https://github.com/apache/druid/pull/19162)
+- Improved GROUP BY performance [#18952](https://github.com/apache/druid/pull/18952)
+- Improved expression filters to take advantage of specialized virtual columns when possible, resulting in better performance for the query [#18965](https://github.com/apache/druid/pull/18965)
+
### Cluster management
+#### New Broker tier selection strategies
+
+Operators can now configure two new Broker `TierSelectorStrategy` implementations:
+
+- `strict` - Only selects servers whose priorities match the configured list. Example configuration: `druid.broker.select.tier=strict` and `druid.broker.select.tier.strict.priorities=[1]`.
+- `pooled` - Pools servers across the configured priorities and selects among them, allowing queries to utilize multiple priority tiers for improved availability. Example configuration: `druid.broker.select.tier=pooled` and `druid.broker.select.tier.pooled.priorities=[2,1]`.
+
+These strategies can also be configured for realtime servers using `druid.broker.realtime.select.tier`.
+
+[#19094](https://github.com/apache/druid/pull/19094)
+
+#### Druid operator
+
+The Druid operator now resides in its own repository: [`apache/druid-operator`](https://github.com/apache/druid-operator).
+
+[#19156](https://github.com/apache/druid/pull/19156)
+
+#### Cost-based autoscaler algorithm
+
+The algorithm for cost-based autoscaling has been changed:
+
+- Scale up more aggressively when per-partition lag is meaningful
+- Relax the partitions-per-task increase limit based on lag severity and headroom
+- Keep behavior conservative near `taskCountMax` and avoid negative headroom effects
+
+[#18936](https://github.com/apache/druid/pull/18936)
+
#### Other cluster management improvements
+- Added a `ReadOnly` authorizer that allows all READ operations but denies any other operation, such as WRITE [#19243](https://github.com/apache/druid/pull/19243)
+- Added `/status/ready` endpoint for service health so that external load balancers can handle a graceful shutdown better [#19148](https://github.com/apache/druid/pull/19148)
+- Added a configurable option to scale-down during task run time for cost-based autoscaler [#18958](https://github.com/apache/druid/pull/18958)
+- Added `storage_size` to `sys.servers` to facilitate retrieving disk cache size for Historicals when using the virtual storage fabric [#18979](https://github.com/apache/druid/pull/18979)
+- Added a log for new task count computation for the cost-based auto scaler [#18929](https://github.com/apache/druid/pull/18929)
+- Changed how the scaling is calculated from a square root-based scaling formula to a logarithmic formula that provides better emergency recovery at low task counts and millions of lag [#18976](https://github.com/apache/druid/pull/18976)
+- Improved the load speed of cached segments during Historical startup [#18489](https://github.com/apache/druid/pull/18489)
+- Improved Broker startup time by parallelizing buffer initialization [#19025](https://github.com/apache/druid/pull/19025)
+- Improved the stack trace for MSQ task engine worker failures so that they're preserved [#19049](https://github.com/apache/druid/pull/19049)
+- Improved the performance of the cost-based autoscaler during loaded lag conditions [#18991](https://github.com/apache/druid/pull/18991)
+
### Data management
+#### Durable storage cleaner
+
+The durable storage cleaner now supports configurable time-based retention for MSQ query results. Previously, query results were retained for all known tasks, which was unreliable for completed tasks. With this change, query results are retained for a configurable time period based on the task creation time.
+
+The new configuration property `druid.msq.intermediate.storage.cleaner.durationToRetain` controls the retention period for query results. The default retention period is 6 hours.
+
+[#19074](https://github.com/apache/druid/pull/19074)
+
#### Other data management improvements
+- Added the `druid.storage.transfer.asyncHttpClientType` config that specifies which async HTTP client to use for S3 transfers: `crt` for Amazon CRT or `netty` for Netty NIO [#19249](https://github.com/apache/druid/pull/19249)
+- Added a mechanism to automatically clean up intermediary files on HDFS storage [#19187](https://github.com/apache/druid/pull/19187)
+- Changed the default value for `druid.indexing.formats.maxStringLength` to null from 0 [#19198](https://github.com/apache/druid/pull/19198)
+
### Metrics and monitoring
+#### Version field
+
+All Druid metrics now include `version` and `buildRevision` fields to help you identify which version of Druid is emitting a metric. You can use this information to verify that all nodes in a cluster are running the intended revision during rolling deployments.
+
+[#19123](https://github.com/apache/druid/pull/19123)
+
+#### Monitoring supervisor state
+
+Added a new `supervisor/count` metric when `SupervisorStatsMonitor` is enabled in `druid.monitoring.monitors`. The metric reports each supervisor’s state, such as `RUNNING` or `SUSPENDED`, for Prometheus, StatsD, and other metric systems.
+
+[#19114](https://github.com/apache/druid/pull/19114)
+
+#### Improved `groupBy` metrics
+
+`GroupByStatsMonitor` now provides the following metrics:
+
+- `mergeBuffer/bytesUsed`
+- `mergeBuffer/maxAcquisitionTimeNs`
+- `groupBy/maxSpilledBytes`
+- `groupBy/maxMergeDictionarySize`
+
+[#18731](https://github.com/apache/druid/pull/18731) [#18970](https://github.com/apache/druid/pull/18970) [#18934](https://github.com/apache/druid/pull/18934)
+
+#### Filtering metrics
+
+Operators can limit which metrics the logging emitter writes by setting `druid.emitter.logging.shouldFilterMetrics=true` and, if desired, `druid.emitter.logging.allowedMetricsPath` to a JSON object file where the keys are metric names. A missing custom file results in a warning and use of the bundled `defaultMetrics.json`. Alerts and other non-metric events are always logged.
+
+[#19030](https://github.com/apache/druid/pull/19030)
+
+#### New Broker metrics
+
+Added `segment/schemaCache/rowSignature/changed` and `segment/schemaCache/rowSignature/column/count` metrics to get visibility into when the Broker's segment metadata cache's row signature for each datasource is initialized and updated.
+
+[#18966](https://github.com/apache/druid/pull/18966)
+
+#### Other metrics and monitoring improvements
+
+- Added the following metrics to the default for Prometheus: `mergeBuffer/bytesUsed` and `mergeBuffer/maxBytesUsed` [#19110](https://github.com/apache/druid/pull/19110)
+- Added compaction mode to what gets emitted for the `compact/task/count` metric [#19151](https://github.com/apache/druid/pull/19151)
+- Added support for logging and emitting SQL dynamic parameter values [#19067](https://github.com/apache/druid/pull/19067)
+- Added `ingest/rows/published`, which gets emitted from all task types to denote the total row count of successfully published segments [#19177](https://github.com/apache/druid/pull/19177)
+- Added `queries` and `totalQueries` counters, which reflect queries made to realtime servers to get realtime data
+- Added `tier/storage/capacity` metric for the Coordinator to emit that is guaranteed to reflect the total `StorageLocation` size configured across all Historicals in a tier [#18962](https://github.com/apache/druid/pull/18962)
+- Added new metrics for virtual storage fabric to the MSQ task engine `ChannelCounters`: `loadBytes`, `loadTime`, `loadWait`, and `loadFiles` [#18971](https://github.com/apache/druid/pull/18971)
+- Added `storage/virtual/hit/bytes`, `storage/virtual/hold/count` and `storage/virtual/hold/bytes` metric to `StorageMonitor`
+[#18895](https://github.com/apache/druid/pull/18895) [#19217](https://github.com/apache/druid/pull/19217)
+- Added `supervisorId` dimension for streaming tasks to `TaskCountStatsMonitor` [#18920](https://github.com/apache/druid/pull/18920)
+- Changed `StorageMonitor` to always be on [#19048](https://github.com/apache/druid/pull/19048)
+- Improved the metrics for autoscalers, so that they all emit the same metrics: `supervisorId`, `dataSource`, and `stream` [#19097](https://github.com/apache/druid/pull/19097)
+
### Extensions
-### Documentation improvements
+#### gRPC
+
+The gRPC query extension now cancels in-flight queries when clients cancel or disconnect.
+
+[#19005](https://github.com/apache/druid/pull/19005)
+
+#### Iceberg
+
+##### GCS warehouse
+
+The Iceberg input source now supports GCS warehouses. To use this feature, you need to load the `druid-google-extensions` extension in addition to the Iceberg extension.
+[#19137](https://github.com/apache/druid/pull/19137)
+
+##### Filters
+
+You can now configure residual filters for non-partition columns when using the Iceberg input source. Set `residualFilterMode` in the Iceberg input source to one of the following:
+
+- `ignore`: (default) Ingest the residual rows with a warning log unless filtered by `transformSpec`.
+- `fail`: Fail the ingestion job when residual filters are detected. Use this to ensure that filters only target partition columns.
+
+[#18953](https://github.com/apache/druid/pull/18953)
+
+#### HDFS storage
+
+Added support for `lz4` compression. As part of this change, the following metrics are now available:
+
+- `hdfs/pull/size`
+- `hdfs/pull/duration`
+- `hdfs/push/size`
+- `hdfs/push/duration`
+
+[#18982](https://github.com/apache/druid/pull/18982)
## Upgrade notes and incompatible changes
### Upgrade notes
+#### Hadoop-based ingestion
+
+Support for Hadoop-based ingestion has been removed. The feature was deprecated in Druid 34.
+
+Use one of Druid's other supported ingestion methods, such as SQL-based ingestion or MiddleManager-less ingestion using Kubernetes.
+
+[#19109](https://github.com/apache/druid/pull/19109)
+
+#### Segment metadata cache on by default
+
+Starting in Druid 37, the segment metadata cache is on by default. This feature allows the Broker to cache segment metadata polled from the Coordinator, rather than having to fetch metadata for every query against the `sys.segments` table. This improves performance but increases memory usage on Brokers.
+
+The `druid.sql.planner.metadataSegmentCacheEnable` config controls this feature.
+
+[#19075](https://github.com/apache/druid/pull/19075)
+
+#### Streaming ingestion `parser`
+
+Support for the deprecated `parser` has been removed for streaming ingest tasks such as Kafka and Kinesis. Operators must now specify `inputSource`/`inputFormat` on the `ioConfig` of the supervisor spec, and the `dataSchema` must not specify a parser. Do this before upgrading to Druid 37 or newer.
+
+[#19173](https://github.com/apache/druid/pull/19173) [#19166](https://github.com/apache/druid/pull/19166)
+
+#### Rolling upgrades from Druid versions prior to version 0.23
+
+You can't perform a rolling upgrade from versions earlier than Druid 0.23.
+
+[#18961](https://github.com/apache/druid/pull/18961)
+
+#### Metadata storage for auto-compaction with compaction supervisors
+
+Automatic compaction using compaction supervisors now requires incremental segment metadata caching to be enabled on the Overlord and Coordinator via runtime properties. Specifically, the `druid.manager.segments.useIncrementalCache` config must be set to `always` or `ifSynced`. For more information about the config, see [Segment metadata cache](https://druid.apache.org/docs/latest/configuration/#segment-metadata-cache-experimental).
Review Comment:
same for upgrade notes page
##########
docs/release-info/release-notes.md:
##########
@@ -57,50 +57,551 @@ For tips about how to write a good release note, see [Release notes](https://git
This section contains important information about new and existing features.
+### Hadoop-based ingestion
+
+Support for Hadoop-based ingestion has been removed. The feature was deprecated in Druid 34.
+
+Use one of Druid's other supported ingestion methods, such as SQL-based ingestion or MiddleManager-less ingestion using Kubernetes.
+
+[#19109](https://github.com/apache/druid/pull/19109)
+
+### Query blocklist
+
+You can now create a query blocklist for dynamically blocking queries without restarting your deployment. You can block queries by datasource, query type, or query context using the new `/druid/coordinator/v1/config/broker` API. Block rules use `AND` logic, which means all criteria must match. Druid stores the rules in the metadata database.
+
+The following example blocks all groupBy queries on the `wikipedia` datasource with a query context parameter of `priority` equal to `0`:
+
+```
+POST /druid/coordinator/v1/config/broker
+ {
+ "queryBlocklist": [
+ {
+ "ruleName": "block-wikipedia-groupbys",
+ "dataSources": ["wikipedia"],
+ "queryTypes": ["groupBy"],
+ "contextMatches": {"priority": "0"}
+ }
+ ]
+ }
+```
+
+[#19011](https://github.com/apache/druid/pull/19011)
+
+### Minor compaction for Overlord-based compaction (experimental)
+
+You can now configure minor compaction. With minor compaction, Druid only compacts newly ingested segments while upgrading existing compacted segments. When Druid upgrades segments, it updates the metadata instead of using resources to compact it again. You can use the native compaction engine or the MSQ task engine.
+
+Use the `mostFragmentedFirst` compaction policy and set either a percentage of rows-based or byte-based threshold for minor compaction.
+
+[#19059](https://github.com/apache/druid/pull/19059) [#19205](https://github.com/apache/druid/pull/19205) [#19016](https://github.com/apache/druid/pull/19016)
+
+### Thrift input format
+
+As part of the Thrift contributor extension, Druid now supports Thrift-encoded data for Kafka and Kinesis streaming ingestion.
+
+[#19111](https://github.com/apache/druid/pull/19111)
+
+To use this feature, you must add `druid-thrift-extensions` to your extension load list.
+
+### Kubernetes-based task management
+
+This extension is now generally available.
+
+[#19128](https://github.com/apache/druid/pull/19128)
+
+### Tombstones
+
+Tombstones for JSON-based native batch ingestion (the `dropExisting` flag for `ioConfig`) are now generally available.
+
+[#19128](https://github.com/apache/druid/pull/19128)
+
+### Dynamic default query context
+
+You can now add default query context parameters as a dynamic configuration to the Broker. This allows you to override static defaults set in your runtime properties without restarting your deployment or having to update multiple queries individually. Druid resolves which query context parameters to use based on the following priority:
+
+1. The query context included with the query
+1. The query context set as a dynamic configuration on the Broker
+1. The query context parameters set in the runtime properties
+1. The defaults that ship with Druid
+
+Note that like other Broker dynamic configuration, this is best-effort. Settings may not be applied in certain
+cases, such as when a Broker has recently started and hasn't received the config yet, or if the
+Broker can't contact the Coordinator. If a query context parameter is critical for all your queries, set it in the runtime properties.
+
+[#19146](https://github.com/apache/druid/pull/19146)
+
+### `sys.queries` table (experimental)
+
+The new system queries table provides information about currently running and recently completed queries that use the Dart engine. This table is off by default. To enable the table, set the following config:
+
+```
+druid.sql.planner.enableSysQueriesTable = true
+```
+
+As part of this change, the `/druid/v2/sql/queries` API now supports an `includeComplete` parameter that shows recently completed queries.
+
+[#18923](https://github.com/apache/druid/pull/18923)
+
+### Auto-compaction with compaction supervisors
+
+Auto-compaction using compaction supervisors has been improved and is now generally available.
+
+As part of the improvement, instead of individual compacted segments having to store their full compaction state in the metadata store, states are now stored in a central location, a new `indexingStates` table, in the metadata store. Individual segments only need to store a unique reference (`indexing_state_fingerprint`) to their full compaction state.
+
+Since many segments in a single datasource share the same underlying compaction state, this greatly reduces metadata storage requirements for automatic compaction.
+
+For backwards compatibility, detailed compaction state will continue to be persisted in each segment for now. This will be removed in a future release.
+
+You can stop storing detailed compaction state by setting `storeCompactionStatePerSegment` to `false` in the cluster compaction config. If you turn it off and need to downgrade, Druid needs to re-compact any segments that have been compacted since you changed the config.
+
+This change has upgrade impacts for metadata storage and metadata caching. For more information, see the [Metadata storage for auto-compaction with compaction supervisors](#metadata-storage-for-auto-compaction-with-compaction-supervisors) upgrade note.
+
+[#19113](https://github.com/apache/druid/pull/19113) [#18844](https://github.com/apache/druid/pull/18844)
+
+### Broker tier selection for realtime servers
+
+Added `druid.broker.realtime.select.tier` and `druid.broker.realtime.balancer.type` on the Brokers to optionally override the Broker’s tier selection and balancer strategies for realtime servers. If these properties are not set (the default), realtime servers continue to use the existing `druid.broker.select` and `druid.broker.balancer` configurations that apply to both historical and realtime servers.
+
+[#19062](https://github.com/apache/druid/pull/19062)
+
+### Manual Broker routing in the web console
+
+You can now configure which Broker the Router uses for queries issued from the web console. You may want to do this if there are Brokers that don't have visibility into certain data tiers, and you know you're querying data available only on a certain tier.
+
+To specify a Broker, add the following config to `web-console/console-config.js`:
+
+```js
+consoleBrokerService: 'druid/BROKER_NAME'
+```
+
+[#19069](https://github.com/apache/druid/pull/19069)
+
+### Consul extension
+
+The contributor extension `druid-consul-extensions` lets Druid clusters use Consul for service discovery and
+Coordinator/Overlord leader election instead of ZooKeeper. The extension supports ACLs, TLS/mTLS, and metrics.
+
+Before you switch to Consul, you need to set
+`druid.serverview.type=http` and `druid.indexer.runner.type=httpRemote` cluster wide.
+
+[#18843](https://github.com/apache/druid/pull/18843)
+
## Functional area and related changes
This section contains detailed release notes separated by areas.
### Web console
+#### Changed storage column displays
+
+- Improved the compaction config view to
+- Renamed **Current size** to **Assigned size**.
+- Renamed **Max size** to **Effective size**. It now displays the smaller value between `max_size` and `storage_size`. The max size is still shown as a tooltip.
+- Changed usage calculation to use `effective_size`
+
+[#19007](https://github.com/apache/druid/pull/19007)
+
#### Other web console improvements
+- Added `workerDesc` to `WorkerStats`, which makes it easier to identify where a worker is running [#19171](https://github.com/apache/druid/pull/19171)
+- Added the Dart unique execution ID (`dartQueryId`) and the `sqlQueryId` to the **Details** pane in the web console [#19185](https://github.com/apache/druid/pull/19185)
+- Added support for showing completed Dart queries in the web console [#18940](https://github.com/apache/druid/pull/18940)
+- Added a detail dialog to the **Services** page [#18960](https://github.com/apache/druid/pull/18960)
+- Added icons to indicate when data is loaded into virtual storage, including a tooltip that shows all the counters for the data [#19010](https://github.com/apache/druid/pull/19010)
+- Added support for Dart reports [#18897](https://github.com/apache/druid/pull/18897)
+- Changed what's considered an active worker: any nonzero rows, files, bytes, frames, or wall time is enough to consider a worker active [#19183](https://github.com/apache/druid/pull/19183)
+- Changed the **Cancel query** option to show only if a query is in an accepted or running state [#19182](https://github.com/apache/druid/pull/19182)
+- Changed the ordering of the current Dart queries panel to show queries in the following order: RUNNING, ACCEPTED, and then COMPLETED. RUNNING and ACCEPTED queries are ordered by the most recent first (based on timestamp). COMPLETED queries are sorted by finish time [#19237](https://github.com/apache/druid/pull/19237)
+
### Ingestion
#### SQL-based ingestion
##### Other SQL-based ingestion improvements
+- Added support for virtual storage fabric when performing SQL-based ingestion [#18873](https://github.com/apache/druid/pull/18873)
+- Added support for `StorageMonitor` so that MSQ task engine tasks always emit `taskId` and `groupId` [#19048](https://github.com/apache/druid/pull/19048)
+- Improved worker cancellation [#18931](https://github.com/apache/druid/pull/18931)
+- Improved exception handling [#19234](https://github.com/apache/druid/pull/19234)
+
#### Streaming ingestion
+##### Changed how tasks get launched for autoscaling
+
+The behavior of `taskCountMin` and `taskCountStart` has been changed for autoscaling. Druid now computes the initial number of tasks to launch by checking the configs in the following order: `taskCountStart` (optional), then `taskCount` (in `ioConfig`), then `taskCountMin`.
+
+[#19091](https://github.com/apache/druid/pull/19091)
+
+##### Improved query isolation
+
+Added `serverPriorityToReplicas` parameter to the streaming supervisor specs (Kafka, Kinesis, and Rabbit). This allows operators to distribute task replicas across different server priorities for realtime indexing tasks. Similar to Historical tiering, this enables query isolation for mixed workload scenarios on the Peons, allowing some task replicas to handle queries of specific priorities.
+
+[#19040](https://github.com/apache/druid/pull/19040)
+
##### Other streaming ingestion improvements
+- Improved cost-based autoscaler performance in high lag scenarios [#19045](https://github.com/apache/druid/pull/19045)
+- Improved the performance of realtime task scheduling by ordering schedule requests by priority on the `TaskQueue` [#19203](https://github.com/apache/druid/pull/19203)
+
+
+#### Ingestion improvements
+
+- Sped up task scheduling on the Overlord [#19199](https://github.com/apache/druid/pull/19199)
+
### Querying
+#### groupBy query configuration
+
+Added a new groupBy query configuration property `druid.query.groupBy.maxSpillFileCount` to limit the maximum number of spill files created per query. When the limit is exceeded, the query fails with a clear error message instead of causing Historical nodes to run out of memory during spill file merging. The limit can also be overridden per query using the query context parameter `maxSpillFileCount`.
+
+[#19141](https://github.com/apache/druid/pull/19141)
+
+#### Improved handling of nested aggregates
+
+Druid can now merge two Aggregates with a Projection between them. For example, the following query:
+
+```sql
+SELECT
+ hr,
+ UPPER(t1.x) x,
+ SUM(t1.cnt) cnt,
+ MIN(t1.mn) mn,
+ MAX(t1.mx) mx
+FROM (
+ SELECT
+ floor(__time to hour) hr,
+ dim2 x,
+ COUNT(*) cnt,
+ MIN(m1 * 5) mn,
+ MAX(m1 + m2) mx
+ FROM druid.foo
+ WHERE dim2 IN ('abc', 'def', 'a', 'b', '')
+ GROUP BY 1, 2
+) t1
+WHERE t1.x IN ('abc', 'foo', 'bar', 'a', '')
+GROUP BY 1, 2
+```
+
+can be simplified to the following:
+
+```sql
+SELECT
+ FLOOR(__time TO hour) hr,
+ UPPER(dim2) x,
+ COUNT(*) cnt,
+ MIN(m1 * 5) mn,
+ MAX(m1 + m2) mx
+FROM druid.foo
+WHERE dim2 IN ('abc', 'a', '')
+GROUP BY 1, 2
+```
+
+[#18498](https://github.com/apache/druid/pull/18498)
+
#### Other querying improvements
+- Added `durationMs` to Dart query reports [#19169](https://github.com/apache/druid/pull/19169)
+- Improved error handling so that row signature column order is preserved when column analysis encounters an error [#19162](https://github.com/apache/druid/pull/19162)
+- Improved GROUP BY performance [#18952](https://github.com/apache/druid/pull/18952)
+- Improved expression filters to take advantage of specialized virtual columns when possible, resulting in better performance for the query [#18965](https://github.com/apache/druid/pull/18965)
+
### Cluster management
+#### New Broker tier selection strategies
+
+Operators can now configure two new Broker `TierSelectorStrategy` implementations:
+
+- `strict` - Only selects servers whose priorities match the configured list. Example configuration: `druid.broker.select.tier=strict` and `druid.broker.select.tier.strict.priorities=[1]`.
+- `pooled` - Pools servers across the configured priorities and selects among them, allowing queries to utilize multiple priority tiers for improved availability. Example configuration: `druid.broker.select.tier=pooled` and `druid.broker.select.tier.pooled.priorities=[2,1]`.
+
+These strategies can also be configured for realtime servers using `druid.broker.realtime.select.tier`.
Review Comment:
```suggestion
You can also use `druid.broker.realtime.select.tier` to configure these strategies for realtime servers.
```
##########
docs/release-info/release-notes.md:
##########
@@ -57,50 +57,551 @@ For tips about how to write a good release note, see [Release notes](https://git
This section contains important information about new and existing features.
+### Hadoop-based ingestion
+
+Support for Hadoop-based ingestion has been removed. The feature was deprecated in Druid 34.
+
+Use one of Druid's other supported ingestion methods, such as SQL-based ingestion or MiddleManager-less ingestion using Kubernetes.
+
+[#19109](https://github.com/apache/druid/pull/19109)
+
+### Query blocklist
+
+You can now create a query blocklist for dynamically blocking queries without restarting your deployment. You can block queries by datasource, query type, or query context using the new `/druid/coordinator/v1/config/broker` API. Block rules use `AND` logic, which means all criteria must match. Druid stores the rules in the metadata database.
+
+The following example blocks all groupBy queries on the `wikipedia` datasource with a query context parameter of `priority` equal to `0`:
+
+```
+POST /druid/coordinator/v1/config/broker
+ {
+ "queryBlocklist": [
+ {
+ "ruleName": "block-wikipedia-groupbys",
+ "dataSources": ["wikipedia"],
+ "queryTypes": ["groupBy"],
+ "contextMatches": {"priority": "0"}
+ }
+ ]
+ }
+```
+
+[#19011](https://github.com/apache/druid/pull/19011)
+
+### Minor compaction for Overlord-based compaction (experimental)
+
+You can now configure minor compaction. With minor compaction, Druid only compacts newly ingested segments while upgrading existing compacted segments. When Druid upgrades segments, it updates the metadata instead of using resources to compact it again. You can use the native compaction engine or the MSQ task engine.
+
+Use the `mostFragmentedFirst` compaction policy and set either a percentage of rows-based or byte-based threshold for minor compaction.
+
+[#19059](https://github.com/apache/druid/pull/19059) [#19205](https://github.com/apache/druid/pull/19205) [#19016](https://github.com/apache/druid/pull/19016)
+
+### Thrift input format
+
+As part of the Thrift contributor extension, Druid now supports Thrift-encoded data for Kafka and Kinesis streaming ingestion.
+
+[#19111](https://github.com/apache/druid/pull/19111)
+
+To use this feature, you must add `druid-thrift-extensions` to your extension load list.
+
+### Kubernetes-based task management
+
+This extension is now generally available.
+
+[#19128](https://github.com/apache/druid/pull/19128)
+
+### Tombstones
+
+Tombstones for JSON-based native batch ingestion (the `dropExisting` flag for `ioConfig`) are now generally available.
+
+[#19128](https://github.com/apache/druid/pull/19128)
+
+### Dynamic default query context
+
+You can now add default query context parameters as a dynamic configuration to the Broker. This allows you to override static defaults set in your runtime properties without restarting your deployment or having to update multiple queries individually. Druid resolves which query context parameters to use based on the following priority:
+
+1. The query context included with the query
+1. The query context set as a dynamic configuration on the Broker
+1. The query context parameters set in the runtime properties
+1. The defaults that ship with Druid
+
+Note that like other Broker dynamic configuration, this is best-effort. Settings may not be applied in certain
+cases, such as when a Broker has recently started and hasn't received the config yet, or if the
+Broker can't contact the Coordinator. If a query context parameter is critical for all your queries, set it in the runtime properties.
+
+[#19146](https://github.com/apache/druid/pull/19146)
+
+### `sys.queries` table (experimental)
+
+The new system queries table provides information about currently running and recently completed queries that use the Dart engine. This table is off by default. To enable the table, set the following config:
+
+```
+druid.sql.planner.enableSysQueriesTable = true
+```
+
+As part of this change, the `/druid/v2/sql/queries` API now supports an `includeComplete` parameter that shows recently completed queries.
+
+[#18923](https://github.com/apache/druid/pull/18923)
+
+### Auto-compaction with compaction supervisors
+
+Auto-compaction using compaction supervisors has been improved and is now generally available.
+
+As part of the improvement, instead of individual compacted segments having to store their full compaction state in the metadata store, states are now stored in a central location, a new `indexingStates` table, in the metadata store. Individual segments only need to store a unique reference (`indexing_state_fingerprint`) to their full compaction state.
+
+Since many segments in a single datasource share the same underlying compaction state, this greatly reduces metadata storage requirements for automatic compaction.
+
+For backwards compatibility, detailed compaction state will continue to be persisted in each segment for now. This will be removed in a future release.
+
+You can stop storing detailed compaction state by setting `storeCompactionStatePerSegment` to `false` in the cluster compaction config. If you turn it off and need to downgrade, Druid needs to re-compact any segments that have been compacted since you changed the config.
+
+This change has upgrade impacts for metadata storage and metadata caching. For more information, see the [Metadata storage for auto-compaction with compaction supervisors](#metadata-storage-for-auto-compaction-with-compaction-supervisors) upgrade note.
+
+[#19113](https://github.com/apache/druid/pull/19113) [#18844](https://github.com/apache/druid/pull/18844)
+
+### Broker tier selection for realtime servers
+
+Added `druid.broker.realtime.select.tier` and `druid.broker.realtime.balancer.type` on the Brokers to optionally override the Broker’s tier selection and balancer strategies for realtime servers. If these properties are not set (the default), realtime servers continue to use the existing `druid.broker.select` and `druid.broker.balancer` configurations that apply to both historical and realtime servers.
+
+[#19062](https://github.com/apache/druid/pull/19062)
+
+### Manual Broker routing in the web console
+
+You can now configure which Broker the Router uses for queries issued from the web console. You may want to do this if there are Brokers that don't have visibility into certain data tiers, and you know you're querying data available only on a certain tier.
+
+To specify a Broker, add the following config to `web-console/console-config.js`:
+
+```js
+consoleBrokerService: 'druid/BROKER_NAME'
+```
+
+[#19069](https://github.com/apache/druid/pull/19069)
+
+### Consul extension
+
+The contributor extension `druid-consul-extensions` lets Druid clusters use Consul for service discovery and
+Coordinator/Overlord leader election instead of ZooKeeper. The extension supports ACLs, TLS/mTLS, and metrics.
+
+Before you switch to Consul, you need to set
+`druid.serverview.type=http` and `druid.indexer.runner.type=httpRemote` cluster wide.
+
+[#18843](https://github.com/apache/druid/pull/18843)
+
## Functional area and related changes
This section contains detailed release notes separated by areas.
### Web console
+#### Changed storage column displays
+
+- Improved the compaction config view to
+- Renamed **Current size** to **Assigned size**.
+- Renamed **Max size** to **Effective size**. It now displays the smaller value between `max_size` and `storage_size`. The max size is still shown as a tooltip.
+- Changed usage calculation to use `effective_size`
+
+[#19007](https://github.com/apache/druid/pull/19007)
+
#### Other web console improvements
+- Added `workerDesc` to `WorkerStats`, which makes it easier to identify where a worker is running [#19171](https://github.com/apache/druid/pull/19171)
+- Added the Dart unique execution ID (`dartQueryId`) and the `sqlQueryId` to the **Details** pane in the web console [#19185](https://github.com/apache/druid/pull/19185)
+- Added support for showing completed Dart queries in the web console [#18940](https://github.com/apache/druid/pull/18940)
+- Added a detail dialog to the **Services** page [#18960](https://github.com/apache/druid/pull/18960)
+- Added icons to indicate when data is loaded into virtual storage, including a tooltip that shows all the counters for the data [#19010](https://github.com/apache/druid/pull/19010)
+- Added support for Dart reports [#18897](https://github.com/apache/druid/pull/18897)
+- Changed what's considered an active worker: any nonzero rows, files, bytes, frames, or wall time is enough to consider a worker active [#19183](https://github.com/apache/druid/pull/19183)
+- Changed the **Cancel query** option to show only if a query is in an accepted or running state [#19182](https://github.com/apache/druid/pull/19182)
+- Changed the ordering of the current Dart queries panel to show queries in the following order: RUNNING, ACCEPTED, and then COMPLETED. RUNNING and ACCEPTED queries are ordered by the most recent first (based on timestamp). COMPLETED queries are sorted by finish time [#19237](https://github.com/apache/druid/pull/19237)
+
### Ingestion
#### SQL-based ingestion
##### Other SQL-based ingestion improvements
+- Added support for virtual storage fabric when performing SQL-based ingestion [#18873](https://github.com/apache/druid/pull/18873)
+- Added support for `StorageMonitor` so that MSQ task engine tasks always emit `taskId` and `groupId` [#19048](https://github.com/apache/druid/pull/19048)
+- Improved worker cancellation [#18931](https://github.com/apache/druid/pull/18931)
+- Improved exception handling [#19234](https://github.com/apache/druid/pull/19234)
+
#### Streaming ingestion
+##### Changed how tasks get launched for autoscaling
+
+The behavior of `taskCountMin` and `taskCountStart` has been changed for autoscaling. Druid now computes the initial number of tasks to launch by checking the configs in the following order: `taskCountStart` (optional), then `taskCount` (in `ioConfig`), then `taskCountMin`.
+
+[#19091](https://github.com/apache/druid/pull/19091)
+
+##### Improved query isolation
+
+Added `serverPriorityToReplicas` parameter to the streaming supervisor specs (Kafka, Kinesis, and Rabbit). This allows operators to distribute task replicas across different server priorities for realtime indexing tasks. Similar to Historical tiering, this enables query isolation for mixed workload scenarios on the Peons, allowing some task replicas to handle queries of specific priorities.
+
+[#19040](https://github.com/apache/druid/pull/19040)
+
##### Other streaming ingestion improvements
+- Improved cost-based autoscaler performance in high lag scenarios [#19045](https://github.com/apache/druid/pull/19045)
+- Improved the performance of realtime task scheduling by ordering schedule requests by priority on the `TaskQueue` [#19203](https://github.com/apache/druid/pull/19203)
+
+
+#### Ingestion improvements
+
+- Sped up task scheduling on the Overlord [#19199](https://github.com/apache/druid/pull/19199)
+
### Querying
+#### groupBy query configuration
+
+Added a new groupBy query configuration property `druid.query.groupBy.maxSpillFileCount` to limit the maximum number of spill files created per query. When the limit is exceeded, the query fails with a clear error message instead of causing Historical nodes to run out of memory during spill file merging. The limit can also be overridden per query using the query context parameter `maxSpillFileCount`.
+
+[#19141](https://github.com/apache/druid/pull/19141)
+
+#### Improved handling of nested aggregates
+
+Druid can now merge two Aggregates with a Projection between them. For example, the following query:
+
+```sql
+SELECT
+ hr,
+ UPPER(t1.x) x,
+ SUM(t1.cnt) cnt,
+ MIN(t1.mn) mn,
+ MAX(t1.mx) mx
+FROM (
+ SELECT
+ floor(__time to hour) hr,
+ dim2 x,
+ COUNT(*) cnt,
+ MIN(m1 * 5) mn,
+ MAX(m1 + m2) mx
+ FROM druid.foo
+ WHERE dim2 IN ('abc', 'def', 'a', 'b', '')
+ GROUP BY 1, 2
+) t1
+WHERE t1.x IN ('abc', 'foo', 'bar', 'a', '')
+GROUP BY 1, 2
+```
+
+can be simplified to the following:
+
+```sql
+SELECT
+ FLOOR(__time TO hour) hr,
+ UPPER(dim2) x,
+ COUNT(*) cnt,
+ MIN(m1 * 5) mn,
+ MAX(m1 + m2) mx
+FROM druid.foo
+WHERE dim2 IN ('abc', 'a', '')
+GROUP BY 1, 2
+```
+
+[#18498](https://github.com/apache/druid/pull/18498)
+
#### Other querying improvements
+- Added `durationMs` to Dart query reports [#19169](https://github.com/apache/druid/pull/19169)
+- Improved error handling so that row signature column order is preserved when column analysis encounters an error [#19162](https://github.com/apache/druid/pull/19162)
+- Improved GROUP BY performance [#18952](https://github.com/apache/druid/pull/18952)
+- Improved expression filters to take advantage of specialized virtual columns when possible, resulting in better performance for the query [#18965](https://github.com/apache/druid/pull/18965)
+
### Cluster management
+#### New Broker tier selection strategies
+
+Operators can now configure two new Broker `TierSelectorStrategy` implementations:
+
+- `strict` - Only selects servers whose priorities match the configured list. Example configuration: `druid.broker.select.tier=strict` and `druid.broker.select.tier.strict.priorities=[1]`.
+- `pooled` - Pools servers across the configured priorities and selects among them, allowing queries to utilize multiple priority tiers for improved availability. Example configuration: `druid.broker.select.tier=pooled` and `druid.broker.select.tier.pooled.priorities=[2,1]`.
+
+These strategies can also be configured for realtime servers using `druid.broker.realtime.select.tier`.
+
+[#19094](https://github.com/apache/druid/pull/19094)
+
+#### Druid operator
+
+The Druid operator now resides in its own repository: [`apache/druid-operator`](https://github.com/apache/druid-operator).
+
+[#19156](https://github.com/apache/druid/pull/19156)
+
+#### Cost-based autoscaler algorithm
+
+The algorithm for cost-based autoscaling has been changed:
+
+- Scale up more aggressively when per-partition lag is meaningful
+- Relax the partitions-per-task increase limit based on lag severity and headroom
+- Keep behavior conservative near `taskCountMax` and avoid negative headroom effects
+
+[#18936](https://github.com/apache/druid/pull/18936)
+
#### Other cluster management improvements
+- Added a `ReadOnly` authorizer that allows all READ operations but denies any other operation, such as WRITE [#19243](https://github.com/apache/druid/pull/19243)
+- Added `/status/ready` endpoint for service health so that external load balancers can handle a graceful shutdown better [#19148](https://github.com/apache/druid/pull/19148)
+- Added a configurable option to scale-down during task run time for cost-based autoscaler [#18958](https://github.com/apache/druid/pull/18958)
+- Added `storage_size` to `sys.servers` to facilitate retrieving disk cache size for Historicals when using the virtual storage fabric [#18979](https://github.com/apache/druid/pull/18979)
+- Added a log for new task count computation for the cost-based auto scaler [#18929](https://github.com/apache/druid/pull/18929)
+- Changed how the scaling is calculated from a square root-based scaling formula to a logarithmic formula that provides better emergency recovery at low task counts and millions of lag [#18976](https://github.com/apache/druid/pull/18976)
+- Improved the load speed of cached segments during Historical startup [#18489](https://github.com/apache/druid/pull/18489)
+- Improved Broker startup time by parallelizing buffer initialization [#19025](https://github.com/apache/druid/pull/19025)
+- Improved the stack trace for MSQ task engine worker failures so that they're preserved [#19049](https://github.com/apache/druid/pull/19049)
+- Improved the performance of the cost-based autoscaler during loaded lag conditions [#18991](https://github.com/apache/druid/pull/18991)
+
### Data management
+#### Durable storage cleaner
+
+The durable storage cleaner now supports configurable time-based retention for MSQ query results. Previously, query results were retained for all known tasks, which was unreliable for completed tasks. With this change, query results are retained for a configurable time period based on the task creation time.
+
+The new configuration property `druid.msq.intermediate.storage.cleaner.durationToRetain` controls the retention period for query results. The default retention period is 6 hours.
+
+[#19074](https://github.com/apache/druid/pull/19074)
+
#### Other data management improvements
+- Added the `druid.storage.transfer.asyncHttpClientType` config that specifies which async HTTP client to use for S3 transfers: `crt` for Amazon CRT or `netty` for Netty NIO [#19249](https://github.com/apache/druid/pull/19249)
+- Added a mechanism to automatically clean up intermediary files on HDFS storage [#19187](https://github.com/apache/druid/pull/19187)
+- Changed the default value for `druid.indexing.formats.maxStringLength` to null from 0 [#19198](https://github.com/apache/druid/pull/19198)
+
### Metrics and monitoring
+#### Version field
+
+All Druid metrics now include `version` and `buildRevision` fields to help you identify which version of Druid is emitting a metric. You can use this information to verify that all nodes in a cluster are running the intended revision during rolling deployments.
+
+[#19123](https://github.com/apache/druid/pull/19123)
+
+#### Monitoring supervisor state
+
+Added a new `supervisor/count` metric when `SupervisorStatsMonitor` is enabled in `druid.monitoring.monitors`. The metric reports each supervisor’s state, such as `RUNNING` or `SUSPENDED`, for Prometheus, StatsD, and other metric systems.
+
+[#19114](https://github.com/apache/druid/pull/19114)
+
+#### Improved `groupBy` metrics
+
+`GroupByStatsMonitor` now provides the following metrics:
+
+- `mergeBuffer/bytesUsed`
+- `mergeBuffer/maxAcquisitionTimeNs`
+- `groupBy/maxSpilledBytes`
+- `groupBy/maxMergeDictionarySize`
+
+[#18731](https://github.com/apache/druid/pull/18731) [#18970](https://github.com/apache/druid/pull/18970) [#18934](https://github.com/apache/druid/pull/18934)
+
+#### Filtering metrics
+
+Operators can limit which metrics the logging emitter writes by setting `druid.emitter.logging.shouldFilterMetrics=true` and, if desired, `druid.emitter.logging.allowedMetricsPath` to a JSON object file where the keys are metric names. A missing custom file results in a warning and use of the bundled `defaultMetrics.json`. Alerts and other non-metric events are always logged.
+
+[#19030](https://github.com/apache/druid/pull/19030)
+
+#### New Broker metrics
+
+Added `segment/schemaCache/rowSignature/changed` and `segment/schemaCache/rowSignature/column/count` metrics to get visibility into when the Broker's segment metadata cache's row signature for each datasource is initialized and updated.
+
+[#18966](https://github.com/apache/druid/pull/18966)
+
+#### Other metrics and monitoring improvements
+
+- Added the following metrics to the default for Prometheus: `mergeBuffer/bytesUsed` and `mergeBuffer/maxBytesUsed` [#19110](https://github.com/apache/druid/pull/19110)
+- Added compaction mode to what gets emitted for the `compact/task/count` metric [#19151](https://github.com/apache/druid/pull/19151)
+- Added support for logging and emitting SQL dynamic parameter values [#19067](https://github.com/apache/druid/pull/19067)
+- Added `ingest/rows/published`, which gets emitted from all task types to denote the total row count of successfully published segments [#19177](https://github.com/apache/druid/pull/19177)
+- Added `queries` and `totalQueries` counters, which reflect queries made to realtime servers to get realtime data
+- Added `tier/storage/capacity` metric for the Coordinator to emit that is guaranteed to reflect the total `StorageLocation` size configured across all Historicals in a tier [#18962](https://github.com/apache/druid/pull/18962)
+- Added new metrics for virtual storage fabric to the MSQ task engine `ChannelCounters`: `loadBytes`, `loadTime`, `loadWait`, and `loadFiles` [#18971](https://github.com/apache/druid/pull/18971)
+- Added `storage/virtual/hit/bytes`, `storage/virtual/hold/count` and `storage/virtual/hold/bytes` metric to `StorageMonitor`
+[#18895](https://github.com/apache/druid/pull/18895) [#19217](https://github.com/apache/druid/pull/19217)
+- Added `supervisorId` dimension for streaming tasks to `TaskCountStatsMonitor` [#18920](https://github.com/apache/druid/pull/18920)
+- Changed `StorageMonitor` to always be on [#19048](https://github.com/apache/druid/pull/19048)
+- Improved the metrics for autoscalers, so that they all emit the same metrics: `supervisorId`, `dataSource`, and `stream` [#19097](https://github.com/apache/druid/pull/19097)
+
### Extensions
-### Documentation improvements
+#### gRPC
+
+The gRPC query extension now cancels in-flight queries when clients cancel or disconnect.
+
+[#19005](https://github.com/apache/druid/pull/19005)
+
+#### Iceberg
+
+##### GCS warehouse
+
+The Iceberg input source now supports GCS warehouses. To use this feature, you need to load the `druid-google-extensions` extension in addition to the Iceberg extension.
+[#19137](https://github.com/apache/druid/pull/19137)
+
+##### Filters
+
+You can now configure residual filters for non-partition columns when using the Iceberg input source. Set `residualFilterMode` in the Iceberg input source to one of the following:
+
+- `ignore`: (default) Ingest the residual rows with a warning log unless filtered by `transformSpec`.
+- `fail`: Fail the ingestion job when residual filters are detected. Use this to ensure that filters only target partition columns.
+
+[#18953](https://github.com/apache/druid/pull/18953)
+
+#### HDFS storage
+
+Added support for `lz4` compression. As part of this change, the following metrics are now available:
+
+- `hdfs/pull/size`
+- `hdfs/pull/duration`
+- `hdfs/push/size`
+- `hdfs/push/duration`
+
+[#18982](https://github.com/apache/druid/pull/18982)
## Upgrade notes and incompatible changes
### Upgrade notes
+#### Hadoop-based ingestion
+
+Support for Hadoop-based ingestion has been removed. The feature was deprecated in Druid 34.
+
+Use one of Druid's other supported ingestion methods, such as SQL-based ingestion or MiddleManager-less ingestion using Kubernetes.
+
+[#19109](https://github.com/apache/druid/pull/19109)
+
+#### Segment metadata cache on by default
+
+Starting in Druid 37, the segment metadata cache is on by default. This feature allows the Broker to cache segment metadata polled from the Coordinator, rather than having to fetch metadata for every query against the `sys.segments` table. This improves performance but increases memory usage on Brokers.
+
+The `druid.sql.planner.metadataSegmentCacheEnable` config controls this feature.
+
+[#19075](https://github.com/apache/druid/pull/19075)
+
+#### Streaming ingestion `parser`
+
+Support for the deprecated `parser` has been removed for streaming ingest tasks such as Kafka and Kinesis. Operators must now specify `inputSource`/`inputFormat` on the `ioConfig` of the supervisor spec, and the `dataSchema` must not specify a parser. Do this before upgrading to Druid 37 or newer.
+
+[#19173](https://github.com/apache/druid/pull/19173) [#19166](https://github.com/apache/druid/pull/19166)
+
+#### Rolling upgrades from Druid versions prior to version 0.23
+
+You can't perform a rolling upgrade from versions earlier than Druid 0.23.
+
+[#18961](https://github.com/apache/druid/pull/18961)
+
+#### Metadata storage for auto-compaction with compaction supervisors
+
+Automatic compaction using compaction supervisors now requires incremental segment metadata caching to be enabled on the Overlord and Coordinator via runtime properties. Specifically, the `druid.manager.segments.useIncrementalCache` config must be set to `always` or `ifSynced`. For more information about the config, see [Segment metadata cache](https://druid.apache.org/docs/latest/configuration/#segment-metadata-cache-experimental).
Review Comment:
```suggestion
Automatic compaction using compaction supervisors now requires incremental segment metadata caching to be enabled on the Overlord and Coordinator in the runtime properties. Specifically, the `druid.manager.segments.useIncrementalCache` config must be set to `always` or `ifSynced`. For more information about the config, see [Segment metadata cache](https://druid.apache.org/docs/latest/configuration/#segment-metadata-cache-experimental).
```
##########
docs/release-info/release-notes.md:
##########
@@ -57,50 +57,551 @@ For tips about how to write a good release note, see [Release notes](https://git
This section contains important information about new and existing features.
+### Hadoop-based ingestion
+
+Support for Hadoop-based ingestion has been removed. The feature was deprecated in Druid 34.
+
+Use one of Druid's other supported ingestion methods, such as SQL-based ingestion or MiddleManager-less ingestion using Kubernetes.
+
+[#19109](https://github.com/apache/druid/pull/19109)
+
+### Query blocklist
+
+You can now create a query blocklist for dynamically blocking queries without restarting your deployment. You can block queries by datasource, query type, or query context using the new `/druid/coordinator/v1/config/broker` API. Block rules use `AND` logic, which means all criteria must match. Druid stores the rules in the metadata database.
+
+The following example blocks all groupBy queries on the `wikipedia` datasource with a query context parameter of `priority` equal to `0`:
+
+```
+POST /druid/coordinator/v1/config/broker
+ {
+ "queryBlocklist": [
+ {
+ "ruleName": "block-wikipedia-groupbys",
+ "dataSources": ["wikipedia"],
+ "queryTypes": ["groupBy"],
+ "contextMatches": {"priority": "0"}
+ }
+ ]
+ }
+```
+
+[#19011](https://github.com/apache/druid/pull/19011)
+
+### Minor compaction for Overlord-based compaction (experimental)
+
+You can now configure minor compaction. With minor compaction, Druid only compacts newly ingested segments while upgrading existing compacted segments. When Druid upgrades segments, it updates the metadata instead of using resources to compact it again. You can use the native compaction engine or the MSQ task engine.
Review Comment:
```suggestion
You can now configure minor compaction to compact only newly ingested segments while upgrading existing compacted segments. When Druid upgrades segments, it updates the metadata instead of using resources to compact it again. You can use the native compaction engine or the MSQ task engine.
```
reduce repetittion
##########
docs/release-info/release-notes.md:
##########
@@ -57,50 +57,551 @@ For tips about how to write a good release note, see [Release notes](https://git
This section contains important information about new and existing features.
+### Hadoop-based ingestion
+
+Support for Hadoop-based ingestion has been removed. The feature was deprecated in Druid 34.
+
+Use one of Druid's other supported ingestion methods, such as SQL-based ingestion or MiddleManager-less ingestion using Kubernetes.
+
+[#19109](https://github.com/apache/druid/pull/19109)
+
+### Query blocklist
+
+You can now create a query blocklist for dynamically blocking queries without restarting your deployment. You can block queries by datasource, query type, or query context using the new `/druid/coordinator/v1/config/broker` API. Block rules use `AND` logic, which means all criteria must match. Druid stores the rules in the metadata database.
+
+The following example blocks all groupBy queries on the `wikipedia` datasource with a query context parameter of `priority` equal to `0`:
+
+```
+POST /druid/coordinator/v1/config/broker
+ {
+ "queryBlocklist": [
+ {
+ "ruleName": "block-wikipedia-groupbys",
+ "dataSources": ["wikipedia"],
+ "queryTypes": ["groupBy"],
+ "contextMatches": {"priority": "0"}
+ }
+ ]
+ }
+```
+
+[#19011](https://github.com/apache/druid/pull/19011)
+
+### Minor compaction for Overlord-based compaction (experimental)
+
+You can now configure minor compaction. With minor compaction, Druid only compacts newly ingested segments while upgrading existing compacted segments. When Druid upgrades segments, it updates the metadata instead of using resources to compact it again. You can use the native compaction engine or the MSQ task engine.
+
+Use the `mostFragmentedFirst` compaction policy and set either a percentage of rows-based or byte-based threshold for minor compaction.
+
+[#19059](https://github.com/apache/druid/pull/19059) [#19205](https://github.com/apache/druid/pull/19205) [#19016](https://github.com/apache/druid/pull/19016)
+
+### Thrift input format
+
+As part of the Thrift contributor extension, Druid now supports Thrift-encoded data for Kafka and Kinesis streaming ingestion.
+
+[#19111](https://github.com/apache/druid/pull/19111)
+
+To use this feature, you must add `druid-thrift-extensions` to your extension load list.
+
+### Kubernetes-based task management
+
+This extension is now generally available.
+
+[#19128](https://github.com/apache/druid/pull/19128)
+
+### Tombstones
+
+Tombstones for JSON-based native batch ingestion (the `dropExisting` flag for `ioConfig`) are now generally available.
+
+[#19128](https://github.com/apache/druid/pull/19128)
+
+### Dynamic default query context
+
+You can now add default query context parameters as a dynamic configuration to the Broker. This allows you to override static defaults set in your runtime properties without restarting your deployment or having to update multiple queries individually. Druid resolves which query context parameters to use based on the following priority:
+
+1. The query context included with the query
+1. The query context set as a dynamic configuration on the Broker
+1. The query context parameters set in the runtime properties
+1. The defaults that ship with Druid
+
+Note that like other Broker dynamic configuration, this is best-effort. Settings may not be applied in certain
+cases, such as when a Broker has recently started and hasn't received the config yet, or if the
+Broker can't contact the Coordinator. If a query context parameter is critical for all your queries, set it in the runtime properties.
+
+[#19146](https://github.com/apache/druid/pull/19146)
+
+### `sys.queries` table (experimental)
+
+The new system queries table provides information about currently running and recently completed queries that use the Dart engine. This table is off by default. To enable the table, set the following config:
+
+```
+druid.sql.planner.enableSysQueriesTable = true
+```
+
+As part of this change, the `/druid/v2/sql/queries` API now supports an `includeComplete` parameter that shows recently completed queries.
+
+[#18923](https://github.com/apache/druid/pull/18923)
+
+### Auto-compaction with compaction supervisors
+
+Auto-compaction using compaction supervisors has been improved and is now generally available.
+
+As part of the improvement, instead of individual compacted segments having to store their full compaction state in the metadata store, states are now stored in a central location, a new `indexingStates` table, in the metadata store. Individual segments only need to store a unique reference (`indexing_state_fingerprint`) to their full compaction state.
+
+Since many segments in a single datasource share the same underlying compaction state, this greatly reduces metadata storage requirements for automatic compaction.
+
+For backwards compatibility, detailed compaction state will continue to be persisted in each segment for now. This will be removed in a future release.
+
+You can stop storing detailed compaction state by setting `storeCompactionStatePerSegment` to `false` in the cluster compaction config. If you turn it off and need to downgrade, Druid needs to re-compact any segments that have been compacted since you changed the config.
+
+This change has upgrade impacts for metadata storage and metadata caching. For more information, see the [Metadata storage for auto-compaction with compaction supervisors](#metadata-storage-for-auto-compaction-with-compaction-supervisors) upgrade note.
+
+[#19113](https://github.com/apache/druid/pull/19113) [#18844](https://github.com/apache/druid/pull/18844)
+
+### Broker tier selection for realtime servers
+
+Added `druid.broker.realtime.select.tier` and `druid.broker.realtime.balancer.type` on the Brokers to optionally override the Broker’s tier selection and balancer strategies for realtime servers. If these properties are not set (the default), realtime servers continue to use the existing `druid.broker.select` and `druid.broker.balancer` configurations that apply to both historical and realtime servers.
+
+[#19062](https://github.com/apache/druid/pull/19062)
+
+### Manual Broker routing in the web console
+
+You can now configure which Broker the Router uses for queries issued from the web console. You may want to do this if there are Brokers that don't have visibility into certain data tiers, and you know you're querying data available only on a certain tier.
+
+To specify a Broker, add the following config to `web-console/console-config.js`:
+
+```js
+consoleBrokerService: 'druid/BROKER_NAME'
+```
+
+[#19069](https://github.com/apache/druid/pull/19069)
+
+### Consul extension
+
+The contributor extension `druid-consul-extensions` lets Druid clusters use Consul for service discovery and
+Coordinator/Overlord leader election instead of ZooKeeper. The extension supports ACLs, TLS/mTLS, and metrics.
+
+Before you switch to Consul, you need to set
+`druid.serverview.type=http` and `druid.indexer.runner.type=httpRemote` cluster wide.
+
+[#18843](https://github.com/apache/druid/pull/18843)
+
## Functional area and related changes
This section contains detailed release notes separated by areas.
### Web console
+#### Changed storage column displays
+
+- Improved the compaction config view to
+- Renamed **Current size** to **Assigned size**.
+- Renamed **Max size** to **Effective size**. It now displays the smaller value between `max_size` and `storage_size`. The max size is still shown as a tooltip.
+- Changed usage calculation to use `effective_size`
+
+[#19007](https://github.com/apache/druid/pull/19007)
+
#### Other web console improvements
+- Added `workerDesc` to `WorkerStats`, which makes it easier to identify where a worker is running [#19171](https://github.com/apache/druid/pull/19171)
+- Added the Dart unique execution ID (`dartQueryId`) and the `sqlQueryId` to the **Details** pane in the web console [#19185](https://github.com/apache/druid/pull/19185)
+- Added support for showing completed Dart queries in the web console [#18940](https://github.com/apache/druid/pull/18940)
+- Added a detail dialog to the **Services** page [#18960](https://github.com/apache/druid/pull/18960)
+- Added icons to indicate when data is loaded into virtual storage, including a tooltip that shows all the counters for the data [#19010](https://github.com/apache/druid/pull/19010)
+- Added support for Dart reports [#18897](https://github.com/apache/druid/pull/18897)
+- Changed what's considered an active worker: any nonzero rows, files, bytes, frames, or wall time is enough to consider a worker active [#19183](https://github.com/apache/druid/pull/19183)
+- Changed the **Cancel query** option to show only if a query is in an accepted or running state [#19182](https://github.com/apache/druid/pull/19182)
+- Changed the ordering of the current Dart queries panel to show queries in the following order: RUNNING, ACCEPTED, and then COMPLETED. RUNNING and ACCEPTED queries are ordered by the most recent first (based on timestamp). COMPLETED queries are sorted by finish time [#19237](https://github.com/apache/druid/pull/19237)
+
### Ingestion
#### SQL-based ingestion
##### Other SQL-based ingestion improvements
+- Added support for virtual storage fabric when performing SQL-based ingestion [#18873](https://github.com/apache/druid/pull/18873)
+- Added support for `StorageMonitor` so that MSQ task engine tasks always emit `taskId` and `groupId` [#19048](https://github.com/apache/druid/pull/19048)
+- Improved worker cancellation [#18931](https://github.com/apache/druid/pull/18931)
+- Improved exception handling [#19234](https://github.com/apache/druid/pull/19234)
+
#### Streaming ingestion
+##### Changed how tasks get launched for autoscaling
+
+The behavior of `taskCountMin` and `taskCountStart` has been changed for autoscaling. Druid now computes the initial number of tasks to launch by checking the configs in the following order: `taskCountStart` (optional), then `taskCount` (in `ioConfig`), then `taskCountMin`.
+
+[#19091](https://github.com/apache/druid/pull/19091)
+
+##### Improved query isolation
+
+Added `serverPriorityToReplicas` parameter to the streaming supervisor specs (Kafka, Kinesis, and Rabbit). This allows operators to distribute task replicas across different server priorities for realtime indexing tasks. Similar to Historical tiering, this enables query isolation for mixed workload scenarios on the Peons, allowing some task replicas to handle queries of specific priorities.
+
+[#19040](https://github.com/apache/druid/pull/19040)
+
##### Other streaming ingestion improvements
+- Improved cost-based autoscaler performance in high lag scenarios [#19045](https://github.com/apache/druid/pull/19045)
+- Improved the performance of realtime task scheduling by ordering schedule requests by priority on the `TaskQueue` [#19203](https://github.com/apache/druid/pull/19203)
+
+
+#### Ingestion improvements
+
+- Sped up task scheduling on the Overlord [#19199](https://github.com/apache/druid/pull/19199)
+
### Querying
+#### groupBy query configuration
+
+Added a new groupBy query configuration property `druid.query.groupBy.maxSpillFileCount` to limit the maximum number of spill files created per query. When the limit is exceeded, the query fails with a clear error message instead of causing Historical nodes to run out of memory during spill file merging. The limit can also be overridden per query using the query context parameter `maxSpillFileCount`.
+
+[#19141](https://github.com/apache/druid/pull/19141)
+
+#### Improved handling of nested aggregates
+
+Druid can now merge two Aggregates with a Projection between them. For example, the following query:
+
+```sql
+SELECT
+ hr,
+ UPPER(t1.x) x,
+ SUM(t1.cnt) cnt,
+ MIN(t1.mn) mn,
+ MAX(t1.mx) mx
+FROM (
+ SELECT
+ floor(__time to hour) hr,
+ dim2 x,
+ COUNT(*) cnt,
+ MIN(m1 * 5) mn,
+ MAX(m1 + m2) mx
+ FROM druid.foo
+ WHERE dim2 IN ('abc', 'def', 'a', 'b', '')
+ GROUP BY 1, 2
+) t1
+WHERE t1.x IN ('abc', 'foo', 'bar', 'a', '')
+GROUP BY 1, 2
+```
+
+can be simplified to the following:
+
+```sql
+SELECT
+ FLOOR(__time TO hour) hr,
+ UPPER(dim2) x,
+ COUNT(*) cnt,
+ MIN(m1 * 5) mn,
+ MAX(m1 + m2) mx
+FROM druid.foo
+WHERE dim2 IN ('abc', 'a', '')
+GROUP BY 1, 2
+```
+
+[#18498](https://github.com/apache/druid/pull/18498)
+
#### Other querying improvements
+- Added `durationMs` to Dart query reports [#19169](https://github.com/apache/druid/pull/19169)
+- Improved error handling so that row signature column order is preserved when column analysis encounters an error [#19162](https://github.com/apache/druid/pull/19162)
+- Improved GROUP BY performance [#18952](https://github.com/apache/druid/pull/18952)
+- Improved expression filters to take advantage of specialized virtual columns when possible, resulting in better performance for the query [#18965](https://github.com/apache/druid/pull/18965)
+
### Cluster management
+#### New Broker tier selection strategies
+
+Operators can now configure two new Broker `TierSelectorStrategy` implementations:
+
+- `strict` - Only selects servers whose priorities match the configured list. Example configuration: `druid.broker.select.tier=strict` and `druid.broker.select.tier.strict.priorities=[1]`.
+- `pooled` - Pools servers across the configured priorities and selects among them, allowing queries to utilize multiple priority tiers for improved availability. Example configuration: `druid.broker.select.tier=pooled` and `druid.broker.select.tier.pooled.priorities=[2,1]`.
+
+These strategies can also be configured for realtime servers using `druid.broker.realtime.select.tier`.
+
+[#19094](https://github.com/apache/druid/pull/19094)
+
+#### Druid operator
+
+The Druid operator now resides in its own repository: [`apache/druid-operator`](https://github.com/apache/druid-operator).
+
+[#19156](https://github.com/apache/druid/pull/19156)
+
+#### Cost-based autoscaler algorithm
+
+The algorithm for cost-based autoscaling has been changed:
+
+- Scale up more aggressively when per-partition lag is meaningful
+- Relax the partitions-per-task increase limit based on lag severity and headroom
+- Keep behavior conservative near `taskCountMax` and avoid negative headroom effects
+
+[#18936](https://github.com/apache/druid/pull/18936)
+
#### Other cluster management improvements
+- Added a `ReadOnly` authorizer that allows all READ operations but denies any other operation, such as WRITE [#19243](https://github.com/apache/druid/pull/19243)
+- Added `/status/ready` endpoint for service health so that external load balancers can handle a graceful shutdown better [#19148](https://github.com/apache/druid/pull/19148)
+- Added a configurable option to scale-down during task run time for cost-based autoscaler [#18958](https://github.com/apache/druid/pull/18958)
+- Added `storage_size` to `sys.servers` to facilitate retrieving disk cache size for Historicals when using the virtual storage fabric [#18979](https://github.com/apache/druid/pull/18979)
+- Added a log for new task count computation for the cost-based auto scaler [#18929](https://github.com/apache/druid/pull/18929)
+- Changed how the scaling is calculated from a square root-based scaling formula to a logarithmic formula that provides better emergency recovery at low task counts and millions of lag [#18976](https://github.com/apache/druid/pull/18976)
+- Improved the load speed of cached segments during Historical startup [#18489](https://github.com/apache/druid/pull/18489)
+- Improved Broker startup time by parallelizing buffer initialization [#19025](https://github.com/apache/druid/pull/19025)
+- Improved the stack trace for MSQ task engine worker failures so that they're preserved [#19049](https://github.com/apache/druid/pull/19049)
+- Improved the performance of the cost-based autoscaler during loaded lag conditions [#18991](https://github.com/apache/druid/pull/18991)
+
### Data management
+#### Durable storage cleaner
+
+The durable storage cleaner now supports configurable time-based retention for MSQ query results. Previously, query results were retained for all known tasks, which was unreliable for completed tasks. With this change, query results are retained for a configurable time period based on the task creation time.
+
+The new configuration property `druid.msq.intermediate.storage.cleaner.durationToRetain` controls the retention period for query results. The default retention period is 6 hours.
+
+[#19074](https://github.com/apache/druid/pull/19074)
+
#### Other data management improvements
+- Added the `druid.storage.transfer.asyncHttpClientType` config that specifies which async HTTP client to use for S3 transfers: `crt` for Amazon CRT or `netty` for Netty NIO [#19249](https://github.com/apache/druid/pull/19249)
+- Added a mechanism to automatically clean up intermediary files on HDFS storage [#19187](https://github.com/apache/druid/pull/19187)
+- Changed the default value for `druid.indexing.formats.maxStringLength` to null from 0 [#19198](https://github.com/apache/druid/pull/19198)
+
### Metrics and monitoring
+#### Version field
+
+All Druid metrics now include `version` and `buildRevision` fields to help you identify which version of Druid is emitting a metric. You can use this information to verify that all nodes in a cluster are running the intended revision during rolling deployments.
+
+[#19123](https://github.com/apache/druid/pull/19123)
+
+#### Monitoring supervisor state
+
+Added a new `supervisor/count` metric when `SupervisorStatsMonitor` is enabled in `druid.monitoring.monitors`. The metric reports each supervisor’s state, such as `RUNNING` or `SUSPENDED`, for Prometheus, StatsD, and other metric systems.
+
+[#19114](https://github.com/apache/druid/pull/19114)
+
+#### Improved `groupBy` metrics
+
+`GroupByStatsMonitor` now provides the following metrics:
+
+- `mergeBuffer/bytesUsed`
+- `mergeBuffer/maxAcquisitionTimeNs`
+- `groupBy/maxSpilledBytes`
+- `groupBy/maxMergeDictionarySize`
+
+[#18731](https://github.com/apache/druid/pull/18731) [#18970](https://github.com/apache/druid/pull/18970) [#18934](https://github.com/apache/druid/pull/18934)
+
+#### Filtering metrics
+
+Operators can limit which metrics the logging emitter writes by setting `druid.emitter.logging.shouldFilterMetrics=true` and, if desired, `druid.emitter.logging.allowedMetricsPath` to a JSON object file where the keys are metric names. A missing custom file results in a warning and use of the bundled `defaultMetrics.json`. Alerts and other non-metric events are always logged.
+
+[#19030](https://github.com/apache/druid/pull/19030)
+
+#### New Broker metrics
+
+Added `segment/schemaCache/rowSignature/changed` and `segment/schemaCache/rowSignature/column/count` metrics to get visibility into when the Broker's segment metadata cache's row signature for each datasource is initialized and updated.
+
+[#18966](https://github.com/apache/druid/pull/18966)
+
+#### Other metrics and monitoring improvements
+
+- Added the following metrics to the default for Prometheus: `mergeBuffer/bytesUsed` and `mergeBuffer/maxBytesUsed` [#19110](https://github.com/apache/druid/pull/19110)
+- Added compaction mode to what gets emitted for the `compact/task/count` metric [#19151](https://github.com/apache/druid/pull/19151)
+- Added support for logging and emitting SQL dynamic parameter values [#19067](https://github.com/apache/druid/pull/19067)
+- Added `ingest/rows/published`, which gets emitted from all task types to denote the total row count of successfully published segments [#19177](https://github.com/apache/druid/pull/19177)
Review Comment:
```suggestion
- Added `ingest/rows/published`, which all task types emit to denote the total row count of successfully published segments [#19177](https://github.com/apache/druid/pull/19177)
```
##########
docs/release-info/release-notes.md:
##########
@@ -57,50 +57,551 @@ For tips about how to write a good release note, see [Release notes](https://git
This section contains important information about new and existing features.
+### Hadoop-based ingestion
+
+Support for Hadoop-based ingestion has been removed. The feature was deprecated in Druid 34.
+
+Use one of Druid's other supported ingestion methods, such as SQL-based ingestion or MiddleManager-less ingestion using Kubernetes.
+
+[#19109](https://github.com/apache/druid/pull/19109)
+
+### Query blocklist
+
+You can now create a query blocklist for dynamically blocking queries without restarting your deployment. You can block queries by datasource, query type, or query context using the new `/druid/coordinator/v1/config/broker` API. Block rules use `AND` logic, which means all criteria must match. Druid stores the rules in the metadata database.
Review Comment:
```suggestion
You can now use the using the `/druid/coordinator/v1/config/broker` API tp create a query blocklist to dynamically block queries by datasource, query type, or query context. The blocklist takes effect without a restarting Druid. Block rules use `AND` logic, which means all criteria must match.
```
Style suggestion. Also, not sure that it's important to note that the rules are stored in the metadata db.
##########
docs/release-info/release-notes.md:
##########
@@ -57,50 +57,551 @@ For tips about how to write a good release note, see [Release notes](https://git
This section contains important information about new and existing features.
+### Hadoop-based ingestion
+
+Support for Hadoop-based ingestion has been removed. The feature was deprecated in Druid 34.
+
+Use one of Druid's other supported ingestion methods, such as SQL-based ingestion or MiddleManager-less ingestion using Kubernetes.
+
+[#19109](https://github.com/apache/druid/pull/19109)
+
+### Query blocklist
+
+You can now create a query blocklist for dynamically blocking queries without restarting your deployment. You can block queries by datasource, query type, or query context using the new `/druid/coordinator/v1/config/broker` API. Block rules use `AND` logic, which means all criteria must match. Druid stores the rules in the metadata database.
+
+The following example blocks all groupBy queries on the `wikipedia` datasource with a query context parameter of `priority` equal to `0`:
+
+```
+POST /druid/coordinator/v1/config/broker
+ {
+ "queryBlocklist": [
+ {
+ "ruleName": "block-wikipedia-groupbys",
+ "dataSources": ["wikipedia"],
+ "queryTypes": ["groupBy"],
+ "contextMatches": {"priority": "0"}
+ }
+ ]
+ }
+```
+
+[#19011](https://github.com/apache/druid/pull/19011)
+
+### Minor compaction for Overlord-based compaction (experimental)
+
+You can now configure minor compaction. With minor compaction, Druid only compacts newly ingested segments while upgrading existing compacted segments. When Druid upgrades segments, it updates the metadata instead of using resources to compact it again. You can use the native compaction engine or the MSQ task engine.
+
+Use the `mostFragmentedFirst` compaction policy and set either a percentage of rows-based or byte-based threshold for minor compaction.
+
+[#19059](https://github.com/apache/druid/pull/19059) [#19205](https://github.com/apache/druid/pull/19205) [#19016](https://github.com/apache/druid/pull/19016)
+
+### Thrift input format
+
+As part of the Thrift contributor extension, Druid now supports Thrift-encoded data for Kafka and Kinesis streaming ingestion.
+
+[#19111](https://github.com/apache/druid/pull/19111)
+
+To use this feature, you must add `druid-thrift-extensions` to your extension load list.
+
+### Kubernetes-based task management
+
+This extension is now generally available.
+
+[#19128](https://github.com/apache/druid/pull/19128)
+
+### Tombstones
+
+Tombstones for JSON-based native batch ingestion (the `dropExisting` flag for `ioConfig`) are now generally available.
+
+[#19128](https://github.com/apache/druid/pull/19128)
+
+### Dynamic default query context
+
+You can now add default query context parameters as a dynamic configuration to the Broker. This allows you to override static defaults set in your runtime properties without restarting your deployment or having to update multiple queries individually. Druid resolves which query context parameters to use based on the following priority:
+
+1. The query context included with the query
+1. The query context set as a dynamic configuration on the Broker
+1. The query context parameters set in the runtime properties
+1. The defaults that ship with Druid
+
+Note that like other Broker dynamic configuration, this is best-effort. Settings may not be applied in certain
+cases, such as when a Broker has recently started and hasn't received the config yet, or if the
+Broker can't contact the Coordinator. If a query context parameter is critical for all your queries, set it in the runtime properties.
+
+[#19146](https://github.com/apache/druid/pull/19146)
+
+### `sys.queries` table (experimental)
+
+The new system queries table provides information about currently running and recently completed queries that use the Dart engine. This table is off by default. To enable the table, set the following config:
+
+```
+druid.sql.planner.enableSysQueriesTable = true
+```
+
+As part of this change, the `/druid/v2/sql/queries` API now supports an `includeComplete` parameter that shows recently completed queries.
+
+[#18923](https://github.com/apache/druid/pull/18923)
+
+### Auto-compaction with compaction supervisors
+
+Auto-compaction using compaction supervisors has been improved and is now generally available.
+
+As part of the improvement, instead of individual compacted segments having to store their full compaction state in the metadata store, states are now stored in a central location, a new `indexingStates` table, in the metadata store. Individual segments only need to store a unique reference (`indexing_state_fingerprint`) to their full compaction state.
Review Comment:
```suggestion
As part of the improvement compaction states are now stored in a central location, a new `indexingStates` table. Individual segments only need to store a unique reference (`indexing_state_fingerprint`) to their full compaction state.
```
Not sure why the previous functionality is relevant
##########
docs/release-info/release-notes.md:
##########
@@ -57,50 +57,551 @@ For tips about how to write a good release note, see [Release notes](https://git
This section contains important information about new and existing features.
+### Hadoop-based ingestion
+
+Support for Hadoop-based ingestion has been removed. The feature was deprecated in Druid 34.
+
+Use one of Druid's other supported ingestion methods, such as SQL-based ingestion or MiddleManager-less ingestion using Kubernetes.
+
+[#19109](https://github.com/apache/druid/pull/19109)
+
+### Query blocklist
+
+You can now create a query blocklist for dynamically blocking queries without restarting your deployment. You can block queries by datasource, query type, or query context using the new `/druid/coordinator/v1/config/broker` API. Block rules use `AND` logic, which means all criteria must match. Druid stores the rules in the metadata database.
+
+The following example blocks all groupBy queries on the `wikipedia` datasource with a query context parameter of `priority` equal to `0`:
+
+```
+POST /druid/coordinator/v1/config/broker
+ {
+ "queryBlocklist": [
+ {
+ "ruleName": "block-wikipedia-groupbys",
+ "dataSources": ["wikipedia"],
+ "queryTypes": ["groupBy"],
+ "contextMatches": {"priority": "0"}
+ }
+ ]
+ }
+```
+
+[#19011](https://github.com/apache/druid/pull/19011)
+
+### Minor compaction for Overlord-based compaction (experimental)
+
+You can now configure minor compaction. With minor compaction, Druid only compacts newly ingested segments while upgrading existing compacted segments. When Druid upgrades segments, it updates the metadata instead of using resources to compact it again. You can use the native compaction engine or the MSQ task engine.
+
+Use the `mostFragmentedFirst` compaction policy and set either a percentage of rows-based or byte-based threshold for minor compaction.
+
+[#19059](https://github.com/apache/druid/pull/19059) [#19205](https://github.com/apache/druid/pull/19205) [#19016](https://github.com/apache/druid/pull/19016)
+
+### Thrift input format
+
+As part of the Thrift contributor extension, Druid now supports Thrift-encoded data for Kafka and Kinesis streaming ingestion.
+
+[#19111](https://github.com/apache/druid/pull/19111)
+
+To use this feature, you must add `druid-thrift-extensions` to your extension load list.
+
+### Kubernetes-based task management
+
+This extension is now generally available.
+
+[#19128](https://github.com/apache/druid/pull/19128)
+
+### Tombstones
+
+Tombstones for JSON-based native batch ingestion (the `dropExisting` flag for `ioConfig`) are now generally available.
+
+[#19128](https://github.com/apache/druid/pull/19128)
+
+### Dynamic default query context
+
+You can now add default query context parameters as a dynamic configuration to the Broker. This allows you to override static defaults set in your runtime properties without restarting your deployment or having to update multiple queries individually. Druid resolves which query context parameters to use based on the following priority:
+
+1. The query context included with the query
+1. The query context set as a dynamic configuration on the Broker
+1. The query context parameters set in the runtime properties
+1. The defaults that ship with Druid
+
+Note that like other Broker dynamic configuration, this is best-effort. Settings may not be applied in certain
+cases, such as when a Broker has recently started and hasn't received the config yet, or if the
+Broker can't contact the Coordinator. If a query context parameter is critical for all your queries, set it in the runtime properties.
+
+[#19146](https://github.com/apache/druid/pull/19146)
+
+### `sys.queries` table (experimental)
+
+The new system queries table provides information about currently running and recently completed queries that use the Dart engine. This table is off by default. To enable the table, set the following config:
+
+```
+druid.sql.planner.enableSysQueriesTable = true
+```
+
+As part of this change, the `/druid/v2/sql/queries` API now supports an `includeComplete` parameter that shows recently completed queries.
+
+[#18923](https://github.com/apache/druid/pull/18923)
+
+### Auto-compaction with compaction supervisors
+
+Auto-compaction using compaction supervisors has been improved and is now generally available.
+
+As part of the improvement, instead of individual compacted segments having to store their full compaction state in the metadata store, states are now stored in a central location, a new `indexingStates` table, in the metadata store. Individual segments only need to store a unique reference (`indexing_state_fingerprint`) to their full compaction state.
+
+Since many segments in a single datasource share the same underlying compaction state, this greatly reduces metadata storage requirements for automatic compaction.
+
+For backwards compatibility, detailed compaction state will continue to be persisted in each segment for now. This will be removed in a future release.
+
+You can stop storing detailed compaction state by setting `storeCompactionStatePerSegment` to `false` in the cluster compaction config. If you turn it off and need to downgrade, Druid needs to re-compact any segments that have been compacted since you changed the config.
+
+This change has upgrade impacts for metadata storage and metadata caching. For more information, see the [Metadata storage for auto-compaction with compaction supervisors](#metadata-storage-for-auto-compaction-with-compaction-supervisors) upgrade note.
+
+[#19113](https://github.com/apache/druid/pull/19113) [#18844](https://github.com/apache/druid/pull/18844)
+
+### Broker tier selection for realtime servers
+
+Added `druid.broker.realtime.select.tier` and `druid.broker.realtime.balancer.type` on the Brokers to optionally override the Broker’s tier selection and balancer strategies for realtime servers. If these properties are not set (the default), realtime servers continue to use the existing `druid.broker.select` and `druid.broker.balancer` configurations that apply to both historical and realtime servers.
+
+[#19062](https://github.com/apache/druid/pull/19062)
+
+### Manual Broker routing in the web console
+
+You can now configure which Broker the Router uses for queries issued from the web console. You may want to do this if there are Brokers that don't have visibility into certain data tiers, and you know you're querying data available only on a certain tier.
+
+To specify a Broker, add the following config to `web-console/console-config.js`:
+
+```js
+consoleBrokerService: 'druid/BROKER_NAME'
+```
+
+[#19069](https://github.com/apache/druid/pull/19069)
+
+### Consul extension
+
+The contributor extension `druid-consul-extensions` lets Druid clusters use Consul for service discovery and
+Coordinator/Overlord leader election instead of ZooKeeper. The extension supports ACLs, TLS/mTLS, and metrics.
+
+Before you switch to Consul, you need to set
+`druid.serverview.type=http` and `druid.indexer.runner.type=httpRemote` cluster wide.
+
+[#18843](https://github.com/apache/druid/pull/18843)
+
## Functional area and related changes
This section contains detailed release notes separated by areas.
### Web console
+#### Changed storage column displays
+
+- Improved the compaction config view to
+- Renamed **Current size** to **Assigned size**.
+- Renamed **Max size** to **Effective size**. It now displays the smaller value between `max_size` and `storage_size`. The max size is still shown as a tooltip.
+- Changed usage calculation to use `effective_size`
+
+[#19007](https://github.com/apache/druid/pull/19007)
+
#### Other web console improvements
+- Added `workerDesc` to `WorkerStats`, which makes it easier to identify where a worker is running [#19171](https://github.com/apache/druid/pull/19171)
+- Added the Dart unique execution ID (`dartQueryId`) and the `sqlQueryId` to the **Details** pane in the web console [#19185](https://github.com/apache/druid/pull/19185)
+- Added support for showing completed Dart queries in the web console [#18940](https://github.com/apache/druid/pull/18940)
+- Added a detail dialog to the **Services** page [#18960](https://github.com/apache/druid/pull/18960)
+- Added icons to indicate when data is loaded into virtual storage, including a tooltip that shows all the counters for the data [#19010](https://github.com/apache/druid/pull/19010)
+- Added support for Dart reports [#18897](https://github.com/apache/druid/pull/18897)
+- Changed what's considered an active worker: any nonzero rows, files, bytes, frames, or wall time is enough to consider a worker active [#19183](https://github.com/apache/druid/pull/19183)
+- Changed the **Cancel query** option to show only if a query is in an accepted or running state [#19182](https://github.com/apache/druid/pull/19182)
+- Changed the ordering of the current Dart queries panel to show queries in the following order: RUNNING, ACCEPTED, and then COMPLETED. RUNNING and ACCEPTED queries are ordered by the most recent first (based on timestamp). COMPLETED queries are sorted by finish time [#19237](https://github.com/apache/druid/pull/19237)
+
### Ingestion
#### SQL-based ingestion
##### Other SQL-based ingestion improvements
+- Added support for virtual storage fabric when performing SQL-based ingestion [#18873](https://github.com/apache/druid/pull/18873)
+- Added support for `StorageMonitor` so that MSQ task engine tasks always emit `taskId` and `groupId` [#19048](https://github.com/apache/druid/pull/19048)
+- Improved worker cancellation [#18931](https://github.com/apache/druid/pull/18931)
+- Improved exception handling [#19234](https://github.com/apache/druid/pull/19234)
+
#### Streaming ingestion
+##### Changed how tasks get launched for autoscaling
+
+The behavior of `taskCountMin` and `taskCountStart` has been changed for autoscaling. Druid now computes the initial number of tasks to launch by checking the configs in the following order: `taskCountStart` (optional), then `taskCount` (in `ioConfig`), then `taskCountMin`.
+
+[#19091](https://github.com/apache/druid/pull/19091)
+
+##### Improved query isolation
+
+Added `serverPriorityToReplicas` parameter to the streaming supervisor specs (Kafka, Kinesis, and Rabbit). This allows operators to distribute task replicas across different server priorities for realtime indexing tasks. Similar to Historical tiering, this enables query isolation for mixed workload scenarios on the Peons, allowing some task replicas to handle queries of specific priorities.
+
+[#19040](https://github.com/apache/druid/pull/19040)
+
##### Other streaming ingestion improvements
+- Improved cost-based autoscaler performance in high lag scenarios [#19045](https://github.com/apache/druid/pull/19045)
+- Improved the performance of realtime task scheduling by ordering schedule requests by priority on the `TaskQueue` [#19203](https://github.com/apache/druid/pull/19203)
+
+
+#### Ingestion improvements
+
+- Sped up task scheduling on the Overlord [#19199](https://github.com/apache/druid/pull/19199)
+
### Querying
+#### groupBy query configuration
+
+Added a new groupBy query configuration property `druid.query.groupBy.maxSpillFileCount` to limit the maximum number of spill files created per query. When the limit is exceeded, the query fails with a clear error message instead of causing Historical nodes to run out of memory during spill file merging. The limit can also be overridden per query using the query context parameter `maxSpillFileCount`.
+
+[#19141](https://github.com/apache/druid/pull/19141)
+
+#### Improved handling of nested aggregates
+
+Druid can now merge two Aggregates with a Projection between them. For example, the following query:
+
+```sql
+SELECT
+ hr,
+ UPPER(t1.x) x,
+ SUM(t1.cnt) cnt,
+ MIN(t1.mn) mn,
+ MAX(t1.mx) mx
+FROM (
+ SELECT
+ floor(__time to hour) hr,
+ dim2 x,
+ COUNT(*) cnt,
+ MIN(m1 * 5) mn,
+ MAX(m1 + m2) mx
+ FROM druid.foo
+ WHERE dim2 IN ('abc', 'def', 'a', 'b', '')
+ GROUP BY 1, 2
+) t1
+WHERE t1.x IN ('abc', 'foo', 'bar', 'a', '')
+GROUP BY 1, 2
+```
+
+can be simplified to the following:
+
+```sql
+SELECT
+ FLOOR(__time TO hour) hr,
+ UPPER(dim2) x,
+ COUNT(*) cnt,
+ MIN(m1 * 5) mn,
+ MAX(m1 + m2) mx
+FROM druid.foo
+WHERE dim2 IN ('abc', 'a', '')
+GROUP BY 1, 2
+```
+
+[#18498](https://github.com/apache/druid/pull/18498)
+
#### Other querying improvements
+- Added `durationMs` to Dart query reports [#19169](https://github.com/apache/druid/pull/19169)
+- Improved error handling so that row signature column order is preserved when column analysis encounters an error [#19162](https://github.com/apache/druid/pull/19162)
+- Improved GROUP BY performance [#18952](https://github.com/apache/druid/pull/18952)
+- Improved expression filters to take advantage of specialized virtual columns when possible, resulting in better performance for the query [#18965](https://github.com/apache/druid/pull/18965)
+
### Cluster management
+#### New Broker tier selection strategies
+
+Operators can now configure two new Broker `TierSelectorStrategy` implementations:
+
+- `strict` - Only selects servers whose priorities match the configured list. Example configuration: `druid.broker.select.tier=strict` and `druid.broker.select.tier.strict.priorities=[1]`.
+- `pooled` - Pools servers across the configured priorities and selects among them, allowing queries to utilize multiple priority tiers for improved availability. Example configuration: `druid.broker.select.tier=pooled` and `druid.broker.select.tier.pooled.priorities=[2,1]`.
Review Comment:
```suggestion
- `pooled` - Pools servers across the configured priorities and selects among them, allowing queries to use multiple priority tiers for improved availability. Example configuration: `druid.broker.select.tier=pooled` and `druid.broker.select.tier.pooled.priorities=[2,1]`.
```
##########
docs/release-info/release-notes.md:
##########
@@ -57,50 +57,551 @@ For tips about how to write a good release note, see [Release notes](https://git
This section contains important information about new and existing features.
+### Hadoop-based ingestion
+
+Support for Hadoop-based ingestion has been removed. The feature was deprecated in Druid 34.
+
+Use one of Druid's other supported ingestion methods, such as SQL-based ingestion or MiddleManager-less ingestion using Kubernetes.
+
+[#19109](https://github.com/apache/druid/pull/19109)
+
+### Query blocklist
+
+You can now create a query blocklist for dynamically blocking queries without restarting your deployment. You can block queries by datasource, query type, or query context using the new `/druid/coordinator/v1/config/broker` API. Block rules use `AND` logic, which means all criteria must match. Druid stores the rules in the metadata database.
+
+The following example blocks all groupBy queries on the `wikipedia` datasource with a query context parameter of `priority` equal to `0`:
+
+```
+POST /druid/coordinator/v1/config/broker
+ {
+ "queryBlocklist": [
+ {
+ "ruleName": "block-wikipedia-groupbys",
+ "dataSources": ["wikipedia"],
+ "queryTypes": ["groupBy"],
+ "contextMatches": {"priority": "0"}
+ }
+ ]
+ }
+```
+
+[#19011](https://github.com/apache/druid/pull/19011)
+
+### Minor compaction for Overlord-based compaction (experimental)
+
+You can now configure minor compaction. With minor compaction, Druid only compacts newly ingested segments while upgrading existing compacted segments. When Druid upgrades segments, it updates the metadata instead of using resources to compact it again. You can use the native compaction engine or the MSQ task engine.
+
+Use the `mostFragmentedFirst` compaction policy and set either a percentage of rows-based or byte-based threshold for minor compaction.
+
+[#19059](https://github.com/apache/druid/pull/19059) [#19205](https://github.com/apache/druid/pull/19205) [#19016](https://github.com/apache/druid/pull/19016)
+
+### Thrift input format
+
+As part of the Thrift contributor extension, Druid now supports Thrift-encoded data for Kafka and Kinesis streaming ingestion.
+
+[#19111](https://github.com/apache/druid/pull/19111)
+
+To use this feature, you must add `druid-thrift-extensions` to your extension load list.
+
+### Kubernetes-based task management
+
+This extension is now generally available.
+
+[#19128](https://github.com/apache/druid/pull/19128)
+
+### Tombstones
+
+Tombstones for JSON-based native batch ingestion (the `dropExisting` flag for `ioConfig`) are now generally available.
+
+[#19128](https://github.com/apache/druid/pull/19128)
+
+### Dynamic default query context
+
+You can now add default query context parameters as a dynamic configuration to the Broker. This allows you to override static defaults set in your runtime properties without restarting your deployment or having to update multiple queries individually. Druid resolves which query context parameters to use based on the following priority:
+
+1. The query context included with the query
+1. The query context set as a dynamic configuration on the Broker
+1. The query context parameters set in the runtime properties
+1. The defaults that ship with Druid
+
+Note that like other Broker dynamic configuration, this is best-effort. Settings may not be applied in certain
+cases, such as when a Broker has recently started and hasn't received the config yet, or if the
+Broker can't contact the Coordinator. If a query context parameter is critical for all your queries, set it in the runtime properties.
+
+[#19146](https://github.com/apache/druid/pull/19146)
+
+### `sys.queries` table (experimental)
+
+The new system queries table provides information about currently running and recently completed queries that use the Dart engine. This table is off by default. To enable the table, set the following config:
+
+```
+druid.sql.planner.enableSysQueriesTable = true
+```
+
+As part of this change, the `/druid/v2/sql/queries` API now supports an `includeComplete` parameter that shows recently completed queries.
+
+[#18923](https://github.com/apache/druid/pull/18923)
+
+### Auto-compaction with compaction supervisors
+
+Auto-compaction using compaction supervisors has been improved and is now generally available.
+
+As part of the improvement, instead of individual compacted segments having to store their full compaction state in the metadata store, states are now stored in a central location, a new `indexingStates` table, in the metadata store. Individual segments only need to store a unique reference (`indexing_state_fingerprint`) to their full compaction state.
+
+Since many segments in a single datasource share the same underlying compaction state, this greatly reduces metadata storage requirements for automatic compaction.
+
+For backwards compatibility, detailed compaction state will continue to be persisted in each segment for now. This will be removed in a future release.
+
+You can stop storing detailed compaction state by setting `storeCompactionStatePerSegment` to `false` in the cluster compaction config. If you turn it off and need to downgrade, Druid needs to re-compact any segments that have been compacted since you changed the config.
+
+This change has upgrade impacts for metadata storage and metadata caching. For more information, see the [Metadata storage for auto-compaction with compaction supervisors](#metadata-storage-for-auto-compaction-with-compaction-supervisors) upgrade note.
+
+[#19113](https://github.com/apache/druid/pull/19113) [#18844](https://github.com/apache/druid/pull/18844)
+
+### Broker tier selection for realtime servers
+
+Added `druid.broker.realtime.select.tier` and `druid.broker.realtime.balancer.type` on the Brokers to optionally override the Broker’s tier selection and balancer strategies for realtime servers. If these properties are not set (the default), realtime servers continue to use the existing `druid.broker.select` and `druid.broker.balancer` configurations that apply to both historical and realtime servers.
+
+[#19062](https://github.com/apache/druid/pull/19062)
+
+### Manual Broker routing in the web console
+
+You can now configure which Broker the Router uses for queries issued from the web console. You may want to do this if there are Brokers that don't have visibility into certain data tiers, and you know you're querying data available only on a certain tier.
+
+To specify a Broker, add the following config to `web-console/console-config.js`:
+
+```js
+consoleBrokerService: 'druid/BROKER_NAME'
+```
+
+[#19069](https://github.com/apache/druid/pull/19069)
+
+### Consul extension
+
+The contributor extension `druid-consul-extensions` lets Druid clusters use Consul for service discovery and
+Coordinator/Overlord leader election instead of ZooKeeper. The extension supports ACLs, TLS/mTLS, and metrics.
+
+Before you switch to Consul, you need to set
+`druid.serverview.type=http` and `druid.indexer.runner.type=httpRemote` cluster wide.
+
+[#18843](https://github.com/apache/druid/pull/18843)
+
## Functional area and related changes
This section contains detailed release notes separated by areas.
### Web console
+#### Changed storage column displays
+
+- Improved the compaction config view to
+- Renamed **Current size** to **Assigned size**.
+- Renamed **Max size** to **Effective size**. It now displays the smaller value between `max_size` and `storage_size`. The max size is still shown as a tooltip.
+- Changed usage calculation to use `effective_size`
+
+[#19007](https://github.com/apache/druid/pull/19007)
+
#### Other web console improvements
+- Added `workerDesc` to `WorkerStats`, which makes it easier to identify where a worker is running [#19171](https://github.com/apache/druid/pull/19171)
+- Added the Dart unique execution ID (`dartQueryId`) and the `sqlQueryId` to the **Details** pane in the web console [#19185](https://github.com/apache/druid/pull/19185)
+- Added support for showing completed Dart queries in the web console [#18940](https://github.com/apache/druid/pull/18940)
+- Added a detail dialog to the **Services** page [#18960](https://github.com/apache/druid/pull/18960)
+- Added icons to indicate when data is loaded into virtual storage, including a tooltip that shows all the counters for the data [#19010](https://github.com/apache/druid/pull/19010)
+- Added support for Dart reports [#18897](https://github.com/apache/druid/pull/18897)
+- Changed what's considered an active worker: any nonzero rows, files, bytes, frames, or wall time is enough to consider a worker active [#19183](https://github.com/apache/druid/pull/19183)
+- Changed the **Cancel query** option to show only if a query is in an accepted or running state [#19182](https://github.com/apache/druid/pull/19182)
+- Changed the ordering of the current Dart queries panel to show queries in the following order: RUNNING, ACCEPTED, and then COMPLETED. RUNNING and ACCEPTED queries are ordered by the most recent first (based on timestamp). COMPLETED queries are sorted by finish time [#19237](https://github.com/apache/druid/pull/19237)
+
### Ingestion
#### SQL-based ingestion
##### Other SQL-based ingestion improvements
+- Added support for virtual storage fabric when performing SQL-based ingestion [#18873](https://github.com/apache/druid/pull/18873)
+- Added support for `StorageMonitor` so that MSQ task engine tasks always emit `taskId` and `groupId` [#19048](https://github.com/apache/druid/pull/19048)
+- Improved worker cancellation [#18931](https://github.com/apache/druid/pull/18931)
+- Improved exception handling [#19234](https://github.com/apache/druid/pull/19234)
+
#### Streaming ingestion
+##### Changed how tasks get launched for autoscaling
+
+The behavior of `taskCountMin` and `taskCountStart` has been changed for autoscaling. Druid now computes the initial number of tasks to launch by checking the configs in the following order: `taskCountStart` (optional), then `taskCount` (in `ioConfig`), then `taskCountMin`.
+
+[#19091](https://github.com/apache/druid/pull/19091)
+
+##### Improved query isolation
+
+Added `serverPriorityToReplicas` parameter to the streaming supervisor specs (Kafka, Kinesis, and Rabbit). This allows operators to distribute task replicas across different server priorities for realtime indexing tasks. Similar to Historical tiering, this enables query isolation for mixed workload scenarios on the Peons, allowing some task replicas to handle queries of specific priorities.
+
+[#19040](https://github.com/apache/druid/pull/19040)
+
##### Other streaming ingestion improvements
+- Improved cost-based autoscaler performance in high lag scenarios [#19045](https://github.com/apache/druid/pull/19045)
+- Improved the performance of realtime task scheduling by ordering schedule requests by priority on the `TaskQueue` [#19203](https://github.com/apache/druid/pull/19203)
+
+
+#### Ingestion improvements
+
+- Sped up task scheduling on the Overlord [#19199](https://github.com/apache/druid/pull/19199)
+
### Querying
+#### groupBy query configuration
+
+Added a new groupBy query configuration property `druid.query.groupBy.maxSpillFileCount` to limit the maximum number of spill files created per query. When the limit is exceeded, the query fails with a clear error message instead of causing Historical nodes to run out of memory during spill file merging. The limit can also be overridden per query using the query context parameter `maxSpillFileCount`.
+
+[#19141](https://github.com/apache/druid/pull/19141)
+
+#### Improved handling of nested aggregates
+
+Druid can now merge two Aggregates with a Projection between them. For example, the following query:
+
+```sql
+SELECT
+ hr,
+ UPPER(t1.x) x,
+ SUM(t1.cnt) cnt,
+ MIN(t1.mn) mn,
+ MAX(t1.mx) mx
+FROM (
+ SELECT
+ floor(__time to hour) hr,
+ dim2 x,
+ COUNT(*) cnt,
+ MIN(m1 * 5) mn,
+ MAX(m1 + m2) mx
+ FROM druid.foo
+ WHERE dim2 IN ('abc', 'def', 'a', 'b', '')
+ GROUP BY 1, 2
+) t1
+WHERE t1.x IN ('abc', 'foo', 'bar', 'a', '')
+GROUP BY 1, 2
+```
+
+can be simplified to the following:
+
+```sql
+SELECT
+ FLOOR(__time TO hour) hr,
+ UPPER(dim2) x,
+ COUNT(*) cnt,
+ MIN(m1 * 5) mn,
+ MAX(m1 + m2) mx
+FROM druid.foo
+WHERE dim2 IN ('abc', 'a', '')
+GROUP BY 1, 2
+```
+
+[#18498](https://github.com/apache/druid/pull/18498)
+
#### Other querying improvements
+- Added `durationMs` to Dart query reports [#19169](https://github.com/apache/druid/pull/19169)
+- Improved error handling so that row signature column order is preserved when column analysis encounters an error [#19162](https://github.com/apache/druid/pull/19162)
+- Improved GROUP BY performance [#18952](https://github.com/apache/druid/pull/18952)
+- Improved expression filters to take advantage of specialized virtual columns when possible, resulting in better performance for the query [#18965](https://github.com/apache/druid/pull/18965)
+
### Cluster management
+#### New Broker tier selection strategies
+
+Operators can now configure two new Broker `TierSelectorStrategy` implementations:
+
+- `strict` - Only selects servers whose priorities match the configured list. Example configuration: `druid.broker.select.tier=strict` and `druid.broker.select.tier.strict.priorities=[1]`.
+- `pooled` - Pools servers across the configured priorities and selects among them, allowing queries to utilize multiple priority tiers for improved availability. Example configuration: `druid.broker.select.tier=pooled` and `druid.broker.select.tier.pooled.priorities=[2,1]`.
+
+These strategies can also be configured for realtime servers using `druid.broker.realtime.select.tier`.
+
+[#19094](https://github.com/apache/druid/pull/19094)
+
+#### Druid operator
+
+The Druid operator now resides in its own repository: [`apache/druid-operator`](https://github.com/apache/druid-operator).
+
+[#19156](https://github.com/apache/druid/pull/19156)
+
+#### Cost-based autoscaler algorithm
+
+The algorithm for cost-based autoscaling has been changed:
+
+- Scale up more aggressively when per-partition lag is meaningful
+- Relax the partitions-per-task increase limit based on lag severity and headroom
+- Keep behavior conservative near `taskCountMax` and avoid negative headroom effects
+
+[#18936](https://github.com/apache/druid/pull/18936)
+
#### Other cluster management improvements
+- Added a `ReadOnly` authorizer that allows all READ operations but denies any other operation, such as WRITE [#19243](https://github.com/apache/druid/pull/19243)
+- Added `/status/ready` endpoint for service health so that external load balancers can handle a graceful shutdown better [#19148](https://github.com/apache/druid/pull/19148)
+- Added a configurable option to scale-down during task run time for cost-based autoscaler [#18958](https://github.com/apache/druid/pull/18958)
+- Added `storage_size` to `sys.servers` to facilitate retrieving disk cache size for Historicals when using the virtual storage fabric [#18979](https://github.com/apache/druid/pull/18979)
+- Added a log for new task count computation for the cost-based auto scaler [#18929](https://github.com/apache/druid/pull/18929)
+- Changed how the scaling is calculated from a square root-based scaling formula to a logarithmic formula that provides better emergency recovery at low task counts and millions of lag [#18976](https://github.com/apache/druid/pull/18976)
+- Improved the load speed of cached segments during Historical startup [#18489](https://github.com/apache/druid/pull/18489)
+- Improved Broker startup time by parallelizing buffer initialization [#19025](https://github.com/apache/druid/pull/19025)
+- Improved the stack trace for MSQ task engine worker failures so that they're preserved [#19049](https://github.com/apache/druid/pull/19049)
+- Improved the performance of the cost-based autoscaler during loaded lag conditions [#18991](https://github.com/apache/druid/pull/18991)
+
### Data management
+#### Durable storage cleaner
+
+The durable storage cleaner now supports configurable time-based retention for MSQ query results. Previously, query results were retained for all known tasks, which was unreliable for completed tasks. With this change, query results are retained for a configurable time period based on the task creation time.
+
+The new configuration property `druid.msq.intermediate.storage.cleaner.durationToRetain` controls the retention period for query results. The default retention period is 6 hours.
+
+[#19074](https://github.com/apache/druid/pull/19074)
+
#### Other data management improvements
+- Added the `druid.storage.transfer.asyncHttpClientType` config that specifies which async HTTP client to use for S3 transfers: `crt` for Amazon CRT or `netty` for Netty NIO [#19249](https://github.com/apache/druid/pull/19249)
+- Added a mechanism to automatically clean up intermediary files on HDFS storage [#19187](https://github.com/apache/druid/pull/19187)
+- Changed the default value for `druid.indexing.formats.maxStringLength` to null from 0 [#19198](https://github.com/apache/druid/pull/19198)
+
### Metrics and monitoring
+#### Version field
+
+All Druid metrics now include `version` and `buildRevision` fields to help you identify which version of Druid is emitting a metric. You can use this information to verify that all nodes in a cluster are running the intended revision during rolling deployments.
+
+[#19123](https://github.com/apache/druid/pull/19123)
+
+#### Monitoring supervisor state
+
+Added a new `supervisor/count` metric when `SupervisorStatsMonitor` is enabled in `druid.monitoring.monitors`. The metric reports each supervisor’s state, such as `RUNNING` or `SUSPENDED`, for Prometheus, StatsD, and other metric systems.
+
+[#19114](https://github.com/apache/druid/pull/19114)
+
+#### Improved `groupBy` metrics
+
+`GroupByStatsMonitor` now provides the following metrics:
+
+- `mergeBuffer/bytesUsed`
+- `mergeBuffer/maxAcquisitionTimeNs`
+- `groupBy/maxSpilledBytes`
+- `groupBy/maxMergeDictionarySize`
+
+[#18731](https://github.com/apache/druid/pull/18731) [#18970](https://github.com/apache/druid/pull/18970) [#18934](https://github.com/apache/druid/pull/18934)
+
+#### Filtering metrics
+
+Operators can limit which metrics the logging emitter writes by setting `druid.emitter.logging.shouldFilterMetrics=true` and, if desired, `druid.emitter.logging.allowedMetricsPath` to a JSON object file where the keys are metric names. A missing custom file results in a warning and use of the bundled `defaultMetrics.json`. Alerts and other non-metric events are always logged.
+
+[#19030](https://github.com/apache/druid/pull/19030)
+
+#### New Broker metrics
+
+Added `segment/schemaCache/rowSignature/changed` and `segment/schemaCache/rowSignature/column/count` metrics to get visibility into when the Broker's segment metadata cache's row signature for each datasource is initialized and updated.
Review Comment:
```suggestion
Added `segment/schemaCache/rowSignature/changed` and `segment/schemaCache/rowSignature/column/count` metrics to expose events when the Broker initializes and updates the row signature in the the segment metadata cache for each datasource.
```
##########
docs/release-info/release-notes.md:
##########
@@ -57,50 +57,551 @@ For tips about how to write a good release note, see [Release notes](https://git
This section contains important information about new and existing features.
+### Hadoop-based ingestion
+
+Support for Hadoop-based ingestion has been removed. The feature was deprecated in Druid 34.
+
+Use one of Druid's other supported ingestion methods, such as SQL-based ingestion or MiddleManager-less ingestion using Kubernetes.
+
+[#19109](https://github.com/apache/druid/pull/19109)
+
+### Query blocklist
+
+You can now create a query blocklist for dynamically blocking queries without restarting your deployment. You can block queries by datasource, query type, or query context using the new `/druid/coordinator/v1/config/broker` API. Block rules use `AND` logic, which means all criteria must match. Druid stores the rules in the metadata database.
+
+The following example blocks all groupBy queries on the `wikipedia` datasource with a query context parameter of `priority` equal to `0`:
+
+```
+POST /druid/coordinator/v1/config/broker
+ {
+ "queryBlocklist": [
+ {
+ "ruleName": "block-wikipedia-groupbys",
+ "dataSources": ["wikipedia"],
+ "queryTypes": ["groupBy"],
+ "contextMatches": {"priority": "0"}
+ }
+ ]
+ }
+```
+
+[#19011](https://github.com/apache/druid/pull/19011)
+
+### Minor compaction for Overlord-based compaction (experimental)
+
+You can now configure minor compaction. With minor compaction, Druid only compacts newly ingested segments while upgrading existing compacted segments. When Druid upgrades segments, it updates the metadata instead of using resources to compact it again. You can use the native compaction engine or the MSQ task engine.
+
+Use the `mostFragmentedFirst` compaction policy and set either a percentage of rows-based or byte-based threshold for minor compaction.
+
+[#19059](https://github.com/apache/druid/pull/19059) [#19205](https://github.com/apache/druid/pull/19205) [#19016](https://github.com/apache/druid/pull/19016)
+
+### Thrift input format
+
+As part of the Thrift contributor extension, Druid now supports Thrift-encoded data for Kafka and Kinesis streaming ingestion.
+
+[#19111](https://github.com/apache/druid/pull/19111)
+
+To use this feature, you must add `druid-thrift-extensions` to your extension load list.
+
+### Kubernetes-based task management
+
+This extension is now generally available.
+
+[#19128](https://github.com/apache/druid/pull/19128)
+
+### Tombstones
+
+Tombstones for JSON-based native batch ingestion (the `dropExisting` flag for `ioConfig`) are now generally available.
+
+[#19128](https://github.com/apache/druid/pull/19128)
+
+### Dynamic default query context
+
+You can now add default query context parameters as a dynamic configuration to the Broker. This allows you to override static defaults set in your runtime properties without restarting your deployment or having to update multiple queries individually. Druid resolves which query context parameters to use based on the following priority:
+
+1. The query context included with the query
+1. The query context set as a dynamic configuration on the Broker
+1. The query context parameters set in the runtime properties
+1. The defaults that ship with Druid
+
+Note that like other Broker dynamic configuration, this is best-effort. Settings may not be applied in certain
+cases, such as when a Broker has recently started and hasn't received the config yet, or if the
+Broker can't contact the Coordinator. If a query context parameter is critical for all your queries, set it in the runtime properties.
+
+[#19146](https://github.com/apache/druid/pull/19146)
+
+### `sys.queries` table (experimental)
+
+The new system queries table provides information about currently running and recently completed queries that use the Dart engine. This table is off by default. To enable the table, set the following config:
+
+```
+druid.sql.planner.enableSysQueriesTable = true
+```
+
+As part of this change, the `/druid/v2/sql/queries` API now supports an `includeComplete` parameter that shows recently completed queries.
+
+[#18923](https://github.com/apache/druid/pull/18923)
+
+### Auto-compaction with compaction supervisors
+
+Auto-compaction using compaction supervisors has been improved and is now generally available.
+
+As part of the improvement, instead of individual compacted segments having to store their full compaction state in the metadata store, states are now stored in a central location, a new `indexingStates` table, in the metadata store. Individual segments only need to store a unique reference (`indexing_state_fingerprint`) to their full compaction state.
+
+Since many segments in a single datasource share the same underlying compaction state, this greatly reduces metadata storage requirements for automatic compaction.
+
+For backwards compatibility, detailed compaction state will continue to be persisted in each segment for now. This will be removed in a future release.
+
+You can stop storing detailed compaction state by setting `storeCompactionStatePerSegment` to `false` in the cluster compaction config. If you turn it off and need to downgrade, Druid needs to re-compact any segments that have been compacted since you changed the config.
+
+This change has upgrade impacts for metadata storage and metadata caching. For more information, see the [Metadata storage for auto-compaction with compaction supervisors](#metadata-storage-for-auto-compaction-with-compaction-supervisors) upgrade note.
+
+[#19113](https://github.com/apache/druid/pull/19113) [#18844](https://github.com/apache/druid/pull/18844)
+
+### Broker tier selection for realtime servers
+
+Added `druid.broker.realtime.select.tier` and `druid.broker.realtime.balancer.type` on the Brokers to optionally override the Broker’s tier selection and balancer strategies for realtime servers. If these properties are not set (the default), realtime servers continue to use the existing `druid.broker.select` and `druid.broker.balancer` configurations that apply to both historical and realtime servers.
+
+[#19062](https://github.com/apache/druid/pull/19062)
+
+### Manual Broker routing in the web console
+
+You can now configure which Broker the Router uses for queries issued from the web console. You may want to do this if there are Brokers that don't have visibility into certain data tiers, and you know you're querying data available only on a certain tier.
+
+To specify a Broker, add the following config to `web-console/console-config.js`:
+
+```js
+consoleBrokerService: 'druid/BROKER_NAME'
+```
+
+[#19069](https://github.com/apache/druid/pull/19069)
+
+### Consul extension
+
+The contributor extension `druid-consul-extensions` lets Druid clusters use Consul for service discovery and
+Coordinator/Overlord leader election instead of ZooKeeper. The extension supports ACLs, TLS/mTLS, and metrics.
+
+Before you switch to Consul, you need to set
+`druid.serverview.type=http` and `druid.indexer.runner.type=httpRemote` cluster wide.
+
+[#18843](https://github.com/apache/druid/pull/18843)
+
## Functional area and related changes
This section contains detailed release notes separated by areas.
### Web console
+#### Changed storage column displays
+
+- Improved the compaction config view to
+- Renamed **Current size** to **Assigned size**.
+- Renamed **Max size** to **Effective size**. It now displays the smaller value between `max_size` and `storage_size`. The max size is still shown as a tooltip.
+- Changed usage calculation to use `effective_size`
+
+[#19007](https://github.com/apache/druid/pull/19007)
+
#### Other web console improvements
+- Added `workerDesc` to `WorkerStats`, which makes it easier to identify where a worker is running [#19171](https://github.com/apache/druid/pull/19171)
+- Added the Dart unique execution ID (`dartQueryId`) and the `sqlQueryId` to the **Details** pane in the web console [#19185](https://github.com/apache/druid/pull/19185)
+- Added support for showing completed Dart queries in the web console [#18940](https://github.com/apache/druid/pull/18940)
+- Added a detail dialog to the **Services** page [#18960](https://github.com/apache/druid/pull/18960)
+- Added icons to indicate when data is loaded into virtual storage, including a tooltip that shows all the counters for the data [#19010](https://github.com/apache/druid/pull/19010)
+- Added support for Dart reports [#18897](https://github.com/apache/druid/pull/18897)
+- Changed what's considered an active worker: any nonzero rows, files, bytes, frames, or wall time is enough to consider a worker active [#19183](https://github.com/apache/druid/pull/19183)
+- Changed the **Cancel query** option to show only if a query is in an accepted or running state [#19182](https://github.com/apache/druid/pull/19182)
+- Changed the ordering of the current Dart queries panel to show queries in the following order: RUNNING, ACCEPTED, and then COMPLETED. RUNNING and ACCEPTED queries are ordered by the most recent first (based on timestamp). COMPLETED queries are sorted by finish time [#19237](https://github.com/apache/druid/pull/19237)
+
### Ingestion
#### SQL-based ingestion
##### Other SQL-based ingestion improvements
+- Added support for virtual storage fabric when performing SQL-based ingestion [#18873](https://github.com/apache/druid/pull/18873)
+- Added support for `StorageMonitor` so that MSQ task engine tasks always emit `taskId` and `groupId` [#19048](https://github.com/apache/druid/pull/19048)
+- Improved worker cancellation [#18931](https://github.com/apache/druid/pull/18931)
+- Improved exception handling [#19234](https://github.com/apache/druid/pull/19234)
+
#### Streaming ingestion
+##### Changed how tasks get launched for autoscaling
+
+The behavior of `taskCountMin` and `taskCountStart` has been changed for autoscaling. Druid now computes the initial number of tasks to launch by checking the configs in the following order: `taskCountStart` (optional), then `taskCount` (in `ioConfig`), then `taskCountMin`.
+
+[#19091](https://github.com/apache/druid/pull/19091)
+
+##### Improved query isolation
+
+Added `serverPriorityToReplicas` parameter to the streaming supervisor specs (Kafka, Kinesis, and Rabbit). This allows operators to distribute task replicas across different server priorities for realtime indexing tasks. Similar to Historical tiering, this enables query isolation for mixed workload scenarios on the Peons, allowing some task replicas to handle queries of specific priorities.
+
+[#19040](https://github.com/apache/druid/pull/19040)
+
##### Other streaming ingestion improvements
+- Improved cost-based autoscaler performance in high lag scenarios [#19045](https://github.com/apache/druid/pull/19045)
+- Improved the performance of realtime task scheduling by ordering schedule requests by priority on the `TaskQueue` [#19203](https://github.com/apache/druid/pull/19203)
+
+
+#### Ingestion improvements
+
+- Sped up task scheduling on the Overlord [#19199](https://github.com/apache/druid/pull/19199)
+
### Querying
+#### groupBy query configuration
+
+Added a new groupBy query configuration property `druid.query.groupBy.maxSpillFileCount` to limit the maximum number of spill files created per query. When the limit is exceeded, the query fails with a clear error message instead of causing Historical nodes to run out of memory during spill file merging. The limit can also be overridden per query using the query context parameter `maxSpillFileCount`.
+
+[#19141](https://github.com/apache/druid/pull/19141)
+
+#### Improved handling of nested aggregates
+
+Druid can now merge two Aggregates with a Projection between them. For example, the following query:
Review Comment:
why are aggregates and projections capilalized?
##########
docs/release-info/release-notes.md:
##########
@@ -57,50 +57,551 @@ For tips about how to write a good release note, see [Release notes](https://git
This section contains important information about new and existing features.
+### Hadoop-based ingestion
+
+Support for Hadoop-based ingestion has been removed. The feature was deprecated in Druid 34.
+
+Use one of Druid's other supported ingestion methods, such as SQL-based ingestion or MiddleManager-less ingestion using Kubernetes.
+
+[#19109](https://github.com/apache/druid/pull/19109)
+
+### Query blocklist
+
+You can now create a query blocklist for dynamically blocking queries without restarting your deployment. You can block queries by datasource, query type, or query context using the new `/druid/coordinator/v1/config/broker` API. Block rules use `AND` logic, which means all criteria must match. Druid stores the rules in the metadata database.
+
+The following example blocks all groupBy queries on the `wikipedia` datasource with a query context parameter of `priority` equal to `0`:
+
+```
+POST /druid/coordinator/v1/config/broker
+ {
+ "queryBlocklist": [
+ {
+ "ruleName": "block-wikipedia-groupbys",
+ "dataSources": ["wikipedia"],
+ "queryTypes": ["groupBy"],
+ "contextMatches": {"priority": "0"}
+ }
+ ]
+ }
+```
+
+[#19011](https://github.com/apache/druid/pull/19011)
+
+### Minor compaction for Overlord-based compaction (experimental)
+
+You can now configure minor compaction. With minor compaction, Druid only compacts newly ingested segments while upgrading existing compacted segments. When Druid upgrades segments, it updates the metadata instead of using resources to compact it again. You can use the native compaction engine or the MSQ task engine.
+
+Use the `mostFragmentedFirst` compaction policy and set either a percentage of rows-based or byte-based threshold for minor compaction.
+
+[#19059](https://github.com/apache/druid/pull/19059) [#19205](https://github.com/apache/druid/pull/19205) [#19016](https://github.com/apache/druid/pull/19016)
+
+### Thrift input format
+
+As part of the Thrift contributor extension, Druid now supports Thrift-encoded data for Kafka and Kinesis streaming ingestion.
+
+[#19111](https://github.com/apache/druid/pull/19111)
+
+To use this feature, you must add `druid-thrift-extensions` to your extension load list.
+
+### Kubernetes-based task management
+
+This extension is now generally available.
+
+[#19128](https://github.com/apache/druid/pull/19128)
+
+### Tombstones
+
+Tombstones for JSON-based native batch ingestion (the `dropExisting` flag for `ioConfig`) are now generally available.
+
+[#19128](https://github.com/apache/druid/pull/19128)
+
+### Dynamic default query context
+
+You can now add default query context parameters as a dynamic configuration to the Broker. This allows you to override static defaults set in your runtime properties without restarting your deployment or having to update multiple queries individually. Druid resolves which query context parameters to use based on the following priority:
+
+1. The query context included with the query
+1. The query context set as a dynamic configuration on the Broker
+1. The query context parameters set in the runtime properties
+1. The defaults that ship with Druid
+
+Note that like other Broker dynamic configuration, this is best-effort. Settings may not be applied in certain
+cases, such as when a Broker has recently started and hasn't received the config yet, or if the
+Broker can't contact the Coordinator. If a query context parameter is critical for all your queries, set it in the runtime properties.
+
+[#19146](https://github.com/apache/druid/pull/19146)
+
+### `sys.queries` table (experimental)
+
+The new system queries table provides information about currently running and recently completed queries that use the Dart engine. This table is off by default. To enable the table, set the following config:
+
+```
+druid.sql.planner.enableSysQueriesTable = true
+```
+
+As part of this change, the `/druid/v2/sql/queries` API now supports an `includeComplete` parameter that shows recently completed queries.
+
+[#18923](https://github.com/apache/druid/pull/18923)
+
+### Auto-compaction with compaction supervisors
+
+Auto-compaction using compaction supervisors has been improved and is now generally available.
+
+As part of the improvement, instead of individual compacted segments having to store their full compaction state in the metadata store, states are now stored in a central location, a new `indexingStates` table, in the metadata store. Individual segments only need to store a unique reference (`indexing_state_fingerprint`) to their full compaction state.
+
+Since many segments in a single datasource share the same underlying compaction state, this greatly reduces metadata storage requirements for automatic compaction.
+
+For backwards compatibility, detailed compaction state will continue to be persisted in each segment for now. This will be removed in a future release.
+
+You can stop storing detailed compaction state by setting `storeCompactionStatePerSegment` to `false` in the cluster compaction config. If you turn it off and need to downgrade, Druid needs to re-compact any segments that have been compacted since you changed the config.
+
+This change has upgrade impacts for metadata storage and metadata caching. For more information, see the [Metadata storage for auto-compaction with compaction supervisors](#metadata-storage-for-auto-compaction-with-compaction-supervisors) upgrade note.
+
+[#19113](https://github.com/apache/druid/pull/19113) [#18844](https://github.com/apache/druid/pull/18844)
+
+### Broker tier selection for realtime servers
+
+Added `druid.broker.realtime.select.tier` and `druid.broker.realtime.balancer.type` on the Brokers to optionally override the Broker’s tier selection and balancer strategies for realtime servers. If these properties are not set (the default), realtime servers continue to use the existing `druid.broker.select` and `druid.broker.balancer` configurations that apply to both historical and realtime servers.
+
+[#19062](https://github.com/apache/druid/pull/19062)
+
+### Manual Broker routing in the web console
+
+You can now configure which Broker the Router uses for queries issued from the web console. You may want to do this if there are Brokers that don't have visibility into certain data tiers, and you know you're querying data available only on a certain tier.
+
+To specify a Broker, add the following config to `web-console/console-config.js`:
+
+```js
+consoleBrokerService: 'druid/BROKER_NAME'
+```
+
+[#19069](https://github.com/apache/druid/pull/19069)
+
+### Consul extension
+
+The contributor extension `druid-consul-extensions` lets Druid clusters use Consul for service discovery and
+Coordinator/Overlord leader election instead of ZooKeeper. The extension supports ACLs, TLS/mTLS, and metrics.
+
+Before you switch to Consul, you need to set
+`druid.serverview.type=http` and `druid.indexer.runner.type=httpRemote` cluster wide.
+
+[#18843](https://github.com/apache/druid/pull/18843)
+
## Functional area and related changes
This section contains detailed release notes separated by areas.
### Web console
+#### Changed storage column displays
+
+- Improved the compaction config view to
+- Renamed **Current size** to **Assigned size**.
+- Renamed **Max size** to **Effective size**. It now displays the smaller value between `max_size` and `storage_size`. The max size is still shown as a tooltip.
+- Changed usage calculation to use `effective_size`
+
+[#19007](https://github.com/apache/druid/pull/19007)
+
#### Other web console improvements
+- Added `workerDesc` to `WorkerStats`, which makes it easier to identify where a worker is running [#19171](https://github.com/apache/druid/pull/19171)
+- Added the Dart unique execution ID (`dartQueryId`) and the `sqlQueryId` to the **Details** pane in the web console [#19185](https://github.com/apache/druid/pull/19185)
+- Added support for showing completed Dart queries in the web console [#18940](https://github.com/apache/druid/pull/18940)
+- Added a detail dialog to the **Services** page [#18960](https://github.com/apache/druid/pull/18960)
+- Added icons to indicate when data is loaded into virtual storage, including a tooltip that shows all the counters for the data [#19010](https://github.com/apache/druid/pull/19010)
+- Added support for Dart reports [#18897](https://github.com/apache/druid/pull/18897)
+- Changed what's considered an active worker: any nonzero rows, files, bytes, frames, or wall time is enough to consider a worker active [#19183](https://github.com/apache/druid/pull/19183)
+- Changed the **Cancel query** option to show only if a query is in an accepted or running state [#19182](https://github.com/apache/druid/pull/19182)
+- Changed the ordering of the current Dart queries panel to show queries in the following order: RUNNING, ACCEPTED, and then COMPLETED. RUNNING and ACCEPTED queries are ordered by the most recent first (based on timestamp). COMPLETED queries are sorted by finish time [#19237](https://github.com/apache/druid/pull/19237)
+
### Ingestion
#### SQL-based ingestion
##### Other SQL-based ingestion improvements
+- Added support for virtual storage fabric when performing SQL-based ingestion [#18873](https://github.com/apache/druid/pull/18873)
+- Added support for `StorageMonitor` so that MSQ task engine tasks always emit `taskId` and `groupId` [#19048](https://github.com/apache/druid/pull/19048)
+- Improved worker cancellation [#18931](https://github.com/apache/druid/pull/18931)
+- Improved exception handling [#19234](https://github.com/apache/druid/pull/19234)
+
#### Streaming ingestion
+##### Changed how tasks get launched for autoscaling
+
+The behavior of `taskCountMin` and `taskCountStart` has been changed for autoscaling. Druid now computes the initial number of tasks to launch by checking the configs in the following order: `taskCountStart` (optional), then `taskCount` (in `ioConfig`), then `taskCountMin`.
+
+[#19091](https://github.com/apache/druid/pull/19091)
+
+##### Improved query isolation
+
+Added `serverPriorityToReplicas` parameter to the streaming supervisor specs (Kafka, Kinesis, and Rabbit). This allows operators to distribute task replicas across different server priorities for realtime indexing tasks. Similar to Historical tiering, this enables query isolation for mixed workload scenarios on the Peons, allowing some task replicas to handle queries of specific priorities.
+
+[#19040](https://github.com/apache/druid/pull/19040)
+
##### Other streaming ingestion improvements
+- Improved cost-based autoscaler performance in high lag scenarios [#19045](https://github.com/apache/druid/pull/19045)
+- Improved the performance of realtime task scheduling by ordering schedule requests by priority on the `TaskQueue` [#19203](https://github.com/apache/druid/pull/19203)
+
+
+#### Ingestion improvements
+
+- Sped up task scheduling on the Overlord [#19199](https://github.com/apache/druid/pull/19199)
+
### Querying
+#### groupBy query configuration
+
+Added a new groupBy query configuration property `druid.query.groupBy.maxSpillFileCount` to limit the maximum number of spill files created per query. When the limit is exceeded, the query fails with a clear error message instead of causing Historical nodes to run out of memory during spill file merging. The limit can also be overridden per query using the query context parameter `maxSpillFileCount`.
+
+[#19141](https://github.com/apache/druid/pull/19141)
+
+#### Improved handling of nested aggregates
+
+Druid can now merge two Aggregates with a Projection between them. For example, the following query:
+
+```sql
+SELECT
+ hr,
+ UPPER(t1.x) x,
+ SUM(t1.cnt) cnt,
+ MIN(t1.mn) mn,
+ MAX(t1.mx) mx
+FROM (
+ SELECT
+ floor(__time to hour) hr,
+ dim2 x,
+ COUNT(*) cnt,
+ MIN(m1 * 5) mn,
+ MAX(m1 + m2) mx
+ FROM druid.foo
+ WHERE dim2 IN ('abc', 'def', 'a', 'b', '')
+ GROUP BY 1, 2
+) t1
+WHERE t1.x IN ('abc', 'foo', 'bar', 'a', '')
+GROUP BY 1, 2
+```
+
+can be simplified to the following:
+
+```sql
+SELECT
+ FLOOR(__time TO hour) hr,
+ UPPER(dim2) x,
+ COUNT(*) cnt,
+ MIN(m1 * 5) mn,
+ MAX(m1 + m2) mx
+FROM druid.foo
+WHERE dim2 IN ('abc', 'a', '')
+GROUP BY 1, 2
+```
+
+[#18498](https://github.com/apache/druid/pull/18498)
+
#### Other querying improvements
+- Added `durationMs` to Dart query reports [#19169](https://github.com/apache/druid/pull/19169)
+- Improved error handling so that row signature column order is preserved when column analysis encounters an error [#19162](https://github.com/apache/druid/pull/19162)
+- Improved GROUP BY performance [#18952](https://github.com/apache/druid/pull/18952)
+- Improved expression filters to take advantage of specialized virtual columns when possible, resulting in better performance for the query [#18965](https://github.com/apache/druid/pull/18965)
+
### Cluster management
+#### New Broker tier selection strategies
+
+Operators can now configure two new Broker `TierSelectorStrategy` implementations:
+
+- `strict` - Only selects servers whose priorities match the configured list. Example configuration: `druid.broker.select.tier=strict` and `druid.broker.select.tier.strict.priorities=[1]`.
+- `pooled` - Pools servers across the configured priorities and selects among them, allowing queries to utilize multiple priority tiers for improved availability. Example configuration: `druid.broker.select.tier=pooled` and `druid.broker.select.tier.pooled.priorities=[2,1]`.
+
+These strategies can also be configured for realtime servers using `druid.broker.realtime.select.tier`.
+
+[#19094](https://github.com/apache/druid/pull/19094)
+
+#### Druid operator
+
+The Druid operator now resides in its own repository: [`apache/druid-operator`](https://github.com/apache/druid-operator).
+
+[#19156](https://github.com/apache/druid/pull/19156)
+
+#### Cost-based autoscaler algorithm
+
+The algorithm for cost-based autoscaling has been changed:
+
+- Scale up more aggressively when per-partition lag is meaningful
+- Relax the partitions-per-task increase limit based on lag severity and headroom
+- Keep behavior conservative near `taskCountMax` and avoid negative headroom effects
+
+[#18936](https://github.com/apache/druid/pull/18936)
+
#### Other cluster management improvements
+- Added a `ReadOnly` authorizer that allows all READ operations but denies any other operation, such as WRITE [#19243](https://github.com/apache/druid/pull/19243)
+- Added `/status/ready` endpoint for service health so that external load balancers can handle a graceful shutdown better [#19148](https://github.com/apache/druid/pull/19148)
+- Added a configurable option to scale-down during task run time for cost-based autoscaler [#18958](https://github.com/apache/druid/pull/18958)
+- Added `storage_size` to `sys.servers` to facilitate retrieving disk cache size for Historicals when using the virtual storage fabric [#18979](https://github.com/apache/druid/pull/18979)
+- Added a log for new task count computation for the cost-based auto scaler [#18929](https://github.com/apache/druid/pull/18929)
+- Changed how the scaling is calculated from a square root-based scaling formula to a logarithmic formula that provides better emergency recovery at low task counts and millions of lag [#18976](https://github.com/apache/druid/pull/18976)
+- Improved the load speed of cached segments during Historical startup [#18489](https://github.com/apache/druid/pull/18489)
+- Improved Broker startup time by parallelizing buffer initialization [#19025](https://github.com/apache/druid/pull/19025)
+- Improved the stack trace for MSQ task engine worker failures so that they're preserved [#19049](https://github.com/apache/druid/pull/19049)
+- Improved the performance of the cost-based autoscaler during loaded lag conditions [#18991](https://github.com/apache/druid/pull/18991)
+
### Data management
+#### Durable storage cleaner
+
+The durable storage cleaner now supports configurable time-based retention for MSQ query results. Previously, query results were retained for all known tasks, which was unreliable for completed tasks. With this change, query results are retained for a configurable time period based on the task creation time.
+
+The new configuration property `druid.msq.intermediate.storage.cleaner.durationToRetain` controls the retention period for query results. The default retention period is 6 hours.
+
+[#19074](https://github.com/apache/druid/pull/19074)
+
#### Other data management improvements
+- Added the `druid.storage.transfer.asyncHttpClientType` config that specifies which async HTTP client to use for S3 transfers: `crt` for Amazon CRT or `netty` for Netty NIO [#19249](https://github.com/apache/druid/pull/19249)
+- Added a mechanism to automatically clean up intermediary files on HDFS storage [#19187](https://github.com/apache/druid/pull/19187)
+- Changed the default value for `druid.indexing.formats.maxStringLength` to null from 0 [#19198](https://github.com/apache/druid/pull/19198)
+
### Metrics and monitoring
+#### Version field
+
+All Druid metrics now include `version` and `buildRevision` fields to help you identify which version of Druid is emitting a metric. You can use this information to verify that all nodes in a cluster are running the intended revision during rolling deployments.
+
+[#19123](https://github.com/apache/druid/pull/19123)
+
+#### Monitoring supervisor state
+
+Added a new `supervisor/count` metric when `SupervisorStatsMonitor` is enabled in `druid.monitoring.monitors`. The metric reports each supervisor’s state, such as `RUNNING` or `SUSPENDED`, for Prometheus, StatsD, and other metric systems.
+
+[#19114](https://github.com/apache/druid/pull/19114)
+
+#### Improved `groupBy` metrics
+
+`GroupByStatsMonitor` now provides the following metrics:
+
+- `mergeBuffer/bytesUsed`
+- `mergeBuffer/maxAcquisitionTimeNs`
+- `groupBy/maxSpilledBytes`
+- `groupBy/maxMergeDictionarySize`
+
+[#18731](https://github.com/apache/druid/pull/18731) [#18970](https://github.com/apache/druid/pull/18970) [#18934](https://github.com/apache/druid/pull/18934)
+
+#### Filtering metrics
+
+Operators can limit which metrics the logging emitter writes by setting `druid.emitter.logging.shouldFilterMetrics=true` and, if desired, `druid.emitter.logging.allowedMetricsPath` to a JSON object file where the keys are metric names. A missing custom file results in a warning and use of the bundled `defaultMetrics.json`. Alerts and other non-metric events are always logged.
+
+[#19030](https://github.com/apache/druid/pull/19030)
+
+#### New Broker metrics
+
+Added `segment/schemaCache/rowSignature/changed` and `segment/schemaCache/rowSignature/column/count` metrics to get visibility into when the Broker's segment metadata cache's row signature for each datasource is initialized and updated.
+
+[#18966](https://github.com/apache/druid/pull/18966)
+
+#### Other metrics and monitoring improvements
+
+- Added the following metrics to the default for Prometheus: `mergeBuffer/bytesUsed` and `mergeBuffer/maxBytesUsed` [#19110](https://github.com/apache/druid/pull/19110)
+- Added compaction mode to what gets emitted for the `compact/task/count` metric [#19151](https://github.com/apache/druid/pull/19151)
+- Added support for logging and emitting SQL dynamic parameter values [#19067](https://github.com/apache/druid/pull/19067)
+- Added `ingest/rows/published`, which gets emitted from all task types to denote the total row count of successfully published segments [#19177](https://github.com/apache/druid/pull/19177)
+- Added `queries` and `totalQueries` counters, which reflect queries made to realtime servers to get realtime data
+- Added `tier/storage/capacity` metric for the Coordinator to emit that is guaranteed to reflect the total `StorageLocation` size configured across all Historicals in a tier [#18962](https://github.com/apache/druid/pull/18962)
Review Comment:
```suggestion
- Added `tier/storage/capacity` metric for the Coordinator. This metric is guaranteed to reflect the total `StorageLocation` size configured across all Historicals in a tier [#18962](https://github.com/apache/druid/pull/18962)
```
##########
docs/release-info/release-notes.md:
##########
@@ -57,50 +57,551 @@ For tips about how to write a good release note, see [Release notes](https://git
This section contains important information about new and existing features.
+### Hadoop-based ingestion
+
+Support for Hadoop-based ingestion has been removed. The feature was deprecated in Druid 34.
+
+Use one of Druid's other supported ingestion methods, such as SQL-based ingestion or MiddleManager-less ingestion using Kubernetes.
+
+[#19109](https://github.com/apache/druid/pull/19109)
+
+### Query blocklist
+
+You can now create a query blocklist for dynamically blocking queries without restarting your deployment. You can block queries by datasource, query type, or query context using the new `/druid/coordinator/v1/config/broker` API. Block rules use `AND` logic, which means all criteria must match. Druid stores the rules in the metadata database.
+
+The following example blocks all groupBy queries on the `wikipedia` datasource with a query context parameter of `priority` equal to `0`:
+
+```
+POST /druid/coordinator/v1/config/broker
+ {
+ "queryBlocklist": [
+ {
+ "ruleName": "block-wikipedia-groupbys",
+ "dataSources": ["wikipedia"],
+ "queryTypes": ["groupBy"],
+ "contextMatches": {"priority": "0"}
+ }
+ ]
+ }
+```
+
+[#19011](https://github.com/apache/druid/pull/19011)
+
+### Minor compaction for Overlord-based compaction (experimental)
+
+You can now configure minor compaction. With minor compaction, Druid only compacts newly ingested segments while upgrading existing compacted segments. When Druid upgrades segments, it updates the metadata instead of using resources to compact it again. You can use the native compaction engine or the MSQ task engine.
+
+Use the `mostFragmentedFirst` compaction policy and set either a percentage of rows-based or byte-based threshold for minor compaction.
+
+[#19059](https://github.com/apache/druid/pull/19059) [#19205](https://github.com/apache/druid/pull/19205) [#19016](https://github.com/apache/druid/pull/19016)
+
+### Thrift input format
+
+As part of the Thrift contributor extension, Druid now supports Thrift-encoded data for Kafka and Kinesis streaming ingestion.
+
+[#19111](https://github.com/apache/druid/pull/19111)
+
+To use this feature, you must add `druid-thrift-extensions` to your extension load list.
+
+### Kubernetes-based task management
+
+This extension is now generally available.
+
+[#19128](https://github.com/apache/druid/pull/19128)
+
+### Tombstones
+
+Tombstones for JSON-based native batch ingestion (the `dropExisting` flag for `ioConfig`) are now generally available.
+
+[#19128](https://github.com/apache/druid/pull/19128)
+
+### Dynamic default query context
+
+You can now add default query context parameters as a dynamic configuration to the Broker. This allows you to override static defaults set in your runtime properties without restarting your deployment or having to update multiple queries individually. Druid resolves which query context parameters to use based on the following priority:
+
+1. The query context included with the query
+1. The query context set as a dynamic configuration on the Broker
+1. The query context parameters set in the runtime properties
+1. The defaults that ship with Druid
+
+Note that like other Broker dynamic configuration, this is best-effort. Settings may not be applied in certain
+cases, such as when a Broker has recently started and hasn't received the config yet, or if the
+Broker can't contact the Coordinator. If a query context parameter is critical for all your queries, set it in the runtime properties.
+
+[#19146](https://github.com/apache/druid/pull/19146)
+
+### `sys.queries` table (experimental)
+
+The new system queries table provides information about currently running and recently completed queries that use the Dart engine. This table is off by default. To enable the table, set the following config:
+
+```
+druid.sql.planner.enableSysQueriesTable = true
+```
+
+As part of this change, the `/druid/v2/sql/queries` API now supports an `includeComplete` parameter that shows recently completed queries.
+
+[#18923](https://github.com/apache/druid/pull/18923)
+
+### Auto-compaction with compaction supervisors
+
+Auto-compaction using compaction supervisors has been improved and is now generally available.
+
+As part of the improvement, instead of individual compacted segments having to store their full compaction state in the metadata store, states are now stored in a central location, a new `indexingStates` table, in the metadata store. Individual segments only need to store a unique reference (`indexing_state_fingerprint`) to their full compaction state.
+
+Since many segments in a single datasource share the same underlying compaction state, this greatly reduces metadata storage requirements for automatic compaction.
+
+For backwards compatibility, detailed compaction state will continue to be persisted in each segment for now. This will be removed in a future release.
+
+You can stop storing detailed compaction state by setting `storeCompactionStatePerSegment` to `false` in the cluster compaction config. If you turn it off and need to downgrade, Druid needs to re-compact any segments that have been compacted since you changed the config.
+
+This change has upgrade impacts for metadata storage and metadata caching. For more information, see the [Metadata storage for auto-compaction with compaction supervisors](#metadata-storage-for-auto-compaction-with-compaction-supervisors) upgrade note.
+
+[#19113](https://github.com/apache/druid/pull/19113) [#18844](https://github.com/apache/druid/pull/18844)
+
+### Broker tier selection for realtime servers
+
+Added `druid.broker.realtime.select.tier` and `druid.broker.realtime.balancer.type` on the Brokers to optionally override the Broker’s tier selection and balancer strategies for realtime servers. If these properties are not set (the default), realtime servers continue to use the existing `druid.broker.select` and `druid.broker.balancer` configurations that apply to both historical and realtime servers.
+
+[#19062](https://github.com/apache/druid/pull/19062)
+
+### Manual Broker routing in the web console
+
+You can now configure which Broker the Router uses for queries issued from the web console. You may want to do this if there are Brokers that don't have visibility into certain data tiers, and you know you're querying data available only on a certain tier.
+
+To specify a Broker, add the following config to `web-console/console-config.js`:
+
+```js
+consoleBrokerService: 'druid/BROKER_NAME'
+```
+
+[#19069](https://github.com/apache/druid/pull/19069)
+
+### Consul extension
+
+The contributor extension `druid-consul-extensions` lets Druid clusters use Consul for service discovery and
+Coordinator/Overlord leader election instead of ZooKeeper. The extension supports ACLs, TLS/mTLS, and metrics.
+
+Before you switch to Consul, you need to set
+`druid.serverview.type=http` and `druid.indexer.runner.type=httpRemote` cluster wide.
+
+[#18843](https://github.com/apache/druid/pull/18843)
+
## Functional area and related changes
This section contains detailed release notes separated by areas.
### Web console
+#### Changed storage column displays
+
+- Improved the compaction config view to
+- Renamed **Current size** to **Assigned size**.
+- Renamed **Max size** to **Effective size**. It now displays the smaller value between `max_size` and `storage_size`. The max size is still shown as a tooltip.
+- Changed usage calculation to use `effective_size`
+
+[#19007](https://github.com/apache/druid/pull/19007)
+
#### Other web console improvements
+- Added `workerDesc` to `WorkerStats`, which makes it easier to identify where a worker is running [#19171](https://github.com/apache/druid/pull/19171)
+- Added the Dart unique execution ID (`dartQueryId`) and the `sqlQueryId` to the **Details** pane in the web console [#19185](https://github.com/apache/druid/pull/19185)
+- Added support for showing completed Dart queries in the web console [#18940](https://github.com/apache/druid/pull/18940)
+- Added a detail dialog to the **Services** page [#18960](https://github.com/apache/druid/pull/18960)
+- Added icons to indicate when data is loaded into virtual storage, including a tooltip that shows all the counters for the data [#19010](https://github.com/apache/druid/pull/19010)
+- Added support for Dart reports [#18897](https://github.com/apache/druid/pull/18897)
+- Changed what's considered an active worker: any nonzero rows, files, bytes, frames, or wall time is enough to consider a worker active [#19183](https://github.com/apache/druid/pull/19183)
+- Changed the **Cancel query** option to show only if a query is in an accepted or running state [#19182](https://github.com/apache/druid/pull/19182)
+- Changed the ordering of the current Dart queries panel to show queries in the following order: RUNNING, ACCEPTED, and then COMPLETED. RUNNING and ACCEPTED queries are ordered by the most recent first (based on timestamp). COMPLETED queries are sorted by finish time [#19237](https://github.com/apache/druid/pull/19237)
+
### Ingestion
#### SQL-based ingestion
##### Other SQL-based ingestion improvements
+- Added support for virtual storage fabric when performing SQL-based ingestion [#18873](https://github.com/apache/druid/pull/18873)
+- Added support for `StorageMonitor` so that MSQ task engine tasks always emit `taskId` and `groupId` [#19048](https://github.com/apache/druid/pull/19048)
+- Improved worker cancellation [#18931](https://github.com/apache/druid/pull/18931)
+- Improved exception handling [#19234](https://github.com/apache/druid/pull/19234)
+
#### Streaming ingestion
+##### Changed how tasks get launched for autoscaling
+
+The behavior of `taskCountMin` and `taskCountStart` has been changed for autoscaling. Druid now computes the initial number of tasks to launch by checking the configs in the following order: `taskCountStart` (optional), then `taskCount` (in `ioConfig`), then `taskCountMin`.
+
+[#19091](https://github.com/apache/druid/pull/19091)
+
+##### Improved query isolation
+
+Added `serverPriorityToReplicas` parameter to the streaming supervisor specs (Kafka, Kinesis, and Rabbit). This allows operators to distribute task replicas across different server priorities for realtime indexing tasks. Similar to Historical tiering, this enables query isolation for mixed workload scenarios on the Peons, allowing some task replicas to handle queries of specific priorities.
+
+[#19040](https://github.com/apache/druid/pull/19040)
+
##### Other streaming ingestion improvements
+- Improved cost-based autoscaler performance in high lag scenarios [#19045](https://github.com/apache/druid/pull/19045)
+- Improved the performance of realtime task scheduling by ordering schedule requests by priority on the `TaskQueue` [#19203](https://github.com/apache/druid/pull/19203)
+
+
+#### Ingestion improvements
+
+- Sped up task scheduling on the Overlord [#19199](https://github.com/apache/druid/pull/19199)
+
### Querying
+#### groupBy query configuration
+
+Added a new groupBy query configuration property `druid.query.groupBy.maxSpillFileCount` to limit the maximum number of spill files created per query. When the limit is exceeded, the query fails with a clear error message instead of causing Historical nodes to run out of memory during spill file merging. The limit can also be overridden per query using the query context parameter `maxSpillFileCount`.
+
+[#19141](https://github.com/apache/druid/pull/19141)
+
+#### Improved handling of nested aggregates
+
+Druid can now merge two Aggregates with a Projection between them. For example, the following query:
+
+```sql
+SELECT
+ hr,
+ UPPER(t1.x) x,
+ SUM(t1.cnt) cnt,
+ MIN(t1.mn) mn,
+ MAX(t1.mx) mx
+FROM (
+ SELECT
+ floor(__time to hour) hr,
+ dim2 x,
+ COUNT(*) cnt,
+ MIN(m1 * 5) mn,
+ MAX(m1 + m2) mx
+ FROM druid.foo
+ WHERE dim2 IN ('abc', 'def', 'a', 'b', '')
+ GROUP BY 1, 2
+) t1
+WHERE t1.x IN ('abc', 'foo', 'bar', 'a', '')
+GROUP BY 1, 2
+```
+
+can be simplified to the following:
+
+```sql
+SELECT
+ FLOOR(__time TO hour) hr,
+ UPPER(dim2) x,
+ COUNT(*) cnt,
+ MIN(m1 * 5) mn,
+ MAX(m1 + m2) mx
+FROM druid.foo
+WHERE dim2 IN ('abc', 'a', '')
+GROUP BY 1, 2
+```
+
+[#18498](https://github.com/apache/druid/pull/18498)
+
#### Other querying improvements
+- Added `durationMs` to Dart query reports [#19169](https://github.com/apache/druid/pull/19169)
+- Improved error handling so that row signature column order is preserved when column analysis encounters an error [#19162](https://github.com/apache/druid/pull/19162)
+- Improved GROUP BY performance [#18952](https://github.com/apache/druid/pull/18952)
+- Improved expression filters to take advantage of specialized virtual columns when possible, resulting in better performance for the query [#18965](https://github.com/apache/druid/pull/18965)
+
### Cluster management
+#### New Broker tier selection strategies
+
+Operators can now configure two new Broker `TierSelectorStrategy` implementations:
+
+- `strict` - Only selects servers whose priorities match the configured list. Example configuration: `druid.broker.select.tier=strict` and `druid.broker.select.tier.strict.priorities=[1]`.
+- `pooled` - Pools servers across the configured priorities and selects among them, allowing queries to utilize multiple priority tiers for improved availability. Example configuration: `druid.broker.select.tier=pooled` and `druid.broker.select.tier.pooled.priorities=[2,1]`.
+
+These strategies can also be configured for realtime servers using `druid.broker.realtime.select.tier`.
+
+[#19094](https://github.com/apache/druid/pull/19094)
+
+#### Druid operator
+
+The Druid operator now resides in its own repository: [`apache/druid-operator`](https://github.com/apache/druid-operator).
+
+[#19156](https://github.com/apache/druid/pull/19156)
+
+#### Cost-based autoscaler algorithm
+
+The algorithm for cost-based autoscaling has been changed:
+
+- Scale up more aggressively when per-partition lag is meaningful
+- Relax the partitions-per-task increase limit based on lag severity and headroom
+- Keep behavior conservative near `taskCountMax` and avoid negative headroom effects
+
+[#18936](https://github.com/apache/druid/pull/18936)
+
#### Other cluster management improvements
+- Added a `ReadOnly` authorizer that allows all READ operations but denies any other operation, such as WRITE [#19243](https://github.com/apache/druid/pull/19243)
+- Added `/status/ready` endpoint for service health so that external load balancers can handle a graceful shutdown better [#19148](https://github.com/apache/druid/pull/19148)
+- Added a configurable option to scale-down during task run time for cost-based autoscaler [#18958](https://github.com/apache/druid/pull/18958)
+- Added `storage_size` to `sys.servers` to facilitate retrieving disk cache size for Historicals when using the virtual storage fabric [#18979](https://github.com/apache/druid/pull/18979)
+- Added a log for new task count computation for the cost-based auto scaler [#18929](https://github.com/apache/druid/pull/18929)
+- Changed how the scaling is calculated from a square root-based scaling formula to a logarithmic formula that provides better emergency recovery at low task counts and millions of lag [#18976](https://github.com/apache/druid/pull/18976)
+- Improved the load speed of cached segments during Historical startup [#18489](https://github.com/apache/druid/pull/18489)
+- Improved Broker startup time by parallelizing buffer initialization [#19025](https://github.com/apache/druid/pull/19025)
+- Improved the stack trace for MSQ task engine worker failures so that they're preserved [#19049](https://github.com/apache/druid/pull/19049)
+- Improved the performance of the cost-based autoscaler during loaded lag conditions [#18991](https://github.com/apache/druid/pull/18991)
+
### Data management
+#### Durable storage cleaner
+
+The durable storage cleaner now supports configurable time-based retention for MSQ query results. Previously, query results were retained for all known tasks, which was unreliable for completed tasks. With this change, query results are retained for a configurable time period based on the task creation time.
+
+The new configuration property `druid.msq.intermediate.storage.cleaner.durationToRetain` controls the retention period for query results. The default retention period is 6 hours.
+
+[#19074](https://github.com/apache/druid/pull/19074)
+
#### Other data management improvements
+- Added the `druid.storage.transfer.asyncHttpClientType` config that specifies which async HTTP client to use for S3 transfers: `crt` for Amazon CRT or `netty` for Netty NIO [#19249](https://github.com/apache/druid/pull/19249)
+- Added a mechanism to automatically clean up intermediary files on HDFS storage [#19187](https://github.com/apache/druid/pull/19187)
+- Changed the default value for `druid.indexing.formats.maxStringLength` to null from 0 [#19198](https://github.com/apache/druid/pull/19198)
+
### Metrics and monitoring
+#### Version field
+
+All Druid metrics now include `version` and `buildRevision` fields to help you identify which version of Druid is emitting a metric. You can use this information to verify that all nodes in a cluster are running the intended revision during rolling deployments.
+
+[#19123](https://github.com/apache/druid/pull/19123)
+
+#### Monitoring supervisor state
+
+Added a new `supervisor/count` metric when `SupervisorStatsMonitor` is enabled in `druid.monitoring.monitors`. The metric reports each supervisor’s state, such as `RUNNING` or `SUSPENDED`, for Prometheus, StatsD, and other metric systems.
+
+[#19114](https://github.com/apache/druid/pull/19114)
+
+#### Improved `groupBy` metrics
+
+`GroupByStatsMonitor` now provides the following metrics:
+
+- `mergeBuffer/bytesUsed`
+- `mergeBuffer/maxAcquisitionTimeNs`
+- `groupBy/maxSpilledBytes`
+- `groupBy/maxMergeDictionarySize`
+
+[#18731](https://github.com/apache/druid/pull/18731) [#18970](https://github.com/apache/druid/pull/18970) [#18934](https://github.com/apache/druid/pull/18934)
+
+#### Filtering metrics
+
+Operators can limit which metrics the logging emitter writes by setting `druid.emitter.logging.shouldFilterMetrics=true` and, if desired, `druid.emitter.logging.allowedMetricsPath` to a JSON object file where the keys are metric names. A missing custom file results in a warning and use of the bundled `defaultMetrics.json`. Alerts and other non-metric events are always logged.
+
+[#19030](https://github.com/apache/druid/pull/19030)
+
+#### New Broker metrics
+
+Added `segment/schemaCache/rowSignature/changed` and `segment/schemaCache/rowSignature/column/count` metrics to get visibility into when the Broker's segment metadata cache's row signature for each datasource is initialized and updated.
+
+[#18966](https://github.com/apache/druid/pull/18966)
+
+#### Other metrics and monitoring improvements
+
+- Added the following metrics to the default for Prometheus: `mergeBuffer/bytesUsed` and `mergeBuffer/maxBytesUsed` [#19110](https://github.com/apache/druid/pull/19110)
+- Added compaction mode to what gets emitted for the `compact/task/count` metric [#19151](https://github.com/apache/druid/pull/19151)
Review Comment:
```suggestion
- Added compaction mode to the `compact/task/count` metric [#19151](https://github.com/apache/druid/pull/19151)
```
##########
docs/release-info/release-notes.md:
##########
@@ -57,50 +57,551 @@ For tips about how to write a good release note, see [Release notes](https://git
This section contains important information about new and existing features.
+### Hadoop-based ingestion
+
+Support for Hadoop-based ingestion has been removed. The feature was deprecated in Druid 34.
+
+Use one of Druid's other supported ingestion methods, such as SQL-based ingestion or MiddleManager-less ingestion using Kubernetes.
+
+[#19109](https://github.com/apache/druid/pull/19109)
+
+### Query blocklist
+
+You can now create a query blocklist for dynamically blocking queries without restarting your deployment. You can block queries by datasource, query type, or query context using the new `/druid/coordinator/v1/config/broker` API. Block rules use `AND` logic, which means all criteria must match. Druid stores the rules in the metadata database.
+
+The following example blocks all groupBy queries on the `wikipedia` datasource with a query context parameter of `priority` equal to `0`:
+
+```
+POST /druid/coordinator/v1/config/broker
+ {
+ "queryBlocklist": [
+ {
+ "ruleName": "block-wikipedia-groupbys",
+ "dataSources": ["wikipedia"],
+ "queryTypes": ["groupBy"],
+ "contextMatches": {"priority": "0"}
+ }
+ ]
+ }
+```
+
+[#19011](https://github.com/apache/druid/pull/19011)
+
+### Minor compaction for Overlord-based compaction (experimental)
+
+You can now configure minor compaction. With minor compaction, Druid only compacts newly ingested segments while upgrading existing compacted segments. When Druid upgrades segments, it updates the metadata instead of using resources to compact it again. You can use the native compaction engine or the MSQ task engine.
+
+Use the `mostFragmentedFirst` compaction policy and set either a percentage of rows-based or byte-based threshold for minor compaction.
+
+[#19059](https://github.com/apache/druid/pull/19059) [#19205](https://github.com/apache/druid/pull/19205) [#19016](https://github.com/apache/druid/pull/19016)
+
+### Thrift input format
+
+As part of the Thrift contributor extension, Druid now supports Thrift-encoded data for Kafka and Kinesis streaming ingestion.
+
+[#19111](https://github.com/apache/druid/pull/19111)
+
+To use this feature, you must add `druid-thrift-extensions` to your extension load list.
+
+### Kubernetes-based task management
+
+This extension is now generally available.
+
+[#19128](https://github.com/apache/druid/pull/19128)
+
+### Tombstones
+
+Tombstones for JSON-based native batch ingestion (the `dropExisting` flag for `ioConfig`) are now generally available.
+
+[#19128](https://github.com/apache/druid/pull/19128)
+
+### Dynamic default query context
+
+You can now add default query context parameters as a dynamic configuration to the Broker. This allows you to override static defaults set in your runtime properties without restarting your deployment or having to update multiple queries individually. Druid resolves which query context parameters to use based on the following priority:
+
+1. The query context included with the query
+1. The query context set as a dynamic configuration on the Broker
+1. The query context parameters set in the runtime properties
+1. The defaults that ship with Druid
+
+Note that like other Broker dynamic configuration, this is best-effort. Settings may not be applied in certain
+cases, such as when a Broker has recently started and hasn't received the config yet, or if the
+Broker can't contact the Coordinator. If a query context parameter is critical for all your queries, set it in the runtime properties.
+
+[#19146](https://github.com/apache/druid/pull/19146)
+
+### `sys.queries` table (experimental)
+
+The new system queries table provides information about currently running and recently completed queries that use the Dart engine. This table is off by default. To enable the table, set the following config:
+
+```
+druid.sql.planner.enableSysQueriesTable = true
+```
+
+As part of this change, the `/druid/v2/sql/queries` API now supports an `includeComplete` parameter that shows recently completed queries.
+
+[#18923](https://github.com/apache/druid/pull/18923)
+
+### Auto-compaction with compaction supervisors
+
+Auto-compaction using compaction supervisors has been improved and is now generally available.
+
+As part of the improvement, instead of individual compacted segments having to store their full compaction state in the metadata store, states are now stored in a central location, a new `indexingStates` table, in the metadata store. Individual segments only need to store a unique reference (`indexing_state_fingerprint`) to their full compaction state.
+
+Since many segments in a single datasource share the same underlying compaction state, this greatly reduces metadata storage requirements for automatic compaction.
+
+For backwards compatibility, detailed compaction state will continue to be persisted in each segment for now. This will be removed in a future release.
+
+You can stop storing detailed compaction state by setting `storeCompactionStatePerSegment` to `false` in the cluster compaction config. If you turn it off and need to downgrade, Druid needs to re-compact any segments that have been compacted since you changed the config.
+
+This change has upgrade impacts for metadata storage and metadata caching. For more information, see the [Metadata storage for auto-compaction with compaction supervisors](#metadata-storage-for-auto-compaction-with-compaction-supervisors) upgrade note.
+
+[#19113](https://github.com/apache/druid/pull/19113) [#18844](https://github.com/apache/druid/pull/18844)
+
+### Broker tier selection for realtime servers
+
+Added `druid.broker.realtime.select.tier` and `druid.broker.realtime.balancer.type` on the Brokers to optionally override the Broker’s tier selection and balancer strategies for realtime servers. If these properties are not set (the default), realtime servers continue to use the existing `druid.broker.select` and `druid.broker.balancer` configurations that apply to both historical and realtime servers.
+
+[#19062](https://github.com/apache/druid/pull/19062)
+
+### Manual Broker routing in the web console
+
+You can now configure which Broker the Router uses for queries issued from the web console. You may want to do this if there are Brokers that don't have visibility into certain data tiers, and you know you're querying data available only on a certain tier.
+
+To specify a Broker, add the following config to `web-console/console-config.js`:
+
+```js
+consoleBrokerService: 'druid/BROKER_NAME'
+```
+
+[#19069](https://github.com/apache/druid/pull/19069)
+
+### Consul extension
+
+The contributor extension `druid-consul-extensions` lets Druid clusters use Consul for service discovery and
+Coordinator/Overlord leader election instead of ZooKeeper. The extension supports ACLs, TLS/mTLS, and metrics.
+
+Before you switch to Consul, you need to set
+`druid.serverview.type=http` and `druid.indexer.runner.type=httpRemote` cluster wide.
+
+[#18843](https://github.com/apache/druid/pull/18843)
+
## Functional area and related changes
This section contains detailed release notes separated by areas.
### Web console
+#### Changed storage column displays
+
+- Improved the compaction config view to
+- Renamed **Current size** to **Assigned size**.
+- Renamed **Max size** to **Effective size**. It now displays the smaller value between `max_size` and `storage_size`. The max size is still shown as a tooltip.
+- Changed usage calculation to use `effective_size`
+
+[#19007](https://github.com/apache/druid/pull/19007)
+
#### Other web console improvements
+- Added `workerDesc` to `WorkerStats`, which makes it easier to identify where a worker is running [#19171](https://github.com/apache/druid/pull/19171)
+- Added the Dart unique execution ID (`dartQueryId`) and the `sqlQueryId` to the **Details** pane in the web console [#19185](https://github.com/apache/druid/pull/19185)
+- Added support for showing completed Dart queries in the web console [#18940](https://github.com/apache/druid/pull/18940)
+- Added a detail dialog to the **Services** page [#18960](https://github.com/apache/druid/pull/18960)
+- Added icons to indicate when data is loaded into virtual storage, including a tooltip that shows all the counters for the data [#19010](https://github.com/apache/druid/pull/19010)
+- Added support for Dart reports [#18897](https://github.com/apache/druid/pull/18897)
+- Changed what's considered an active worker: any nonzero rows, files, bytes, frames, or wall time is enough to consider a worker active [#19183](https://github.com/apache/druid/pull/19183)
+- Changed the **Cancel query** option to show only if a query is in an accepted or running state [#19182](https://github.com/apache/druid/pull/19182)
+- Changed the ordering of the current Dart queries panel to show queries in the following order: RUNNING, ACCEPTED, and then COMPLETED. RUNNING and ACCEPTED queries are ordered by the most recent first (based on timestamp). COMPLETED queries are sorted by finish time [#19237](https://github.com/apache/druid/pull/19237)
+
### Ingestion
#### SQL-based ingestion
##### Other SQL-based ingestion improvements
+- Added support for virtual storage fabric when performing SQL-based ingestion [#18873](https://github.com/apache/druid/pull/18873)
+- Added support for `StorageMonitor` so that MSQ task engine tasks always emit `taskId` and `groupId` [#19048](https://github.com/apache/druid/pull/19048)
+- Improved worker cancellation [#18931](https://github.com/apache/druid/pull/18931)
+- Improved exception handling [#19234](https://github.com/apache/druid/pull/19234)
+
#### Streaming ingestion
+##### Changed how tasks get launched for autoscaling
+
+The behavior of `taskCountMin` and `taskCountStart` has been changed for autoscaling. Druid now computes the initial number of tasks to launch by checking the configs in the following order: `taskCountStart` (optional), then `taskCount` (in `ioConfig`), then `taskCountMin`.
+
+[#19091](https://github.com/apache/druid/pull/19091)
+
+##### Improved query isolation
+
+Added `serverPriorityToReplicas` parameter to the streaming supervisor specs (Kafka, Kinesis, and Rabbit). This allows operators to distribute task replicas across different server priorities for realtime indexing tasks. Similar to Historical tiering, this enables query isolation for mixed workload scenarios on the Peons, allowing some task replicas to handle queries of specific priorities.
+
+[#19040](https://github.com/apache/druid/pull/19040)
+
##### Other streaming ingestion improvements
+- Improved cost-based autoscaler performance in high lag scenarios [#19045](https://github.com/apache/druid/pull/19045)
+- Improved the performance of realtime task scheduling by ordering schedule requests by priority on the `TaskQueue` [#19203](https://github.com/apache/druid/pull/19203)
+
+
+#### Ingestion improvements
+
+- Sped up task scheduling on the Overlord [#19199](https://github.com/apache/druid/pull/19199)
+
### Querying
+#### groupBy query configuration
+
+Added a new groupBy query configuration property `druid.query.groupBy.maxSpillFileCount` to limit the maximum number of spill files created per query. When the limit is exceeded, the query fails with a clear error message instead of causing Historical nodes to run out of memory during spill file merging. The limit can also be overridden per query using the query context parameter `maxSpillFileCount`.
+
+[#19141](https://github.com/apache/druid/pull/19141)
+
+#### Improved handling of nested aggregates
+
+Druid can now merge two Aggregates with a Projection between them. For example, the following query:
+
+```sql
+SELECT
+ hr,
+ UPPER(t1.x) x,
+ SUM(t1.cnt) cnt,
+ MIN(t1.mn) mn,
+ MAX(t1.mx) mx
+FROM (
+ SELECT
+ floor(__time to hour) hr,
+ dim2 x,
+ COUNT(*) cnt,
+ MIN(m1 * 5) mn,
+ MAX(m1 + m2) mx
+ FROM druid.foo
+ WHERE dim2 IN ('abc', 'def', 'a', 'b', '')
+ GROUP BY 1, 2
+) t1
+WHERE t1.x IN ('abc', 'foo', 'bar', 'a', '')
+GROUP BY 1, 2
+```
+
+can be simplified to the following:
+
+```sql
+SELECT
+ FLOOR(__time TO hour) hr,
+ UPPER(dim2) x,
+ COUNT(*) cnt,
+ MIN(m1 * 5) mn,
+ MAX(m1 + m2) mx
+FROM druid.foo
+WHERE dim2 IN ('abc', 'a', '')
+GROUP BY 1, 2
+```
+
+[#18498](https://github.com/apache/druid/pull/18498)
+
#### Other querying improvements
+- Added `durationMs` to Dart query reports [#19169](https://github.com/apache/druid/pull/19169)
+- Improved error handling so that row signature column order is preserved when column analysis encounters an error [#19162](https://github.com/apache/druid/pull/19162)
+- Improved GROUP BY performance [#18952](https://github.com/apache/druid/pull/18952)
+- Improved expression filters to take advantage of specialized virtual columns when possible, resulting in better performance for the query [#18965](https://github.com/apache/druid/pull/18965)
+
### Cluster management
+#### New Broker tier selection strategies
+
+Operators can now configure two new Broker `TierSelectorStrategy` implementations:
+
+- `strict` - Only selects servers whose priorities match the configured list. Example configuration: `druid.broker.select.tier=strict` and `druid.broker.select.tier.strict.priorities=[1]`.
+- `pooled` - Pools servers across the configured priorities and selects among them, allowing queries to utilize multiple priority tiers for improved availability. Example configuration: `druid.broker.select.tier=pooled` and `druid.broker.select.tier.pooled.priorities=[2,1]`.
+
+These strategies can also be configured for realtime servers using `druid.broker.realtime.select.tier`.
+
+[#19094](https://github.com/apache/druid/pull/19094)
+
+#### Druid operator
+
+The Druid operator now resides in its own repository: [`apache/druid-operator`](https://github.com/apache/druid-operator).
+
+[#19156](https://github.com/apache/druid/pull/19156)
+
+#### Cost-based autoscaler algorithm
+
+The algorithm for cost-based autoscaling has been changed:
+
+- Scale up more aggressively when per-partition lag is meaningful
+- Relax the partitions-per-task increase limit based on lag severity and headroom
+- Keep behavior conservative near `taskCountMax` and avoid negative headroom effects
+
+[#18936](https://github.com/apache/druid/pull/18936)
+
#### Other cluster management improvements
+- Added a `ReadOnly` authorizer that allows all READ operations but denies any other operation, such as WRITE [#19243](https://github.com/apache/druid/pull/19243)
+- Added `/status/ready` endpoint for service health so that external load balancers can handle a graceful shutdown better [#19148](https://github.com/apache/druid/pull/19148)
+- Added a configurable option to scale-down during task run time for cost-based autoscaler [#18958](https://github.com/apache/druid/pull/18958)
+- Added `storage_size` to `sys.servers` to facilitate retrieving disk cache size for Historicals when using the virtual storage fabric [#18979](https://github.com/apache/druid/pull/18979)
+- Added a log for new task count computation for the cost-based auto scaler [#18929](https://github.com/apache/druid/pull/18929)
+- Changed how the scaling is calculated from a square root-based scaling formula to a logarithmic formula that provides better emergency recovery at low task counts and millions of lag [#18976](https://github.com/apache/druid/pull/18976)
+- Improved the load speed of cached segments during Historical startup [#18489](https://github.com/apache/druid/pull/18489)
+- Improved Broker startup time by parallelizing buffer initialization [#19025](https://github.com/apache/druid/pull/19025)
+- Improved the stack trace for MSQ task engine worker failures so that they're preserved [#19049](https://github.com/apache/druid/pull/19049)
+- Improved the performance of the cost-based autoscaler during loaded lag conditions [#18991](https://github.com/apache/druid/pull/18991)
+
### Data management
+#### Durable storage cleaner
+
+The durable storage cleaner now supports configurable time-based retention for MSQ query results. Previously, query results were retained for all known tasks, which was unreliable for completed tasks. With this change, query results are retained for a configurable time period based on the task creation time.
+
+The new configuration property `druid.msq.intermediate.storage.cleaner.durationToRetain` controls the retention period for query results. The default retention period is 6 hours.
+
+[#19074](https://github.com/apache/druid/pull/19074)
+
#### Other data management improvements
+- Added the `druid.storage.transfer.asyncHttpClientType` config that specifies which async HTTP client to use for S3 transfers: `crt` for Amazon CRT or `netty` for Netty NIO [#19249](https://github.com/apache/druid/pull/19249)
+- Added a mechanism to automatically clean up intermediary files on HDFS storage [#19187](https://github.com/apache/druid/pull/19187)
+- Changed the default value for `druid.indexing.formats.maxStringLength` to null from 0 [#19198](https://github.com/apache/druid/pull/19198)
+
### Metrics and monitoring
+#### Version field
+
+All Druid metrics now include `version` and `buildRevision` fields to help you identify which version of Druid is emitting a metric. You can use this information to verify that all nodes in a cluster are running the intended revision during rolling deployments.
+
+[#19123](https://github.com/apache/druid/pull/19123)
+
+#### Monitoring supervisor state
+
+Added a new `supervisor/count` metric when `SupervisorStatsMonitor` is enabled in `druid.monitoring.monitors`. The metric reports each supervisor’s state, such as `RUNNING` or `SUSPENDED`, for Prometheus, StatsD, and other metric systems.
+
+[#19114](https://github.com/apache/druid/pull/19114)
+
+#### Improved `groupBy` metrics
+
+`GroupByStatsMonitor` now provides the following metrics:
+
+- `mergeBuffer/bytesUsed`
+- `mergeBuffer/maxAcquisitionTimeNs`
+- `groupBy/maxSpilledBytes`
+- `groupBy/maxMergeDictionarySize`
+
+[#18731](https://github.com/apache/druid/pull/18731) [#18970](https://github.com/apache/druid/pull/18970) [#18934](https://github.com/apache/druid/pull/18934)
+
+#### Filtering metrics
+
+Operators can limit which metrics the logging emitter writes by setting `druid.emitter.logging.shouldFilterMetrics=true` and, if desired, `druid.emitter.logging.allowedMetricsPath` to a JSON object file where the keys are metric names. A missing custom file results in a warning and use of the bundled `defaultMetrics.json`. Alerts and other non-metric events are always logged.
+
+[#19030](https://github.com/apache/druid/pull/19030)
+
+#### New Broker metrics
+
+Added `segment/schemaCache/rowSignature/changed` and `segment/schemaCache/rowSignature/column/count` metrics to get visibility into when the Broker's segment metadata cache's row signature for each datasource is initialized and updated.
+
+[#18966](https://github.com/apache/druid/pull/18966)
+
+#### Other metrics and monitoring improvements
+
+- Added the following metrics to the default for Prometheus: `mergeBuffer/bytesUsed` and `mergeBuffer/maxBytesUsed` [#19110](https://github.com/apache/druid/pull/19110)
+- Added compaction mode to what gets emitted for the `compact/task/count` metric [#19151](https://github.com/apache/druid/pull/19151)
+- Added support for logging and emitting SQL dynamic parameter values [#19067](https://github.com/apache/druid/pull/19067)
+- Added `ingest/rows/published`, which gets emitted from all task types to denote the total row count of successfully published segments [#19177](https://github.com/apache/druid/pull/19177)
+- Added `queries` and `totalQueries` counters, which reflect queries made to realtime servers to get realtime data
+- Added `tier/storage/capacity` metric for the Coordinator to emit that is guaranteed to reflect the total `StorageLocation` size configured across all Historicals in a tier [#18962](https://github.com/apache/druid/pull/18962)
+- Added new metrics for virtual storage fabric to the MSQ task engine `ChannelCounters`: `loadBytes`, `loadTime`, `loadWait`, and `loadFiles` [#18971](https://github.com/apache/druid/pull/18971)
+- Added `storage/virtual/hit/bytes`, `storage/virtual/hold/count` and `storage/virtual/hold/bytes` metric to `StorageMonitor`
+[#18895](https://github.com/apache/druid/pull/18895) [#19217](https://github.com/apache/druid/pull/19217)
+- Added `supervisorId` dimension for streaming tasks to `TaskCountStatsMonitor` [#18920](https://github.com/apache/druid/pull/18920)
+- Changed `StorageMonitor` to always be on [#19048](https://github.com/apache/druid/pull/19048)
+- Improved the metrics for autoscalers, so that they all emit the same metrics: `supervisorId`, `dataSource`, and `stream` [#19097](https://github.com/apache/druid/pull/19097)
+
### Extensions
-### Documentation improvements
+#### gRPC
+
+The gRPC query extension now cancels in-flight queries when clients cancel or disconnect.
+
+[#19005](https://github.com/apache/druid/pull/19005)
+
+#### Iceberg
+
+##### GCS warehouse
+
+The Iceberg input source now supports GCS warehouses. To use this feature, you need to load the `druid-google-extensions` extension in addition to the Iceberg extension.
Review Comment:
```suggestion
The Iceberg input source now supports GCS warehouses. To use this feature, you must load the `druid-google-extensions` extension in addition to the Iceberg extension.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org