You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "317brian (via GitHub)" <gi...@apache.org> on 2023/07/17 19:37:53 UTC
[PR] DO NOT MERGE - 27.0.0 WIP release notes (druid)
317brian opened a new pull request, #14600:
URL: https://github.com/apache/druid/pull/14600
Draft PR used to collaborate on the release notes so that we can use the comment/suggestion system.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] DO NOT MERGE - 27.0.0 WIP release notes (druid)
Posted by "2bethere (via GitHub)" <gi...@apache.org>.
2bethere commented on code in PR #14600:
URL: https://github.com/apache/druid/pull/14600#discussion_r1281389920
##########
docs/do-not-merge.md:
##########
@@ -0,0 +1,464 @@
+<!--Intentionally, there's no Apache license so that the GHA fails it. This file is not meant to be merged.
+
+- https://github.com/apache/druid/pull/14266 - we removed input source security from 26 (https://github.com/apache/druid/pull/14003). Should we not include this in 27 release notes?
+
+-->
+
+# Still need
+
+search for TBD
+
+* Caveats for array column types
+* More info about Hadoop 2 deprecation
+* Info for parameter execution highlight
+* Caching cost for upgrade notes - is it this? [#14484](https://github.com/apache/druid/pull/14484)
+* User impact of https://github.com/apache/druid/pull/14048 for highlights
+* https://github.com/apache/druid/pull/14476 seems to only be documented here in release notes and the PR?
+* Vad's input for web console section
+* PR for Temporary storage as a runtime parameter. MSQ will start honoring changes to the Middle Manager.
+* What to do about input source security. There are no docs for it.
+* New query filters
+* Confirm the correct naming in https://github.com/apache/druid/pull/14359 (OSHI sys mon change). The changed files say one thing and the PR description says another
+* Community extensions. What do we want to say about iceberg and the other one (what was the other one?)
+
+Apache Druid 27.0.0 contains over $NUMBER_FEATURES new features, bug fixes, performance enhancements, documentation improvements, and additional test coverage from $NUMBER_OF_CONTRIBUTORS contributors.
+
+[See the complete set of changes for additional details](https://github.com/apache/druid/issues?q=is%3Aclosed+milestone%3A27.0+sort%3Aupdated-desc+), including [bug fixes](https://github.com/apache/druid/issues?q=is%3Aclosed+milestone%3A27.0+sort%3Aupdated-desc+label%3ABug).
+
+Review the upgrade notes and incompatible changes before you upgrade to Druid 27.0.0.
+
+# Highlights
+
+<!-- HIGHLIGHTS H2. FOR EACH MAJOR FEATURE FOR THE RELEASE -->
+
+## Query from deep storage (experimental)
+
+Druid now supports querying segments that are stored only in deep storage. When you query from deep storage, you can query larger data available for queries without necessarily having to scale your Historical processes to accommodate more data. To take advantage of the potential storage savings, make sure you configure your load rules to not load all your segments onto Historical processes.
+
+Note that at least one segment of a datasource must be loaded onto a Historical process so that the Broker can plan the query. It can be any segment though.
+
+For more information, see the following:
+
+- [Query from deep storage](https://druid.apache.org/docs/latest/querying/query-deep-storage.html)
+- [Query from deep storage tutorial](https://druid.apache.org/docs/latest/tutorials/tutorial-query-deep-storage.html)
+
+[#14416](https://github.com/apache/druid/pull/14416) [#14512](https://github.com/apache/druid/pull/14512) [#14527](https://github.com/apache/druid/pull/14527)
+
+## Schema auto-discovery and array column types
+
+Type-aware schema auto-discovery is now generally available. Druid can determine the schema for the data you ingest rather than you having to manually define the schema.
+
+As part of the type-aware schema discovery improvements, array column types are now generally available. Druid can determine the column types for your schema and assign them to these array column types when you ingest data using type-aware schema auto-discovery with the `auto` column type.
+
+Keep the following in mind when using these features:
+
+- TBD
+
+For more information about this feature, see the following:
+
+- [Type-aware schema discovery](https://druid.apache.org/docs/latest/ingestion/schema-design.html#type-aware-schema-discovery)
+- [26.0.0 release notes for Schema auto-discovery](https://github.com/apache/druid/releases#26.0.0-highlights-auto-type-column-schema-%28experimental%29-schema-auto-discovery-%28experimental%29).
+- [26.0.0 release notes for array column types](https://github.com/apache/druid/releases#26.0.0-highlights-auto-type-column-schema-%28experimental%29).
+
+## Smart segment loading
+
+The Coordinator is now more stable and user-friendly and has the following behavior updates:
+
+- New segments that are underreplicated are now prioritized when you use the new `smartSegmentLoading` mode, which is enabled by default. Previously, it prioritized segments equally.
+- Items in segment load queues can be prioritized or canceled, improving reaction time for changes in the cluster and segment assignment decisions.
+- Leadership changes have less impact now, and the Coordinator doesn't get stuck even if re-election happens while a Coordinator run is in progress.
+
+Lastly, the `cost` balancer strategy performs much better now and is capable of moving more segments in a single Coordinator run. These improvements were made by borrowing ideas from the `cachingCost` strategy.
+
+For more information, see the following:
+
+- [Upgrade note for config changes related to smart segment loading](#segment-loading-config-changes)
+- [New coordinator metrics](#new-coordinator-metrics)
+- [Smart segment loading documentation](https://druid.apache.org/docs/latest/configuration/index.html#smart-segment-loading)
+
+[#13197](https://github.com/apache/druid/pull/13197) [#14385](https://github.com/apache/druid/pull/14385) [#14484](https://github.com/apache/druid/pull/14484)
+
+### Guardrail for subquery results
+
+Users can now add a guardrail to prevent subquery’s results from exceeding the set number of bytes by setting `druid.server.http.maxSubqueryRows` in the Broker's config or `maxSubqueryRows` in the query context. This guardrail is recommended over row-based limiting.
+
+This feature is experimental for now and defaults back to row-based limiting in case it fails to get the accurate size of the results consumed by the query.
+
+[#13952](https://github.com/apache/druid/pull/13952)
+
+### Added a new OSHI system monitor
+
+Added a new OSHI system monitor (`OshiSysMonitor`) to replace `SysMonitor`. The new monitor has a wider support for different machine architectures including ARM instances. We recommend switching to the new monitor. `SysMonitor` is now deprecated and will be removed in future releases.
+
+[#14359](https://github.com/apache/druid/pull/14359)
+
+
+## Java 17 support
+
+Druid now fully supports Java 17.
+
+[#14384](https://github.com/apache/druid/pull/14384)
+
+## Hadoop 2 deprecated
+
+Support for Hadoop 2 is now deprecated. It will be removed in a future release.
Review Comment:
```suggestion
Many of the important dependent libraries that Druid uses no longer supports Hadoop 2. In order for us to stay current and have pathways to mitigate security vulnerabilities, the community has decided to deprecate support for Hadoop 2.x releases starting this release. Starting Druid 28.x, Hadoop 3.x is the only supported Hadoop version.
Please consider migrating to MSQ based ingestion, native ingestion if you are using Hadoop 2.x for ingestion today. If migrating to Druid ingestion is not a possibility, you need to plan to upgrade your Hadoop infrastructure before upgrading to the next Druid release.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] DO NOT MERGE - 27.0.0 WIP release notes (druid)
Posted by "AmatyaAvadhanula (via GitHub)" <gi...@apache.org>.
AmatyaAvadhanula commented on code in PR #14600:
URL: https://github.com/apache/druid/pull/14600#discussion_r1288152398
##########
docs/do-not-merge.md:
##########
@@ -293,8 +293,8 @@ TBD for name change
|Metric|Description|Dimensions|Normal value|
|------|-----------|-----------|----------|
-|`segment/serverview/sync/healthy`|Sync status of the Coordinator/Broker with a segment-loading server such as a Historical or Peon. Emitted by the Coordinator and Brokers only when HTTP-based server view is enabled. This metric can be used in conjunction with `segment/serverview/sync/unstableTime` to debug slow startup of the Coordinator.|`server`, `tier`|1 for fully synced servers, 0 otherwise|
-|`segment/serverview/sync/unstableTime`|Time in milliseconds for which the Coordinator/Broker has been failing to sync with a segment-loading server. Emitted by the Coordinator and Brokers only when HTTP-based server view is enabled.|`server`, `tier`|Not emitted for synced servers.|
+|`serverview/sync/healthy`|Sync status of the Coordinator/Broker with a segment-loading server such as a Historical or Peon. Emitted by the Coordinator and Brokers only when HTTP-based server view is enabled. This metric can be used in conjunction with `segment/serverview/sync/unstableTime` to debug slow startup of the Coordinator.|`server`, `tier`|1 for fully synced servers, 0 otherwise|
Review Comment:
segment/serverview/sync/unstableTime -> serverview/sync/unstableTime
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] DO NOT MERGE - 27.0.0 WIP release notes (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #14600:
URL: https://github.com/apache/druid/pull/14600#discussion_r1275735221
##########
docs/do-not-merge.md:
##########
@@ -0,0 +1,377 @@
+<!--Intentionally, there's no Apache license so that the GHA fails it. This file is not meant to be merged.
+
+- https://github.com/apache/druid/pull/14266 - we removed input source security from 26 (https://github.com/apache/druid/pull/14003). Should we not include this in 27 release notes?
+
+-->
+
+Apache Druid 27.0.0 contains over $NUMBER_FEATURES new features, bug fixes, performance enhancements, documentation improvements, and additional test coverage from $NUMBER_OF_CONTRIBUTORS contributors.
+
+[See the complete set of changes for additional details]($LINK_TO_RELEASE_MILESTONE).
+
+Review the upgrade notes and incompatible changes before you upgrade to Druid 27.0.0.
+
+# Highlights
+
+<!-- HIGHLIGHTS H2. FOR EACH MAJOR FEATURE FOR THE RELEASE -->
+
+## Query from deep storage
+
+TBD
+
+### New statements API
+
+Added a new API /druid/v2/sql/statements/ which allows users to fetch results in an asynchronous manner.
+
+[14416](https://github.com/apache/druid/pull/14416)
+
+### New and updated fields in API to get results
+
+The API response to get results now returns a `pages` field containing information on each page of the results status.
+The `numRows` and `sizeInBytes` fields are renamed to `numTotalRows` and `totalSizeInBytes`, respectively.
+
+[14512](https://github.com/apache/druid/pull/14512)
+
+
+### Durable storage for results
+
+MSQ can now write SELECT query results to durable storage. To do so, set the context flag `selectDestination:DURABLE_STORAGE` while issuing SELECT queries to MSQ.
+
+[14527](https://github.com/apache/druid/pull/14527)
+
+## Java 17 support
+
+Druid now fully supports Java 17.
+[14384](https://github.com/apache/druid/pull/14384)
+
+## Array column types
+
+GA + TBD
+
+## Schema auto-discovery
+
+GA + TBD
+
+## Smart segment loading
+
+[13197](https://github.com/apache/druid/pull/13197)
+
+## Hadoop 2 deprecated
+
+Hadoop 2 is deprecated. It will be removed in a future release.
+
+# Additional features and improvements
+
+## MSQ task engine
+
+### `maxInputBytesPerWorker` context parameter
+
+The context parameter now denotes the estimated weighted size (in bytes) of the input to split on. The MSQ task engine now takes into account the input format and compression format instead of the actual file size reported by the file system.
+
+The default value for the context parameter has also been changed. It is now 512 MiB (previously 10 GiB).
+
+[14307](https://github.com/apache/druid/pull/14307)
+
+### Improved query planning behavior
+
+Druid now fails query planning if a CLUSTERED BY column contains descending order.
+Previously, queries would successfully plan if any CLUSTERED BY columns contained descending order.
+
+The MSQ fault, `InsertCannotOrderByDescending`, is deprecated. An INSERT or REPLACE query containing a CLUSTERED BY expression cannot be in descending order. Druid's segment generation code only supports ascending order. Instead of the fault, Druid now throws a query `ValidationException`.
+
+[14436](https://github.com/apache/druid/pull/14436)
+
+### SELECT query results
+
+SELECT queries executed using MSQ now generate only a subset of the results in the query reports.
+To fetch the complete result set, run the query using the native engine.
+
+[14370](https://github.com/apache/druid/pull/14370)
+
+
+### Other MSQ improvements
+
+- The same aggregator can now have two output names [14367](https://github.com/apache/druid/pull/14367)
+- Changed the default `clusterStatisticsMergeMode` to SEQUENTIAL (#14310)[https://github.com/apache/druid/pull/14310]
+- Added a query context parameter `MultiStageQueryContext` to determine whether the result of an MSQ select query is limited (#14476)[https://github.com/apache/druid/pull/14476]
+- Enabled using functions as inputs for `index` and `length` parameters (#14480)[https://github.com/apache/druid/pull/14480]
+
+## Ingestion
+
+### New property for task completion updates
+
+The new property `druid.indexer.queue.taskCompleteHandlerNumThreads` controls the number of threads used by the Overlord `TaskQueue` to handle task completion updates received from the workers.
+
+The following metrics have been added:
+* `task/status/queue/count`: Monitors the number of queued items
+* `task/status/updated/count`: Monitors the number of processed items
+
+[14533](https://github.com/apache/druid/pull/14533)
+
+### Improved response to max_allowed_packet limit
+
+If the Overlord fails to insert a task into the metadata because of a payload that exceeds the `max_allowed_packet` limit, the response now returns `400 Bad request` instead of `500 Internal server error`. This prevents an `index_parallel` task from retrying the insertion of a bad sub-task indefinitely and causes it to fail immediately.
+
+[14271](https://github.com/apache/druid/pull/14271)
+
+### Improved handling of mixed type arrays
+
+Druid now handles mixed type arrays such as `[["a", "b", "c"], {"x": 123}]` as `ARRAY<COMPLEX<json>` rather than throwing an incompatible type exception.
+
+[14438](https://github.com/apache/druid/pull/14438)
+
+### Other ingestion improvements
+
+* A negative streaming ingestion lag is no longer emitted as a result of stale offsets. [14292](https://github.com/apache/druid/pull/14292)
+* Removed double synchronization on simple map operations in Kubernetes task runner. [14435](https://github.com/apache/druid/pull/14435)
+* Kubernetes overlord extension now cleans up the job if the task pod fails to come up in time. [14425](https://github.com/apache/druid/pull/14425)
+
+## Querying
+
+### New function for regular expression replacement
+
+The new function `REGEXP_REPLACE` allows you to replace all instances of a pattern with a replacement string.
+
+[14460](https://github.com/apache/druid/pull/14460)
+
+### Query results directory
+
+Druid now supports a `query-results` directory in durable storage to store query results after the task finishes. The auto cleaner does not remove this directory unless the task ID is not known to the Overlord.
+
+[14446](https://github.com/apache/druid/pull/14446)
+
+### Limit for subquery results by memory usage
+
+Users can now add a guardrail to prevent subquery’s results from exceeding the set number of bytes by setting `druid.server.http.maxSubqueryRows` in the Broker's config or `maxSubqueryRows` in the query context. This feature is experimental for now and would default back to row-based limiting in case it fails to get the accurate size of the results consumed by the query.
+
+[13952](https://github.com/apache/druid/pull/13952)
+
+### HLL and Theta sketch estimates
+
+You can now use `HLL_SKETCH_ESTIMATE` and `THETA_SKETCH_ESTIMATE` as expressions. These estimates work on sketch columns and have the same behavior as `postAggs`.
+
+[14312](https://github.com/apache/druid/pull/14312)
+
+### String encoding parameter for HLL sketches
+
+HLL sketches now accept a new `stringEncoding` parameter to build sketches with UTF-8 bytes.
+
+[11201](https://github.com/apache/druid/pull/11201)
+
+### EARLIEST_BY and LATEST_BY signatures
+
+Updated EARLIEST_BY and LATEST_BY function signatures as follows:
+
+* Changed `EARLIEST(expr, timeColumn)` to `EARLIEST_BY(expr, timeColumn)`
+* Changed `LATEST(expr, timeColumn)` to `LATEST_BY(expr, timeColumn)`
+
+[14352](https://github.com/apache/druid/pull/14352)
+
+
+### Exposed Druid functions
+
+Exposed information about Druid functions in the `INFORMATION_SCHEMA.ROUTINES` table. The table contains the following columns:
+
+* ROUTINE_CATALOG: Name of the database. Always set to druid.
+* ROUTINE_SCHEMA: Name of the schema. Always set to INFORMATION_SCHEMA.
+* ROUTINE_NAME: Name of the function or operator. For example: APPROX_COUNT_DISTINCT_DS_THETA.
+* ROUTINE_TYPE: Type of routine. Always set to FUNCTION.
+* IS_AGGREGATOR: Determines if the routine is an aggregator. Returns YES for aggregator functions; NO for scalar functions.
+* SIGNATURES: Possible signatures for the routine as a string.
+
+[14378](https://github.com/apache/druid/pull/14378)
+
+### New Broker configuration for SQL schema migrations
+
+A new broker configuration, `druid.sql.planner.metadataColumnTypeMergePolicy`, adds configurable modes to how column types are computed for the SQL table schema when faced with differences between segments.
+
+The new `leastRestrictive` mode chooses the most appropriate type that data across all segments can best be coerced into and is the default behavior. This is a subtle behavior change around when segment driven schema migrations will take effect for the SQL schema:
+
+- With `leastRestrictive` mode, the schema only updates once all segments are reindexed to the new type.
+- With `latestInterval` mode, the SQL schema gets updated as soon as the first job with the new schema publishes segments in the latest time interval of the data.
+
+`leastRestrictive` can have better query time behavior and eliminates some query time errors that can occur when using `latestInterval`.
+
+[14319](https://github.com/apache/druid/pull/14319)
+
+### EXPLAIN PLAN improvements
+
+The EXPLAIN PLAN result includes a new column `ATTRIBUTES` that describes the attributes of a query.
+See [SQL query translation](docs/querying/sql-translation.md) for more information and examples.
+
+[14391](https://github.com/apache/druid/pull/14391)
+
+
+## Metrics and monitoring
+
+### Added a new SysMonitorOshi monitor
+
+Added a new monitor `SysMonitorOshi` to replace `SysMonitor`. The new monitor has a wider support for different machine architectures including ARM instances. We recommend switching to `SysMonitorOshi` as `SysMonitor` is now deprecated and will be removed in future releases.
+
+[14359](https://github.com/apache/druid/pull/14359)
+
+### Monitor for Overlord and Coordinator service health
+
+Added a new monitor `ServiceStatusMonitor` to monitor the service health of the Overlord and Coordinator.
+
+[14443](https://github.com/apache/druid/pull/14443)
+
+### New segment metrics
+
+The following metrics are now available for Brokers:
+
+|Metric|Description|Dimensions |Normal value|
+|------|-----------|------------|
+|`segment/metadatacache/refresh/count`|Number of segments to refresh in broker segment metadata cache. Emitted once per refresh per datasource.|`dataSource`| |
+|`segment/metadatacache/refresh/time`|Time taken to refresh segments in broker segment metadata cache. Emitted once per refresh per datasource.|`dataSource`| |
+| `segment/loadQueue/assigned` | Number of segments assigned for load or drop to the load queue of a server. |`dataSource`,`server` | Varies |
+| `segment/loadQueue/success` |Number of segment assignments that completed successfully.|`dataSource`, `server`|Varies|
+| `segment/loadQueue/cancelled` |Number of segment assignments that were canceled before completion. |`dataSource`,`server` | 0 |
+| `segment/loadQueue/failed` |Number of segment assignments that failed to complete.|`dataSource`, `server`|0|
+| `segment/assigned/broadcast` | |
Review Comment:
Yes, @317brian , these metrics were originally a part of the PR but were later removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] DO NOT MERGE - 27.0.0 WIP release notes (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #14600:
URL: https://github.com/apache/druid/pull/14600#discussion_r1275735766
##########
docs/do-not-merge.md:
##########
@@ -0,0 +1,377 @@
+<!--Intentionally, there's no Apache license so that the GHA fails it. This file is not meant to be merged.
+
+- https://github.com/apache/druid/pull/14266 - we removed input source security from 26 (https://github.com/apache/druid/pull/14003). Should we not include this in 27 release notes?
+
+-->
+
+Apache Druid 27.0.0 contains over $NUMBER_FEATURES new features, bug fixes, performance enhancements, documentation improvements, and additional test coverage from $NUMBER_OF_CONTRIBUTORS contributors.
+
+[See the complete set of changes for additional details]($LINK_TO_RELEASE_MILESTONE).
+
+Review the upgrade notes and incompatible changes before you upgrade to Druid 27.0.0.
+
+# Highlights
+
+<!-- HIGHLIGHTS H2. FOR EACH MAJOR FEATURE FOR THE RELEASE -->
+
+## Query from deep storage (experimental)
+
+Druid now supports querying segments that are stored only in deep storage. When you query from deep storage, you increase the data available for queries without necessarily having to scale your Historical processes to accommodate more data. To take advantage of the potential savings, make sure you configure your load rules to not load all your data onto Historical processes. Then, configure a corresponding drop rule to drop those segments.
+
+Note that at least one segment of a datasource must be loaded onto a Historical process so that Druid can plan the query. It can be any segment though.
+
+As part of this feature, there are new API endpoints available to interact with the query:
+
+- Submit queries using `POST /sql/statements`. The payload is the same as any other Druid SQL query with additional context parameter available.
+ - The required `executionMode: ASYNC` configures Druid to return the results asynchronously. Results are only available after the query completes.
+ - The optional context parameter `selectDestination:DURABLE_STORAGE` configures Druid to write the query results to durable storage.
+
+- Get the status of a query with `GET /sq/statements/:queryID`.
+
+- Get query results with `/sql/statements/:queryID`
+
+[14416](https://github.com/apache/druid/pull/14416)
+[14512](https://github.com/apache/druid/pull/14512)
+[14527](https://github.com/apache/druid/pull/14527)
+
+## Java 17 support
+
+Druid now fully supports Java 17.
+[14384](https://github.com/apache/druid/pull/14384)
+
+## Array column types
+
+Array column types are now generally available.
+
+For more information about this feature, see the [26.0.0 release notes for array column types](https://github.com/apache/druid/releases#26.0.0-highlights-auto-type-column-schema-%28experimental%29).
+
+## Schema auto-discovery
+
+Schema auto-discovery is now generally available.
+
+For more information about this feature, see the [26.0.0 release notes for Schema auto-discovery](https://github.com/apache/druid/releases#26.0.0-highlights-auto-type-column-schema-%28experimental%29-schema-auto-discovery-%28experimental%29).
+
+## Smart segment loading
+
+Revamped Coordinator to make it more stable and user-friendly. This is accompanied by several bug fixes, logging and metric improvements, and a whole new range of capabilities.
+
+Coordinator now supports a `smartSegmentLoading` mode, which is enabled by default. When enabled, users don't need to specify any of the following dynamic configs as they would be ignored by the coordinator. Instead, the coordinator computes the optimal values of the following configs at run time to best utilize coordinator runs:
+
+* `maxSegmentsInNodeLoadingQueue`
+* `maxSegmentsToMove`
+* `replicationThrottleLimit`
+* `useRoundRobinSegmentAssignment`
+* `useBatchedSegmentSampler`
+* `emitBalancingStats`
+
+These configs are now deprecated and will be removed in subsequent releases.
+
+Coordinator is now capable of prioritization and cancellation of items in segment load queues. This means that the coordinator now reacts faster to changes in the cluster and makes better segment assignment decisions.
+
+As part of this change, new metrics are available. For more information, see [New segment metrics](#new-segment-metrics).
+
+[13197](https://github.com/apache/druid/pull/13197)
+
+## Hadoop 2 deprecated
+
+Support for Hadoop 2 is now deprecated. It will be removed in a future release.
+
+# Additional features and improvements
+
+## MSQ task engine
+
+### `maxInputBytesPerWorker` context parameter
+
+The context parameter now denotes the estimated weighted size (in bytes) of the input to split on. The MSQ task engine now takes into account the input format and compression format instead of the actual file size reported by the file system.
+
+The default value for the context parameter has also been changed. It is now 512 MiB (previously 10 GiB).
+
+[14307](https://github.com/apache/druid/pull/14307)
+
+### Improved query planning behavior
+
+Druid now fails query planning if a CLUSTERED BY column contains descending order.
+Previously, queries would successfully plan if any CLUSTERED BY columns contained descending order.
+
+The MSQ fault, `InsertCannotOrderByDescending`, is deprecated. An INSERT or REPLACE query containing a CLUSTERED BY expression cannot be in descending order. Druid's segment generation code only supports ascending order. Instead of the fault, Druid now throws a query `ValidationException`.
+
+[14436](https://github.com/apache/druid/pull/14436)
+
+### SELECT query results
+
+SELECT queries executed using MSQ now generate only a subset of the results in the query reports.
+To fetch the complete result set, run the query using the native engine.
+
+[14370](https://github.com/apache/druid/pull/14370)
+
+
+### Other MSQ improvements
+
+- The same aggregator can now have two output names [14367](https://github.com/apache/druid/pull/14367)
+- Changed the default `clusterStatisticsMergeMode` to SEQUENTIAL (#14310)[https://github.com/apache/druid/pull/14310]
+- Added a query context parameter `MultiStageQueryContext` to determine whether the result of an MSQ select query is limited (#14476)[https://github.com/apache/druid/pull/14476]
+- Enabled using functions as inputs for `index` and `length` parameters (#14480)[https://github.com/apache/druid/pull/14480]
+
+## Ingestion
+
+### New property for task completion updates
+
+The new property `druid.indexer.queue.taskCompleteHandlerNumThreads` controls the number of threads used by the Overlord `TaskQueue` to handle task completion updates received from the workers.
+
+The following metrics have been added:
+* `task/status/queue/count`: Monitors the number of queued items
+* `task/status/updated/count`: Monitors the number of processed items
+
+[14533](https://github.com/apache/druid/pull/14533)
+
+### Improved response to max_allowed_packet limit
+
+If the Overlord fails to insert a task into the metadata because of a payload that exceeds the `max_allowed_packet` limit, the response now returns `400 Bad request` instead of `500 Internal server error`. This prevents an `index_parallel` task from retrying the insertion of a bad sub-task indefinitely and causes it to fail immediately.
+
+[14271](https://github.com/apache/druid/pull/14271)
+
+### Improved handling of mixed type arrays
+
+Druid now handles mixed type arrays such as `[["a", "b", "c"], {"x": 123}]` as `ARRAY<COMPLEX<json>` rather than throwing an incompatible type exception.
+
+[14438](https://github.com/apache/druid/pull/14438)
+
+### Other ingestion improvements
+
+* A negative streaming ingestion lag is no longer emitted as a result of stale offsets. [14292](https://github.com/apache/druid/pull/14292)
+* Removed double synchronization on simple map operations in Kubernetes task runner. [14435](https://github.com/apache/druid/pull/14435)
+* Kubernetes overlord extension now cleans up the job if the task pod fails to come up in time. [14425](https://github.com/apache/druid/pull/14425)
+
+## Querying
+
+### New function for regular expression replacement
+
+The new function `REGEXP_REPLACE` allows you to replace all instances of a pattern with a replacement string.
+
+[14460](https://github.com/apache/druid/pull/14460)
+
+### Query results directory
+
+Druid now supports a `query-results` directory in durable storage to store query results after the task finishes. The auto cleaner does not remove this directory unless the task ID is not known to the Overlord.
+
+[14446](https://github.com/apache/druid/pull/14446)
+
+### Limit for subquery results by memory usage
+
+Users can now add a guardrail to prevent subquery’s results from exceeding the set number of bytes by setting `druid.server.http.maxSubqueryRows` in the Broker's config or `maxSubqueryRows` in the query context. This feature is experimental for now and would default back to row-based limiting in case it fails to get the accurate size of the results consumed by the query.
+
+[13952](https://github.com/apache/druid/pull/13952)
+
+### HLL and Theta sketch estimates
+
+You can now use `HLL_SKETCH_ESTIMATE` and `THETA_SKETCH_ESTIMATE` as expressions. These estimates work on sketch columns and have the same behavior as `postAggs`.
+
+[14312](https://github.com/apache/druid/pull/14312)
+
+### String encoding parameter for HLL sketches
+
+HLL sketches now accept a new `stringEncoding` parameter to build sketches with UTF-8 bytes.
+
+[11201](https://github.com/apache/druid/pull/11201)
+
+### EARLIEST_BY and LATEST_BY signatures
+
+Updated EARLIEST_BY and LATEST_BY function signatures as follows:
+
+* Changed `EARLIEST(expr, timeColumn)` to `EARLIEST_BY(expr, timeColumn)`
+* Changed `LATEST(expr, timeColumn)` to `LATEST_BY(expr, timeColumn)`
+
+[14352](https://github.com/apache/druid/pull/14352)
+
+
+### Exposed Druid functions
+
+Exposed information about Druid functions in the `INFORMATION_SCHEMA.ROUTINES` table. The table contains the following columns:
+
+* ROUTINE_CATALOG: Name of the database. Always set to druid.
+* ROUTINE_SCHEMA: Name of the schema. Always set to INFORMATION_SCHEMA.
+* ROUTINE_NAME: Name of the function or operator. For example: APPROX_COUNT_DISTINCT_DS_THETA.
+* ROUTINE_TYPE: Type of routine. Always set to FUNCTION.
+* IS_AGGREGATOR: Determines if the routine is an aggregator. Returns YES for aggregator functions; NO for scalar functions.
+* SIGNATURES: Possible signatures for the routine as a string.
+
+[14378](https://github.com/apache/druid/pull/14378)
+
+### New Broker configuration for SQL schema migrations
+
+A new broker configuration, `druid.sql.planner.metadataColumnTypeMergePolicy`, adds configurable modes to how column types are computed for the SQL table schema when faced with differences between segments.
+
+The new `leastRestrictive` mode chooses the most appropriate type that data across all segments can best be coerced into and is the default behavior. This is a subtle behavior change around when segment driven schema migrations will take effect for the SQL schema:
+
+- With `leastRestrictive` mode, the schema only updates once all segments are reindexed to the new type.
+- With `latestInterval` mode, the SQL schema gets updated as soon as the first job with the new schema publishes segments in the latest time interval of the data.
+
+`leastRestrictive` can have better query time behavior and eliminates some query time errors that can occur when using `latestInterval`.
+
+[14319](https://github.com/apache/druid/pull/14319)
+
+### EXPLAIN PLAN improvements
+
+The EXPLAIN PLAN result includes a new column `ATTRIBUTES` that describes the attributes of a query.
+See [SQL query translation](docs/querying/sql-translation.md) for more information and examples.
+
+[14391](https://github.com/apache/druid/pull/14391)
+
+
+## Metrics and monitoring
+
+### Added a new SysMonitorOshi monitor
+
+Added a new monitor `SysMonitorOshi` to replace `SysMonitor`. The new monitor has a wider support for different machine architectures including ARM instances. We recommend switching to `SysMonitorOshi` as `SysMonitor` is now deprecated and will be removed in future releases.
+
+[14359](https://github.com/apache/druid/pull/14359)
+
+### Monitor for Overlord and Coordinator service health
+
+Added a new monitor `ServiceStatusMonitor` to monitor the service health of the Overlord and Coordinator.
+
+[14443](https://github.com/apache/druid/pull/14443)
+
+### New segment metrics
+
+The following metrics are now available for Brokers:
+
+|Metric|Description|Dimensions |Normal value|
+|------|-----------|------------|
+|`segment/metadatacache/refresh/count`|Number of segments to refresh in broker segment metadata cache. Emitted once per refresh per datasource.|`dataSource`| |
+|`segment/metadatacache/refresh/time`|Time taken to refresh segments in broker segment metadata cache. Emitted once per refresh per datasource.|`dataSource`| |
+| `segment/loadQueue/assigned` | Number of segments assigned for load or drop to the load queue of a server. |`dataSource`,`server` | Varies |
+| `segment/loadQueue/success` |Number of segment assignments that completed successfully.|`dataSource`, `server`|Varies|
+| `segment/loadQueue/cancelled` |Number of segment assignments that were canceled before completion. |`dataSource`,`server` | 0 |
+| `segment/loadQueue/failed` |Number of segment assignments that failed to complete.|`dataSource`, `server`|0|
+| `segment/assigned/broadcast` | |
+| `segment/dropped/broadcast` | |
Review Comment:
```suggestion
```
##########
docs/do-not-merge.md:
##########
@@ -0,0 +1,377 @@
+<!--Intentionally, there's no Apache license so that the GHA fails it. This file is not meant to be merged.
+
+- https://github.com/apache/druid/pull/14266 - we removed input source security from 26 (https://github.com/apache/druid/pull/14003). Should we not include this in 27 release notes?
+
+-->
+
+Apache Druid 27.0.0 contains over $NUMBER_FEATURES new features, bug fixes, performance enhancements, documentation improvements, and additional test coverage from $NUMBER_OF_CONTRIBUTORS contributors.
+
+[See the complete set of changes for additional details]($LINK_TO_RELEASE_MILESTONE).
+
+Review the upgrade notes and incompatible changes before you upgrade to Druid 27.0.0.
+
+# Highlights
+
+<!-- HIGHLIGHTS H2. FOR EACH MAJOR FEATURE FOR THE RELEASE -->
+
+## Query from deep storage (experimental)
+
+Druid now supports querying segments that are stored only in deep storage. When you query from deep storage, you increase the data available for queries without necessarily having to scale your Historical processes to accommodate more data. To take advantage of the potential savings, make sure you configure your load rules to not load all your data onto Historical processes. Then, configure a corresponding drop rule to drop those segments.
+
+Note that at least one segment of a datasource must be loaded onto a Historical process so that Druid can plan the query. It can be any segment though.
+
+As part of this feature, there are new API endpoints available to interact with the query:
+
+- Submit queries using `POST /sql/statements`. The payload is the same as any other Druid SQL query with additional context parameter available.
+ - The required `executionMode: ASYNC` configures Druid to return the results asynchronously. Results are only available after the query completes.
+ - The optional context parameter `selectDestination:DURABLE_STORAGE` configures Druid to write the query results to durable storage.
+
+- Get the status of a query with `GET /sq/statements/:queryID`.
+
+- Get query results with `/sql/statements/:queryID`
+
+[14416](https://github.com/apache/druid/pull/14416)
+[14512](https://github.com/apache/druid/pull/14512)
+[14527](https://github.com/apache/druid/pull/14527)
+
+## Java 17 support
+
+Druid now fully supports Java 17.
+[14384](https://github.com/apache/druid/pull/14384)
+
+## Array column types
+
+Array column types are now generally available.
+
+For more information about this feature, see the [26.0.0 release notes for array column types](https://github.com/apache/druid/releases#26.0.0-highlights-auto-type-column-schema-%28experimental%29).
+
+## Schema auto-discovery
+
+Schema auto-discovery is now generally available.
+
+For more information about this feature, see the [26.0.0 release notes for Schema auto-discovery](https://github.com/apache/druid/releases#26.0.0-highlights-auto-type-column-schema-%28experimental%29-schema-auto-discovery-%28experimental%29).
+
+## Smart segment loading
+
+Revamped Coordinator to make it more stable and user-friendly. This is accompanied by several bug fixes, logging and metric improvements, and a whole new range of capabilities.
+
+Coordinator now supports a `smartSegmentLoading` mode, which is enabled by default. When enabled, users don't need to specify any of the following dynamic configs as they would be ignored by the coordinator. Instead, the coordinator computes the optimal values of the following configs at run time to best utilize coordinator runs:
+
+* `maxSegmentsInNodeLoadingQueue`
+* `maxSegmentsToMove`
+* `replicationThrottleLimit`
+* `useRoundRobinSegmentAssignment`
+* `useBatchedSegmentSampler`
+* `emitBalancingStats`
+
+These configs are now deprecated and will be removed in subsequent releases.
+
+Coordinator is now capable of prioritization and cancellation of items in segment load queues. This means that the coordinator now reacts faster to changes in the cluster and makes better segment assignment decisions.
+
+As part of this change, new metrics are available. For more information, see [New segment metrics](#new-segment-metrics).
+
+[13197](https://github.com/apache/druid/pull/13197)
+
+## Hadoop 2 deprecated
+
+Support for Hadoop 2 is now deprecated. It will be removed in a future release.
+
+# Additional features and improvements
+
+## MSQ task engine
+
+### `maxInputBytesPerWorker` context parameter
+
+The context parameter now denotes the estimated weighted size (in bytes) of the input to split on. The MSQ task engine now takes into account the input format and compression format instead of the actual file size reported by the file system.
+
+The default value for the context parameter has also been changed. It is now 512 MiB (previously 10 GiB).
+
+[14307](https://github.com/apache/druid/pull/14307)
+
+### Improved query planning behavior
+
+Druid now fails query planning if a CLUSTERED BY column contains descending order.
+Previously, queries would successfully plan if any CLUSTERED BY columns contained descending order.
+
+The MSQ fault, `InsertCannotOrderByDescending`, is deprecated. An INSERT or REPLACE query containing a CLUSTERED BY expression cannot be in descending order. Druid's segment generation code only supports ascending order. Instead of the fault, Druid now throws a query `ValidationException`.
+
+[14436](https://github.com/apache/druid/pull/14436)
+
+### SELECT query results
+
+SELECT queries executed using MSQ now generate only a subset of the results in the query reports.
+To fetch the complete result set, run the query using the native engine.
+
+[14370](https://github.com/apache/druid/pull/14370)
+
+
+### Other MSQ improvements
+
+- The same aggregator can now have two output names [14367](https://github.com/apache/druid/pull/14367)
+- Changed the default `clusterStatisticsMergeMode` to SEQUENTIAL (#14310)[https://github.com/apache/druid/pull/14310]
+- Added a query context parameter `MultiStageQueryContext` to determine whether the result of an MSQ select query is limited (#14476)[https://github.com/apache/druid/pull/14476]
+- Enabled using functions as inputs for `index` and `length` parameters (#14480)[https://github.com/apache/druid/pull/14480]
+
+## Ingestion
+
+### New property for task completion updates
+
+The new property `druid.indexer.queue.taskCompleteHandlerNumThreads` controls the number of threads used by the Overlord `TaskQueue` to handle task completion updates received from the workers.
+
+The following metrics have been added:
+* `task/status/queue/count`: Monitors the number of queued items
+* `task/status/updated/count`: Monitors the number of processed items
+
+[14533](https://github.com/apache/druid/pull/14533)
+
+### Improved response to max_allowed_packet limit
+
+If the Overlord fails to insert a task into the metadata because of a payload that exceeds the `max_allowed_packet` limit, the response now returns `400 Bad request` instead of `500 Internal server error`. This prevents an `index_parallel` task from retrying the insertion of a bad sub-task indefinitely and causes it to fail immediately.
+
+[14271](https://github.com/apache/druid/pull/14271)
+
+### Improved handling of mixed type arrays
+
+Druid now handles mixed type arrays such as `[["a", "b", "c"], {"x": 123}]` as `ARRAY<COMPLEX<json>` rather than throwing an incompatible type exception.
+
+[14438](https://github.com/apache/druid/pull/14438)
+
+### Other ingestion improvements
+
+* A negative streaming ingestion lag is no longer emitted as a result of stale offsets. [14292](https://github.com/apache/druid/pull/14292)
+* Removed double synchronization on simple map operations in Kubernetes task runner. [14435](https://github.com/apache/druid/pull/14435)
+* Kubernetes overlord extension now cleans up the job if the task pod fails to come up in time. [14425](https://github.com/apache/druid/pull/14425)
+
+## Querying
+
+### New function for regular expression replacement
+
+The new function `REGEXP_REPLACE` allows you to replace all instances of a pattern with a replacement string.
+
+[14460](https://github.com/apache/druid/pull/14460)
+
+### Query results directory
+
+Druid now supports a `query-results` directory in durable storage to store query results after the task finishes. The auto cleaner does not remove this directory unless the task ID is not known to the Overlord.
+
+[14446](https://github.com/apache/druid/pull/14446)
+
+### Limit for subquery results by memory usage
+
+Users can now add a guardrail to prevent subquery’s results from exceeding the set number of bytes by setting `druid.server.http.maxSubqueryRows` in the Broker's config or `maxSubqueryRows` in the query context. This feature is experimental for now and would default back to row-based limiting in case it fails to get the accurate size of the results consumed by the query.
+
+[13952](https://github.com/apache/druid/pull/13952)
+
+### HLL and Theta sketch estimates
+
+You can now use `HLL_SKETCH_ESTIMATE` and `THETA_SKETCH_ESTIMATE` as expressions. These estimates work on sketch columns and have the same behavior as `postAggs`.
+
+[14312](https://github.com/apache/druid/pull/14312)
+
+### String encoding parameter for HLL sketches
+
+HLL sketches now accept a new `stringEncoding` parameter to build sketches with UTF-8 bytes.
+
+[11201](https://github.com/apache/druid/pull/11201)
+
+### EARLIEST_BY and LATEST_BY signatures
+
+Updated EARLIEST_BY and LATEST_BY function signatures as follows:
+
+* Changed `EARLIEST(expr, timeColumn)` to `EARLIEST_BY(expr, timeColumn)`
+* Changed `LATEST(expr, timeColumn)` to `LATEST_BY(expr, timeColumn)`
+
+[14352](https://github.com/apache/druid/pull/14352)
+
+
+### Exposed Druid functions
+
+Exposed information about Druid functions in the `INFORMATION_SCHEMA.ROUTINES` table. The table contains the following columns:
+
+* ROUTINE_CATALOG: Name of the database. Always set to druid.
+* ROUTINE_SCHEMA: Name of the schema. Always set to INFORMATION_SCHEMA.
+* ROUTINE_NAME: Name of the function or operator. For example: APPROX_COUNT_DISTINCT_DS_THETA.
+* ROUTINE_TYPE: Type of routine. Always set to FUNCTION.
+* IS_AGGREGATOR: Determines if the routine is an aggregator. Returns YES for aggregator functions; NO for scalar functions.
+* SIGNATURES: Possible signatures for the routine as a string.
+
+[14378](https://github.com/apache/druid/pull/14378)
+
+### New Broker configuration for SQL schema migrations
+
+A new broker configuration, `druid.sql.planner.metadataColumnTypeMergePolicy`, adds configurable modes to how column types are computed for the SQL table schema when faced with differences between segments.
+
+The new `leastRestrictive` mode chooses the most appropriate type that data across all segments can best be coerced into and is the default behavior. This is a subtle behavior change around when segment driven schema migrations will take effect for the SQL schema:
+
+- With `leastRestrictive` mode, the schema only updates once all segments are reindexed to the new type.
+- With `latestInterval` mode, the SQL schema gets updated as soon as the first job with the new schema publishes segments in the latest time interval of the data.
+
+`leastRestrictive` can have better query time behavior and eliminates some query time errors that can occur when using `latestInterval`.
+
+[14319](https://github.com/apache/druid/pull/14319)
+
+### EXPLAIN PLAN improvements
+
+The EXPLAIN PLAN result includes a new column `ATTRIBUTES` that describes the attributes of a query.
+See [SQL query translation](docs/querying/sql-translation.md) for more information and examples.
+
+[14391](https://github.com/apache/druid/pull/14391)
+
+
+## Metrics and monitoring
+
+### Added a new SysMonitorOshi monitor
+
+Added a new monitor `SysMonitorOshi` to replace `SysMonitor`. The new monitor has a wider support for different machine architectures including ARM instances. We recommend switching to `SysMonitorOshi` as `SysMonitor` is now deprecated and will be removed in future releases.
+
+[14359](https://github.com/apache/druid/pull/14359)
+
+### Monitor for Overlord and Coordinator service health
+
+Added a new monitor `ServiceStatusMonitor` to monitor the service health of the Overlord and Coordinator.
+
+[14443](https://github.com/apache/druid/pull/14443)
+
+### New segment metrics
+
+The following metrics are now available for Brokers:
+
+|Metric|Description|Dimensions |Normal value|
+|------|-----------|------------|
+|`segment/metadatacache/refresh/count`|Number of segments to refresh in broker segment metadata cache. Emitted once per refresh per datasource.|`dataSource`| |
+|`segment/metadatacache/refresh/time`|Time taken to refresh segments in broker segment metadata cache. Emitted once per refresh per datasource.|`dataSource`| |
+| `segment/loadQueue/assigned` | Number of segments assigned for load or drop to the load queue of a server. |`dataSource`,`server` | Varies |
+| `segment/loadQueue/success` |Number of segment assignments that completed successfully.|`dataSource`, `server`|Varies|
+| `segment/loadQueue/cancelled` |Number of segment assignments that were canceled before completion. |`dataSource`,`server` | 0 |
+| `segment/loadQueue/failed` |Number of segment assignments that failed to complete.|`dataSource`, `server`|0|
+| `segment/assigned/broadcast` | |
Review Comment:
```suggestion
```
##########
docs/do-not-merge.md:
##########
@@ -0,0 +1,366 @@
+<!--Intentionally, there's no Apache license so that the GHA fails it. This file is not meant to be merged.
+
+- https://github.com/apache/druid/pull/14266 - we removed input source security from 26 (https://github.com/apache/druid/pull/14003). Should we not include this in 27 release notes?
+
+-->
+
+Apache Druid 27.0.0 contains over $NUMBER_FEATURES new features, bug fixes, performance enhancements, documentation improvements, and additional test coverage from $NUMBER_OF_CONTRIBUTORS contributors.
+
+[See the complete set of changes for additional details]($LINK_TO_RELEASE_MILESTONE).
+
+Review the upgrade notes and incompatible changes before you upgrade to Druid 27.0.0.
+
+# Highlights
+
+<!-- HIGHLIGHTS H2. FOR EACH MAJOR FEATURE FOR THE RELEASE -->
+
+## Query from deep storage
+
+TBD
+
+### New statements API
+
+Added a new API /druid/v2/sql/statements/ which allows users to fetch results in an asynchronous manner.
+
+[14416](https://github.com/apache/druid/pull/14416)
+
+### New and updated fields in API to get results
+
+The API response to get results now returns a `pages` field containing information on each page of the results status.
+The `numRows` and `sizeInBytes` fields are renamed to `numTotalRows` and `totalSizeInBytes`, respectively.
+
+[14512](https://github.com/apache/druid/pull/14512)
+
+
+### Durable storage for results
+
+MSQ can now write SELECT query results to durable storage. To do so, set the context flag `selectDestination:DURABLE_STORAGE` while issuing SELECT queries to MSQ.
+
+[14527](https://github.com/apache/druid/pull/14527)
+
+## Java 17 support
+
+Druid now fully supports Java 17.
+[14384](https://github.com/apache/druid/pull/14384)
+
+## Array column types
+
+GA + TBD
+
+## Schema auto-discovery
+
+GA + TBD
+
+## Smart segment loading
+
+[13197](https://github.com/apache/druid/pull/13197)
+
+## Hadoop 2 support dropped
+
+# Additional features and improvements
+
+## MSQ task engine
+
+### `maxInputBytesPerWorker` context parameter
+
+The context parameter now denotes the estimated weighted size (in bytes) of the input to split on. The MSQ task engine now takes into account the input format and compression format instead of the actual file size reported by the file system.
+
+The default value for the context parameter has also been changed. It is now 512 MiB (previously 10 GiB).
+
+[14307](https://github.com/apache/druid/pull/14307)
+
+### Improved query planning behavior
+
+Druid now fails query planning if a CLUSTERED BY column contains descending order.
+Previously, queries would successfully plan if any CLUSTERED BY columns contained descending order.
+
+The MSQ fault, `InsertCannotOrderByDescending`, is deprecated. An INSERT or REPLACE query containing a CLUSTERED BY expression cannot be in descending order. Druid's segment generation code only supports ascending order. Instead of the fault, Druid now throws a query `ValidationException`.
+
+[14436](https://github.com/apache/druid/pull/14436)
+
+### SELECT query results
+
+SELECT queries executed using MSQ now generate only a subset of the results in the query reports.
+To fetch the complete result set, run the query using the native engine.
+
+[14370](https://github.com/apache/druid/pull/14370)
+
+
+### Other MSQ improvements
+
+- The same aggregator can now have two output names [14367](https://github.com/apache/druid/pull/14367)
+- Changed the default `clusterStatisticsMergeMode` to SEQUENTIAL (#14310)[https://github.com/apache/druid/pull/14310]
+- Added a query context parameter `MultiStageQueryContext` to determine whether the result of an MSQ select query is limited (#14476)[https://github.com/apache/druid/pull/14476]
+- Enabled using functions as inputs for `index` and `length` parameters (#14480)[https://github.com/apache/druid/pull/14480]
+
+## Ingestion
+
+### New property for task completion updates
+
+The new property `druid.indexer.queue.taskCompleteHandlerNumThreads` controls the number of threads used by the Overlord `TaskQueue` to handle task completion updates received from the workers.
+
+The following metrics have been added:
+* `task/status/queue/count`: Monitors the number of queued items
+* `task/status/updated/count`: Monitors the number of processed items
+
+[14533](https://github.com/apache/druid/pull/14533)
+
+### Improved response to max_allowed_packet limit
+
+If the Overlord fails to insert a task into the metadata because of a payload that exceeds the `max_allowed_packet` limit, the response now returns `400 Bad request` instead of `500 Internal server error`. This prevents an `index_parallel` task from retrying the insertion of a bad sub-task indefinitely and causes it to fail immediately.
+
+[14271](https://github.com/apache/druid/pull/14271)
+
+### Improved handling of mixed type arrays
+
+Druid now handles mixed type arrays such as `[["a", "b", "c"], {"x": 123}]` as `ARRAY<COMPLEX<json>` rather than throwing an incompatible type exception.
+
+[14438](https://github.com/apache/druid/pull/14438)
+
+### Other ingestion improvements
+
+* A negative streaming ingestion lag is no longer emitted as a result of stale offsets. [14292](https://github.com/apache/druid/pull/14292)
+* Removed double synchronization on simple map operations in Kubernetes task runner. [14435](https://github.com/apache/druid/pull/14435)
+* Kubernetes overlord extension now cleans up the job if the task pod fails to come up in time. [14425](https://github.com/apache/druid/pull/14425)
+
+## Querying
+
+### New function for regular expression replacement
+
+The new function `REGEXP_REPLACE` allows you to replace all instances of a pattern with a replacement string.
+
+[14460](https://github.com/apache/druid/pull/14460)
+
+### Query results directory
+
+Druid now supports a `query-results` directory in durable storage to store query results after the task finishes. The auto cleaner does not remove this directory unless the task ID is not known to the Overlord.
+
+[14446](https://github.com/apache/druid/pull/14446)
+
+### Limit for subquery results by memory usage
+
+Users can now add a guardrail to prevent subquery’s results from exceeding the set number of bytes by setting `druid.server.http.maxSubqueryRows` in the Broker's config or `maxSubqueryRows` in the query context. This feature is experimental for now and would default back to row-based limiting in case it fails to get the accurate size of the results consumed by the query.
+
+[13952](https://github.com/apache/druid/pull/13952)
+
+### HLL and Theta sketch estimates
+
+You can now use HLL_SKETCH_ESTIMATE and THETA_SKETCH_ESTIMATE as expressions. These estimates work on sketch columns and have the same behavior as `postAggs`.
+
+[14312](https://github.com/apache/druid/pull/14312)
+
+### String encoding parameter for HLL sketches
+
+HLL sketches now accept a new `stringEncoding` parameter to build sketches with UTF-8 bytes.
+
+[11201](https://github.com/apache/druid/pull/11201)
+
+### EARLIEST_BY and LATEST_BY signatures
+
+Updated EARLIEST_BY and LATEST_BY function signatures as follows:
+
+* Changed `EARLIEST(expr, timeColumn)` to `EARLIEST_BY(expr, timeColumn)`
+* Changed `LATEST(expr, timeColumn)` to `LATEST_BY(expr, timeColumn)`
+
+[14352](https://github.com/apache/druid/pull/14352)
+
+
+### Exposed Druid functions
+
+Exposed information about Druid functions in the `INFORMATION_SCHEMA.ROUTINES` table. The table contains the following columns:
+
+* ROUTINE_CATALOG: Name of the database. Always set to druid.
+* ROUTINE_SCHEMA: Name of the schema. Always set to INFORMATION_SCHEMA.
+* ROUTINE_NAME: Name of the function or operator. For example: APPROX_COUNT_DISTINCT_DS_THETA.
+* ROUTINE_TYPE: Type of routine. Always set to FUNCTION.
+* IS_AGGREGATOR: Determines if the routine is an aggregator. Returns YES for aggregator functions; NO for scalar functions.
+* SIGNATURES: Possible signatures for the routine as a string.
+
+[14378](https://github.com/apache/druid/pull/14378)
+
+### New Broker configuration for SQL schema migrations
+
+A new broker configuration, `druid.sql.planner.metadataColumnTypeMergePolicy`, adds configurable modes to how column types are computed for the SQL table schema when faced with differences between segments.
+
+The new `leastRestrictive` mode chooses the most appropriate type that data across all segments can best be coerced into and is the default behavior. This is a subtle behavior change around when segment driven schema migrations will take effect for the SQL schema:
+
+- With `leastRestrictive` mode, the schema only updates once all segments are reindexed to the new type.
+- With `latestInterval` mode, the SQL schema gets updated as soon as the first job with the new schema publishes segments in the latest time interval of the data.
+
+`leastRestrictive` can have better query time behavior and eliminates some query time errors that can occur when using `latestInterval`.
+
+[14319](https://github.com/apache/druid/pull/14319)
+
+### EXPLAIN PLAN improvements
+
+The EXPLAIN PLAN result includes a new column `ATTRIBUTES` that describes the attributes of a query.
+See [SQL query translation](docs/querying/sql-translation.md) for more information and examples.
+
+[14391](https://github.com/apache/druid/pull/14391)
+
+
+## Metrics and monitoring
+
+### Added a new SysMonitorOshi monitor
+
+Added a new monitor `SysMonitorOshi` to replace `SysMonitor`. The new monitor has a wider support for different machine architectures including ARM instances. We recommend switching to `SysMonitorOshi` as `SysMonitor` is now deprecated and will be removed in future releases.
+
+[14359](https://github.com/apache/druid/pull/14359)
+
+### Monitor for Overlord and Coordinator service health
+
+Added a new monitor `ServiceStatusMonitor` to monitor the service health of the Overlord and Coordinator.
+
+[14443](https://github.com/apache/druid/pull/14443)
+
+### New segment metrics
+
+The following metrics are now available for Brokers:
+
+|Metric|Description|Normal value|
+|------|-----------|------------|
+|`segment/metadatacache/refresh/count`|Number of segments to refresh in broker segment metadata cache. Emitted once per refresh per datasource.|`dataSource`|
+|`segment/metadatacache/refresh/time`|Time taken to refresh segments in broker segment metadata cache. Emitted once per refresh per datasource.|`dataSource`|
+
+[14453](https://github.com/apache/druid/pull/14453)
+
+### New metrics for task completion updates
+
+The following metrics have been added:
+* `task/status/queue/count`: Monitors the number of queued items
+* `task/status/updated/count`: Monitors the number of processed items
+
+[14533](https://github.com/apache/druid/pull/14533)
+
+### Added `groupId` to Overlord task metrics
+
+Added `groupId` to task metrics emitted by the Overlord. This is helpful for grouping metrics like task/run/time by a single task group, such as a single compaction task or a single MSQ query.
+
+[14402](https://github.com/apache/druid/pull/14402)
+
+## Cluster management
+
+### Enabled cancellation and prioritization of load queue items
+
+Revamped Coordinator to make it more stable and user-friendly. This is accompanied by several bug fixes, logging and metric improvements, and a whole new range of capabilities.
+
+Coordinator now supports a `smartSegmentLoading` mode, which is enabled by default. When enabled, users don't need to specify any of the following dynamic configs as they would be ignored by the coordinator. Instead, the coordinator computes the optimal values of the following configs at run time to best utilize coordinator runs:
+
+* `maxSegmentsInNodeLoadingQueue`
+* `maxSegmentsToMove`
+* `replicationThrottleLimit`
+* `useRoundRobinSegmentAssignment`
+* `useBatchedSegmentSampler`
+* `emitBalancingStats`
+
+These configs are now deprecated and will be removed in subsequent releases.
+
+Coordinator is now capable of prioritization and cancellation of items in segment load queues. This means that the coordinator now reacts faster to changes in the cluster and makes better segment assignment decisions.
+
+[13197](https://github.com/apache/druid/pull/13197)
+
+### Removed unused Coordinator dynamic configuration properties
+
+The following Coordinator dynamic configs have been removed:
+
+* `emitBalancingStats`: Stats for errors encountered while balancing will always be emitted. Other debugging stats will not be emitted but can be logged by setting the appropriate `debugDimensions`.
+* `useBatchedSegmentSampler` and `percentOfSegmentsToConsiderPerMove`: Batched segment sampling is now the standard and will always be on.
+
+[14524](https://github.com/apache/druid/pull/14524)
+
+### Improved segment balancing strategy
+
+The `cost` balancer strategy performs much better now and is capable of moving more segments in a single Coordinator run. These improvements were made by borrowing ideas from the `cachingCost` strategy.
+
+The `cachingCost` strategy itself has been known to cause issues and is now deprecated. It is likely to be removed in future Druid releases and must not be used on any cluster.
+
+[14484](https://github.com/apache/druid/pull/14484)
+
+### Enabled empty tiered replicants for load rules
+
+Druid now allows empty tiered replicants in load rules.
+
+[14432](https://github.com/apache/druid/pull/14432)
+
+### Changed Coordinator configs values
+
+The following Coordinator dynamic configs have new default values:
+
+* `maxsegmentsInNodeLoadingQueue` : 500, previously 100
+* `maxSegmentsToMove`: 100, previously 5
+* `replicationThrottleLimit`: 500, previously 10
+
+These new defaults can improve performance for most use cases.
+
+[14269](https://github.com/apache/druid/pull/14269)
+
+### Improved error messages
+
+Introduced a new unified exception, `DruidException`, for surfacing errors. It is partially compatible with the old way of reporting error messages. Response codes remain the same, all fields that previously existed on the response will continue to exist and be populated, including `errorMessage`. Some error messages have changed to be more consumable by humans and some cases have the message restructured. There should be no impact to the response codes.
+
+[14004](https://github.com/apache/druid/pull/14004)
+
+## Web console
+
+### Supervisors and Tasks
+
+Replaced the **Ingestion** view with two views: **Supervisors** and **Tasks**.
+
+[14395](https://github.com/apache/druid/pull/14395)
+
+### Added replication factor column
+
+Added a new virtual column `replication_factor` to the `sys.segments` table. This returns the total number of replicants of the segment across all tiers. The column is set to -1 if the information is not available.
+
+[14403](https://github.com/apache/druid/pull/14403)
+
+### Other web console improvements
+
+## Extensions
+
+### Improved segment metadata for Kafka emitter extension
+
+The Kafka emitter extension has been improved. You can now publish events related to segments and their metadata to Kafka.
+You can set the new properties such as in the following example:
+
+```
+druid.emitter.kafka.event.types=["metrics", "alerts", "segment_metadata"]
+druid.emitter.kafka.segmentMetadata.topic=foo
+```
+
+[14281](https://github.com/apache/druid/pull/14281)
+
+## Dependency updates
Review Comment:
Yeah, several of them might have been bug fixes but we might have just forgotten to label them appropriately.
##########
docs/do-not-merge.md:
##########
@@ -0,0 +1,377 @@
+<!--Intentionally, there's no Apache license so that the GHA fails it. This file is not meant to be merged.
+
+- https://github.com/apache/druid/pull/14266 - we removed input source security from 26 (https://github.com/apache/druid/pull/14003). Should we not include this in 27 release notes?
+
+-->
+
+Apache Druid 27.0.0 contains over $NUMBER_FEATURES new features, bug fixes, performance enhancements, documentation improvements, and additional test coverage from $NUMBER_OF_CONTRIBUTORS contributors.
+
+[See the complete set of changes for additional details]($LINK_TO_RELEASE_MILESTONE).
+
+Review the upgrade notes and incompatible changes before you upgrade to Druid 27.0.0.
+
+# Highlights
+
+<!-- HIGHLIGHTS H2. FOR EACH MAJOR FEATURE FOR THE RELEASE -->
+
+## Query from deep storage (experimental)
+
+Druid now supports querying segments that are stored only in deep storage. When you query from deep storage, you increase the data available for queries without necessarily having to scale your Historical processes to accommodate more data. To take advantage of the potential savings, make sure you configure your load rules to not load all your data onto Historical processes. Then, configure a corresponding drop rule to drop those segments.
+
+Note that at least one segment of a datasource must be loaded onto a Historical process so that Druid can plan the query. It can be any segment though.
+
+As part of this feature, there are new API endpoints available to interact with the query:
+
+- Submit queries using `POST /sql/statements`. The payload is the same as any other Druid SQL query with additional context parameter available.
+ - The required `executionMode: ASYNC` configures Druid to return the results asynchronously. Results are only available after the query completes.
+ - The optional context parameter `selectDestination:DURABLE_STORAGE` configures Druid to write the query results to durable storage.
+
+- Get the status of a query with `GET /sq/statements/:queryID`.
+
+- Get query results with `/sql/statements/:queryID`
+
+[14416](https://github.com/apache/druid/pull/14416)
+[14512](https://github.com/apache/druid/pull/14512)
+[14527](https://github.com/apache/druid/pull/14527)
+
+## Java 17 support
+
+Druid now fully supports Java 17.
+[14384](https://github.com/apache/druid/pull/14384)
+
+## Array column types
+
+Array column types are now generally available.
+
+For more information about this feature, see the [26.0.0 release notes for array column types](https://github.com/apache/druid/releases#26.0.0-highlights-auto-type-column-schema-%28experimental%29).
+
+## Schema auto-discovery
+
+Schema auto-discovery is now generally available.
+
+For more information about this feature, see the [26.0.0 release notes for Schema auto-discovery](https://github.com/apache/druid/releases#26.0.0-highlights-auto-type-column-schema-%28experimental%29-schema-auto-discovery-%28experimental%29).
+
+## Smart segment loading
+
+Revamped Coordinator to make it more stable and user-friendly. This is accompanied by several bug fixes, logging and metric improvements, and a whole new range of capabilities.
+
+Coordinator now supports a `smartSegmentLoading` mode, which is enabled by default. When enabled, users don't need to specify any of the following dynamic configs as they would be ignored by the coordinator. Instead, the coordinator computes the optimal values of the following configs at run time to best utilize coordinator runs:
+
+* `maxSegmentsInNodeLoadingQueue`
+* `maxSegmentsToMove`
+* `replicationThrottleLimit`
+* `useRoundRobinSegmentAssignment`
+* `useBatchedSegmentSampler`
+* `emitBalancingStats`
+
+These configs are now deprecated and will be removed in subsequent releases.
+
+Coordinator is now capable of prioritization and cancellation of items in segment load queues. This means that the coordinator now reacts faster to changes in the cluster and makes better segment assignment decisions.
+
+As part of this change, new metrics are available. For more information, see [New segment metrics](#new-segment-metrics).
+
+[13197](https://github.com/apache/druid/pull/13197)
+
+## Hadoop 2 deprecated
+
+Support for Hadoop 2 is now deprecated. It will be removed in a future release.
+
+# Additional features and improvements
+
+## MSQ task engine
+
+### `maxInputBytesPerWorker` context parameter
+
+The context parameter now denotes the estimated weighted size (in bytes) of the input to split on. The MSQ task engine now takes into account the input format and compression format instead of the actual file size reported by the file system.
+
+The default value for the context parameter has also been changed. It is now 512 MiB (previously 10 GiB).
+
+[14307](https://github.com/apache/druid/pull/14307)
+
+### Improved query planning behavior
+
+Druid now fails query planning if a CLUSTERED BY column contains descending order.
+Previously, queries would successfully plan if any CLUSTERED BY columns contained descending order.
+
+The MSQ fault, `InsertCannotOrderByDescending`, is deprecated. An INSERT or REPLACE query containing a CLUSTERED BY expression cannot be in descending order. Druid's segment generation code only supports ascending order. Instead of the fault, Druid now throws a query `ValidationException`.
+
+[14436](https://github.com/apache/druid/pull/14436)
+
+### SELECT query results
+
+SELECT queries executed using MSQ now generate only a subset of the results in the query reports.
+To fetch the complete result set, run the query using the native engine.
+
+[14370](https://github.com/apache/druid/pull/14370)
+
+
+### Other MSQ improvements
+
+- The same aggregator can now have two output names [14367](https://github.com/apache/druid/pull/14367)
+- Changed the default `clusterStatisticsMergeMode` to SEQUENTIAL (#14310)[https://github.com/apache/druid/pull/14310]
+- Added a query context parameter `MultiStageQueryContext` to determine whether the result of an MSQ select query is limited (#14476)[https://github.com/apache/druid/pull/14476]
+- Enabled using functions as inputs for `index` and `length` parameters (#14480)[https://github.com/apache/druid/pull/14480]
+
+## Ingestion
+
+### New property for task completion updates
+
+The new property `druid.indexer.queue.taskCompleteHandlerNumThreads` controls the number of threads used by the Overlord `TaskQueue` to handle task completion updates received from the workers.
+
+The following metrics have been added:
+* `task/status/queue/count`: Monitors the number of queued items
+* `task/status/updated/count`: Monitors the number of processed items
+
+[14533](https://github.com/apache/druid/pull/14533)
+
+### Improved response to max_allowed_packet limit
+
+If the Overlord fails to insert a task into the metadata because of a payload that exceeds the `max_allowed_packet` limit, the response now returns `400 Bad request` instead of `500 Internal server error`. This prevents an `index_parallel` task from retrying the insertion of a bad sub-task indefinitely and causes it to fail immediately.
+
+[14271](https://github.com/apache/druid/pull/14271)
+
+### Improved handling of mixed type arrays
+
+Druid now handles mixed type arrays such as `[["a", "b", "c"], {"x": 123}]` as `ARRAY<COMPLEX<json>` rather than throwing an incompatible type exception.
+
+[14438](https://github.com/apache/druid/pull/14438)
+
+### Other ingestion improvements
+
+* A negative streaming ingestion lag is no longer emitted as a result of stale offsets. [14292](https://github.com/apache/druid/pull/14292)
+* Removed double synchronization on simple map operations in Kubernetes task runner. [14435](https://github.com/apache/druid/pull/14435)
+* Kubernetes overlord extension now cleans up the job if the task pod fails to come up in time. [14425](https://github.com/apache/druid/pull/14425)
+
+## Querying
+
+### New function for regular expression replacement
+
+The new function `REGEXP_REPLACE` allows you to replace all instances of a pattern with a replacement string.
+
+[14460](https://github.com/apache/druid/pull/14460)
+
+### Query results directory
+
+Druid now supports a `query-results` directory in durable storage to store query results after the task finishes. The auto cleaner does not remove this directory unless the task ID is not known to the Overlord.
+
+[14446](https://github.com/apache/druid/pull/14446)
+
+### Limit for subquery results by memory usage
+
+Users can now add a guardrail to prevent subquery’s results from exceeding the set number of bytes by setting `druid.server.http.maxSubqueryRows` in the Broker's config or `maxSubqueryRows` in the query context. This feature is experimental for now and would default back to row-based limiting in case it fails to get the accurate size of the results consumed by the query.
+
+[13952](https://github.com/apache/druid/pull/13952)
+
+### HLL and Theta sketch estimates
+
+You can now use `HLL_SKETCH_ESTIMATE` and `THETA_SKETCH_ESTIMATE` as expressions. These estimates work on sketch columns and have the same behavior as `postAggs`.
+
+[14312](https://github.com/apache/druid/pull/14312)
+
+### String encoding parameter for HLL sketches
+
+HLL sketches now accept a new `stringEncoding` parameter to build sketches with UTF-8 bytes.
+
+[11201](https://github.com/apache/druid/pull/11201)
+
+### EARLIEST_BY and LATEST_BY signatures
+
+Updated EARLIEST_BY and LATEST_BY function signatures as follows:
+
+* Changed `EARLIEST(expr, timeColumn)` to `EARLIEST_BY(expr, timeColumn)`
+* Changed `LATEST(expr, timeColumn)` to `LATEST_BY(expr, timeColumn)`
+
+[14352](https://github.com/apache/druid/pull/14352)
+
+
+### Exposed Druid functions
+
+Exposed information about Druid functions in the `INFORMATION_SCHEMA.ROUTINES` table. The table contains the following columns:
+
+* ROUTINE_CATALOG: Name of the database. Always set to druid.
+* ROUTINE_SCHEMA: Name of the schema. Always set to INFORMATION_SCHEMA.
+* ROUTINE_NAME: Name of the function or operator. For example: APPROX_COUNT_DISTINCT_DS_THETA.
+* ROUTINE_TYPE: Type of routine. Always set to FUNCTION.
+* IS_AGGREGATOR: Determines if the routine is an aggregator. Returns YES for aggregator functions; NO for scalar functions.
+* SIGNATURES: Possible signatures for the routine as a string.
+
+[14378](https://github.com/apache/druid/pull/14378)
+
+### New Broker configuration for SQL schema migrations
+
+A new broker configuration, `druid.sql.planner.metadataColumnTypeMergePolicy`, adds configurable modes to how column types are computed for the SQL table schema when faced with differences between segments.
+
+The new `leastRestrictive` mode chooses the most appropriate type that data across all segments can best be coerced into and is the default behavior. This is a subtle behavior change around when segment driven schema migrations will take effect for the SQL schema:
+
+- With `leastRestrictive` mode, the schema only updates once all segments are reindexed to the new type.
+- With `latestInterval` mode, the SQL schema gets updated as soon as the first job with the new schema publishes segments in the latest time interval of the data.
+
+`leastRestrictive` can have better query time behavior and eliminates some query time errors that can occur when using `latestInterval`.
+
+[14319](https://github.com/apache/druid/pull/14319)
+
+### EXPLAIN PLAN improvements
+
+The EXPLAIN PLAN result includes a new column `ATTRIBUTES` that describes the attributes of a query.
+See [SQL query translation](docs/querying/sql-translation.md) for more information and examples.
+
+[14391](https://github.com/apache/druid/pull/14391)
+
+
+## Metrics and monitoring
+
+### Added a new SysMonitorOshi monitor
+
+Added a new monitor `SysMonitorOshi` to replace `SysMonitor`. The new monitor has a wider support for different machine architectures including ARM instances. We recommend switching to `SysMonitorOshi` as `SysMonitor` is now deprecated and will be removed in future releases.
+
+[14359](https://github.com/apache/druid/pull/14359)
+
+### Monitor for Overlord and Coordinator service health
+
+Added a new monitor `ServiceStatusMonitor` to monitor the service health of the Overlord and Coordinator.
+
+[14443](https://github.com/apache/druid/pull/14443)
+
+### New segment metrics
+
+The following metrics are now available for Brokers:
+
+|Metric|Description|Dimensions |Normal value|
+|------|-----------|------------|
+|`segment/metadatacache/refresh/count`|Number of segments to refresh in broker segment metadata cache. Emitted once per refresh per datasource.|`dataSource`| |
+|`segment/metadatacache/refresh/time`|Time taken to refresh segments in broker segment metadata cache. Emitted once per refresh per datasource.|`dataSource`| |
+| `segment/loadQueue/assigned` | Number of segments assigned for load or drop to the load queue of a server. |`dataSource`,`server` | Varies |
+| `segment/loadQueue/success` |Number of segment assignments that completed successfully.|`dataSource`, `server`|Varies|
+| `segment/loadQueue/cancelled` |Number of segment assignments that were canceled before completion. |`dataSource`,`server` | 0 |
+| `segment/loadQueue/failed` |Number of segment assignments that failed to complete.|`dataSource`, `server`|0|
+| `segment/assigned/broadcast` | |
+| `segment/dropped/broadcast` | |
+
+[14453](https://github.com/apache/druid/pull/14453)
+[13197](https://github.com/apache/druid/pull/13197)
+
+### New metrics for task completion updates
+
+The following metrics have been added:
+* `task/status/queue/count`: Monitors the number of queued items
+* `task/status/updated/count`: Monitors the number of processed items
+
+[14533](https://github.com/apache/druid/pull/14533)
+
+### Added `groupId` to Overlord task metrics
+
+Added `groupId` to task metrics emitted by the Overlord. This is helpful for grouping metrics like task/run/time by a single task group, such as a single compaction task or a single MSQ query.
+
+[14402](https://github.com/apache/druid/pull/14402)
+
Review Comment:
```suggestion
### New metrics for monitoring sync status of `HttpServerInventoryView`
|Metric|Description|Dimensions|Normal value|
|------|-----------|-----------|----------|
|`segment/serverview/sync/healthy`|Sync status of the Coordinator/Broker with a segment-loading server such as a Historical or Peon. Emitted by the Coordinator and Brokers only when [HTTP-based server view](../configuration/index.md#segment-management) is enabled. This metric can be used in conjunction with `segment/serverview/sync/unstableTime` to debug slow startup of the Coordinator.|`server`, `tier`|1 for fully synced servers, 0 otherwise|
|`segment/serverview/sync/unstableTime`|Time in milliseconds for which the Coordinator/Broker has been failing to sync with a segment-loading server. Emitted by the Coordinator and Brokers only when [HTTP-based server view](../configuration/index.md#segment-management) is enabled.|`server`, `tier`|Not emitted for synced servers.|```
##########
docs/do-not-merge.md:
##########
@@ -0,0 +1,377 @@
+<!--Intentionally, there's no Apache license so that the GHA fails it. This file is not meant to be merged.
+
+- https://github.com/apache/druid/pull/14266 - we removed input source security from 26 (https://github.com/apache/druid/pull/14003). Should we not include this in 27 release notes?
+
+-->
+
+Apache Druid 27.0.0 contains over $NUMBER_FEATURES new features, bug fixes, performance enhancements, documentation improvements, and additional test coverage from $NUMBER_OF_CONTRIBUTORS contributors.
+
+[See the complete set of changes for additional details]($LINK_TO_RELEASE_MILESTONE).
+
+Review the upgrade notes and incompatible changes before you upgrade to Druid 27.0.0.
+
+# Highlights
+
+<!-- HIGHLIGHTS H2. FOR EACH MAJOR FEATURE FOR THE RELEASE -->
+
+## Query from deep storage (experimental)
+
+Druid now supports querying segments that are stored only in deep storage. When you query from deep storage, you increase the data available for queries without necessarily having to scale your Historical processes to accommodate more data. To take advantage of the potential savings, make sure you configure your load rules to not load all your data onto Historical processes. Then, configure a corresponding drop rule to drop those segments.
+
+Note that at least one segment of a datasource must be loaded onto a Historical process so that Druid can plan the query. It can be any segment though.
+
+As part of this feature, there are new API endpoints available to interact with the query:
+
+- Submit queries using `POST /sql/statements`. The payload is the same as any other Druid SQL query with additional context parameter available.
+ - The required `executionMode: ASYNC` configures Druid to return the results asynchronously. Results are only available after the query completes.
+ - The optional context parameter `selectDestination:DURABLE_STORAGE` configures Druid to write the query results to durable storage.
+
+- Get the status of a query with `GET /sq/statements/:queryID`.
+
+- Get query results with `/sql/statements/:queryID`
+
+[14416](https://github.com/apache/druid/pull/14416)
+[14512](https://github.com/apache/druid/pull/14512)
+[14527](https://github.com/apache/druid/pull/14527)
+
+## Java 17 support
+
+Druid now fully supports Java 17.
+[14384](https://github.com/apache/druid/pull/14384)
+
+## Array column types
+
+Array column types are now generally available.
+
+For more information about this feature, see the [26.0.0 release notes for array column types](https://github.com/apache/druid/releases#26.0.0-highlights-auto-type-column-schema-%28experimental%29).
+
+## Schema auto-discovery
+
+Schema auto-discovery is now generally available.
+
+For more information about this feature, see the [26.0.0 release notes for Schema auto-discovery](https://github.com/apache/druid/releases#26.0.0-highlights-auto-type-column-schema-%28experimental%29-schema-auto-discovery-%28experimental%29).
+
+## Smart segment loading
+
+Revamped Coordinator to make it more stable and user-friendly. This is accompanied by several bug fixes, logging and metric improvements, and a whole new range of capabilities.
+
+Coordinator now supports a `smartSegmentLoading` mode, which is enabled by default. When enabled, users don't need to specify any of the following dynamic configs as they would be ignored by the coordinator. Instead, the coordinator computes the optimal values of the following configs at run time to best utilize coordinator runs:
+
+* `maxSegmentsInNodeLoadingQueue`
+* `maxSegmentsToMove`
+* `replicationThrottleLimit`
+* `useRoundRobinSegmentAssignment`
+* `useBatchedSegmentSampler`
+* `emitBalancingStats`
+
+These configs are now deprecated and will be removed in subsequent releases.
+
+Coordinator is now capable of prioritization and cancellation of items in segment load queues. This means that the coordinator now reacts faster to changes in the cluster and makes better segment assignment decisions.
+
+As part of this change, new metrics are available. For more information, see [New segment metrics](#new-segment-metrics).
+
+[13197](https://github.com/apache/druid/pull/13197)
+
+## Hadoop 2 deprecated
+
+Support for Hadoop 2 is now deprecated. It will be removed in a future release.
+
+# Additional features and improvements
+
+## MSQ task engine
+
+### `maxInputBytesPerWorker` context parameter
+
+The context parameter now denotes the estimated weighted size (in bytes) of the input to split on. The MSQ task engine now takes into account the input format and compression format instead of the actual file size reported by the file system.
+
+The default value for the context parameter has also been changed. It is now 512 MiB (previously 10 GiB).
+
+[14307](https://github.com/apache/druid/pull/14307)
+
+### Improved query planning behavior
+
+Druid now fails query planning if a CLUSTERED BY column contains descending order.
+Previously, queries would successfully plan if any CLUSTERED BY columns contained descending order.
+
+The MSQ fault, `InsertCannotOrderByDescending`, is deprecated. An INSERT or REPLACE query containing a CLUSTERED BY expression cannot be in descending order. Druid's segment generation code only supports ascending order. Instead of the fault, Druid now throws a query `ValidationException`.
+
+[14436](https://github.com/apache/druid/pull/14436)
+
+### SELECT query results
+
+SELECT queries executed using MSQ now generate only a subset of the results in the query reports.
+To fetch the complete result set, run the query using the native engine.
+
+[14370](https://github.com/apache/druid/pull/14370)
+
+
+### Other MSQ improvements
+
+- The same aggregator can now have two output names [14367](https://github.com/apache/druid/pull/14367)
+- Changed the default `clusterStatisticsMergeMode` to SEQUENTIAL (#14310)[https://github.com/apache/druid/pull/14310]
+- Added a query context parameter `MultiStageQueryContext` to determine whether the result of an MSQ select query is limited (#14476)[https://github.com/apache/druid/pull/14476]
+- Enabled using functions as inputs for `index` and `length` parameters (#14480)[https://github.com/apache/druid/pull/14480]
+
+## Ingestion
+
+### New property for task completion updates
+
+The new property `druid.indexer.queue.taskCompleteHandlerNumThreads` controls the number of threads used by the Overlord `TaskQueue` to handle task completion updates received from the workers.
+
+The following metrics have been added:
+* `task/status/queue/count`: Monitors the number of queued items
+* `task/status/updated/count`: Monitors the number of processed items
+
+[14533](https://github.com/apache/druid/pull/14533)
+
+### Improved response to max_allowed_packet limit
+
+If the Overlord fails to insert a task into the metadata because of a payload that exceeds the `max_allowed_packet` limit, the response now returns `400 Bad request` instead of `500 Internal server error`. This prevents an `index_parallel` task from retrying the insertion of a bad sub-task indefinitely and causes it to fail immediately.
+
+[14271](https://github.com/apache/druid/pull/14271)
+
+### Improved handling of mixed type arrays
+
+Druid now handles mixed type arrays such as `[["a", "b", "c"], {"x": 123}]` as `ARRAY<COMPLEX<json>` rather than throwing an incompatible type exception.
+
+[14438](https://github.com/apache/druid/pull/14438)
+
+### Other ingestion improvements
+
+* A negative streaming ingestion lag is no longer emitted as a result of stale offsets. [14292](https://github.com/apache/druid/pull/14292)
+* Removed double synchronization on simple map operations in Kubernetes task runner. [14435](https://github.com/apache/druid/pull/14435)
+* Kubernetes overlord extension now cleans up the job if the task pod fails to come up in time. [14425](https://github.com/apache/druid/pull/14425)
+
+## Querying
+
+### New function for regular expression replacement
+
+The new function `REGEXP_REPLACE` allows you to replace all instances of a pattern with a replacement string.
+
+[14460](https://github.com/apache/druid/pull/14460)
+
+### Query results directory
+
+Druid now supports a `query-results` directory in durable storage to store query results after the task finishes. The auto cleaner does not remove this directory unless the task ID is not known to the Overlord.
+
+[14446](https://github.com/apache/druid/pull/14446)
+
+### Limit for subquery results by memory usage
+
+Users can now add a guardrail to prevent subquery’s results from exceeding the set number of bytes by setting `druid.server.http.maxSubqueryRows` in the Broker's config or `maxSubqueryRows` in the query context. This feature is experimental for now and would default back to row-based limiting in case it fails to get the accurate size of the results consumed by the query.
+
+[13952](https://github.com/apache/druid/pull/13952)
+
+### HLL and Theta sketch estimates
+
+You can now use `HLL_SKETCH_ESTIMATE` and `THETA_SKETCH_ESTIMATE` as expressions. These estimates work on sketch columns and have the same behavior as `postAggs`.
+
+[14312](https://github.com/apache/druid/pull/14312)
+
+### String encoding parameter for HLL sketches
+
+HLL sketches now accept a new `stringEncoding` parameter to build sketches with UTF-8 bytes.
+
+[11201](https://github.com/apache/druid/pull/11201)
+
+### EARLIEST_BY and LATEST_BY signatures
+
+Updated EARLIEST_BY and LATEST_BY function signatures as follows:
+
+* Changed `EARLIEST(expr, timeColumn)` to `EARLIEST_BY(expr, timeColumn)`
+* Changed `LATEST(expr, timeColumn)` to `LATEST_BY(expr, timeColumn)`
+
+[14352](https://github.com/apache/druid/pull/14352)
+
+
+### Exposed Druid functions
+
+Exposed information about Druid functions in the `INFORMATION_SCHEMA.ROUTINES` table. The table contains the following columns:
+
+* ROUTINE_CATALOG: Name of the database. Always set to druid.
+* ROUTINE_SCHEMA: Name of the schema. Always set to INFORMATION_SCHEMA.
+* ROUTINE_NAME: Name of the function or operator. For example: APPROX_COUNT_DISTINCT_DS_THETA.
+* ROUTINE_TYPE: Type of routine. Always set to FUNCTION.
+* IS_AGGREGATOR: Determines if the routine is an aggregator. Returns YES for aggregator functions; NO for scalar functions.
+* SIGNATURES: Possible signatures for the routine as a string.
+
+[14378](https://github.com/apache/druid/pull/14378)
+
+### New Broker configuration for SQL schema migrations
+
+A new broker configuration, `druid.sql.planner.metadataColumnTypeMergePolicy`, adds configurable modes to how column types are computed for the SQL table schema when faced with differences between segments.
+
+The new `leastRestrictive` mode chooses the most appropriate type that data across all segments can best be coerced into and is the default behavior. This is a subtle behavior change around when segment driven schema migrations will take effect for the SQL schema:
+
+- With `leastRestrictive` mode, the schema only updates once all segments are reindexed to the new type.
+- With `latestInterval` mode, the SQL schema gets updated as soon as the first job with the new schema publishes segments in the latest time interval of the data.
+
+`leastRestrictive` can have better query time behavior and eliminates some query time errors that can occur when using `latestInterval`.
+
+[14319](https://github.com/apache/druid/pull/14319)
+
+### EXPLAIN PLAN improvements
+
+The EXPLAIN PLAN result includes a new column `ATTRIBUTES` that describes the attributes of a query.
+See [SQL query translation](docs/querying/sql-translation.md) for more information and examples.
+
+[14391](https://github.com/apache/druid/pull/14391)
+
+
+## Metrics and monitoring
+
+### Added a new SysMonitorOshi monitor
+
+Added a new monitor `SysMonitorOshi` to replace `SysMonitor`. The new monitor has a wider support for different machine architectures including ARM instances. We recommend switching to `SysMonitorOshi` as `SysMonitor` is now deprecated and will be removed in future releases.
+
+[14359](https://github.com/apache/druid/pull/14359)
+
+### Monitor for Overlord and Coordinator service health
+
+Added a new monitor `ServiceStatusMonitor` to monitor the service health of the Overlord and Coordinator.
+
+[14443](https://github.com/apache/druid/pull/14443)
+
+### New segment metrics
+
+The following metrics are now available for Brokers:
+
+|Metric|Description|Dimensions |Normal value|
+|------|-----------|------------|
Review Comment:
```suggestion
|Metric|Description|Dimensions|Normal value|
|------|-----------|-----------|----------|
```
##########
docs/do-not-merge.md:
##########
@@ -0,0 +1,377 @@
+<!--Intentionally, there's no Apache license so that the GHA fails it. This file is not meant to be merged.
+
+- https://github.com/apache/druid/pull/14266 - we removed input source security from 26 (https://github.com/apache/druid/pull/14003). Should we not include this in 27 release notes?
+
+-->
+
+Apache Druid 27.0.0 contains over $NUMBER_FEATURES new features, bug fixes, performance enhancements, documentation improvements, and additional test coverage from $NUMBER_OF_CONTRIBUTORS contributors.
+
+[See the complete set of changes for additional details]($LINK_TO_RELEASE_MILESTONE).
+
+Review the upgrade notes and incompatible changes before you upgrade to Druid 27.0.0.
+
+# Highlights
+
+<!-- HIGHLIGHTS H2. FOR EACH MAJOR FEATURE FOR THE RELEASE -->
+
+## Query from deep storage (experimental)
+
+Druid now supports querying segments that are stored only in deep storage. When you query from deep storage, you increase the data available for queries without necessarily having to scale your Historical processes to accommodate more data. To take advantage of the potential savings, make sure you configure your load rules to not load all your data onto Historical processes. Then, configure a corresponding drop rule to drop those segments.
+
+Note that at least one segment of a datasource must be loaded onto a Historical process so that Druid can plan the query. It can be any segment though.
+
+As part of this feature, there are new API endpoints available to interact with the query:
+
+- Submit queries using `POST /sql/statements`. The payload is the same as any other Druid SQL query with additional context parameter available.
+ - The required `executionMode: ASYNC` configures Druid to return the results asynchronously. Results are only available after the query completes.
+ - The optional context parameter `selectDestination:DURABLE_STORAGE` configures Druid to write the query results to durable storage.
+
+- Get the status of a query with `GET /sq/statements/:queryID`.
+
+- Get query results with `/sql/statements/:queryID`
+
+[14416](https://github.com/apache/druid/pull/14416)
+[14512](https://github.com/apache/druid/pull/14512)
+[14527](https://github.com/apache/druid/pull/14527)
+
+## Java 17 support
+
+Druid now fully supports Java 17.
+[14384](https://github.com/apache/druid/pull/14384)
+
+## Array column types
+
+Array column types are now generally available.
+
+For more information about this feature, see the [26.0.0 release notes for array column types](https://github.com/apache/druid/releases#26.0.0-highlights-auto-type-column-schema-%28experimental%29).
+
+## Schema auto-discovery
+
+Schema auto-discovery is now generally available.
+
+For more information about this feature, see the [26.0.0 release notes for Schema auto-discovery](https://github.com/apache/druid/releases#26.0.0-highlights-auto-type-column-schema-%28experimental%29-schema-auto-discovery-%28experimental%29).
+
+## Smart segment loading
+
+Revamped Coordinator to make it more stable and user-friendly. This is accompanied by several bug fixes, logging and metric improvements, and a whole new range of capabilities.
+
+Coordinator now supports a `smartSegmentLoading` mode, which is enabled by default. When enabled, users don't need to specify any of the following dynamic configs as they would be ignored by the coordinator. Instead, the coordinator computes the optimal values of the following configs at run time to best utilize coordinator runs:
+
+* `maxSegmentsInNodeLoadingQueue`
+* `maxSegmentsToMove`
+* `replicationThrottleLimit`
+* `useRoundRobinSegmentAssignment`
+* `useBatchedSegmentSampler`
+* `emitBalancingStats`
Review Comment:
```suggestion
* `replicantLifetime`
* `maxNonPrimaryReplicantsToLoad`
* `decommissioningMaxPercentOfMaxSegmentsToMove`
```
The two configs `emitBalancingStats` and `useBatchedSegmentSampler` have been completely removed as indicated in the "Cluster managmeent" section below.
`replicantLifetime`, `maxNonPrimaryReplicantsToLoad` and `decommissiongMaxPercent` are also included under `smartSegmentLoading` now.
##########
docs/do-not-merge.md:
##########
@@ -0,0 +1,377 @@
+<!--Intentionally, there's no Apache license so that the GHA fails it. This file is not meant to be merged.
+
+- https://github.com/apache/druid/pull/14266 - we removed input source security from 26 (https://github.com/apache/druid/pull/14003). Should we not include this in 27 release notes?
+
+-->
+
+Apache Druid 27.0.0 contains over $NUMBER_FEATURES new features, bug fixes, performance enhancements, documentation improvements, and additional test coverage from $NUMBER_OF_CONTRIBUTORS contributors.
+
+[See the complete set of changes for additional details]($LINK_TO_RELEASE_MILESTONE).
+
+Review the upgrade notes and incompatible changes before you upgrade to Druid 27.0.0.
+
+# Highlights
+
+<!-- HIGHLIGHTS H2. FOR EACH MAJOR FEATURE FOR THE RELEASE -->
+
+## Query from deep storage (experimental)
+
+Druid now supports querying segments that are stored only in deep storage. When you query from deep storage, you increase the data available for queries without necessarily having to scale your Historical processes to accommodate more data. To take advantage of the potential savings, make sure you configure your load rules to not load all your data onto Historical processes. Then, configure a corresponding drop rule to drop those segments.
+
+Note that at least one segment of a datasource must be loaded onto a Historical process so that Druid can plan the query. It can be any segment though.
+
+As part of this feature, there are new API endpoints available to interact with the query:
+
+- Submit queries using `POST /sql/statements`. The payload is the same as any other Druid SQL query with additional context parameter available.
+ - The required `executionMode: ASYNC` configures Druid to return the results asynchronously. Results are only available after the query completes.
+ - The optional context parameter `selectDestination:DURABLE_STORAGE` configures Druid to write the query results to durable storage.
+
+- Get the status of a query with `GET /sq/statements/:queryID`.
+
+- Get query results with `/sql/statements/:queryID`
+
+[14416](https://github.com/apache/druid/pull/14416)
+[14512](https://github.com/apache/druid/pull/14512)
+[14527](https://github.com/apache/druid/pull/14527)
+
+## Java 17 support
+
+Druid now fully supports Java 17.
+[14384](https://github.com/apache/druid/pull/14384)
+
+## Array column types
+
+Array column types are now generally available.
+
+For more information about this feature, see the [26.0.0 release notes for array column types](https://github.com/apache/druid/releases#26.0.0-highlights-auto-type-column-schema-%28experimental%29).
+
+## Schema auto-discovery
+
+Schema auto-discovery is now generally available.
+
+For more information about this feature, see the [26.0.0 release notes for Schema auto-discovery](https://github.com/apache/druid/releases#26.0.0-highlights-auto-type-column-schema-%28experimental%29-schema-auto-discovery-%28experimental%29).
+
+## Smart segment loading
+
+Revamped Coordinator to make it more stable and user-friendly. This is accompanied by several bug fixes, logging and metric improvements, and a whole new range of capabilities.
+
+Coordinator now supports a `smartSegmentLoading` mode, which is enabled by default. When enabled, users don't need to specify any of the following dynamic configs as they would be ignored by the coordinator. Instead, the coordinator computes the optimal values of the following configs at run time to best utilize coordinator runs:
+
+* `maxSegmentsInNodeLoadingQueue`
+* `maxSegmentsToMove`
+* `replicationThrottleLimit`
+* `useRoundRobinSegmentAssignment`
+* `useBatchedSegmentSampler`
+* `emitBalancingStats`
+
+These configs are now deprecated and will be removed in subsequent releases.
+
+Coordinator is now capable of prioritization and cancellation of items in segment load queues. This means that the coordinator now reacts faster to changes in the cluster and makes better segment assignment decisions.
+
+As part of this change, new metrics are available. For more information, see [New segment metrics](#new-segment-metrics).
+
+[13197](https://github.com/apache/druid/pull/13197)
+
+## Hadoop 2 deprecated
+
+Support for Hadoop 2 is now deprecated. It will be removed in a future release.
+
+# Additional features and improvements
+
+## MSQ task engine
+
+### `maxInputBytesPerWorker` context parameter
+
+The context parameter now denotes the estimated weighted size (in bytes) of the input to split on. The MSQ task engine now takes into account the input format and compression format instead of the actual file size reported by the file system.
+
+The default value for the context parameter has also been changed. It is now 512 MiB (previously 10 GiB).
+
+[14307](https://github.com/apache/druid/pull/14307)
+
+### Improved query planning behavior
+
+Druid now fails query planning if a CLUSTERED BY column contains descending order.
+Previously, queries would successfully plan if any CLUSTERED BY columns contained descending order.
+
+The MSQ fault, `InsertCannotOrderByDescending`, is deprecated. An INSERT or REPLACE query containing a CLUSTERED BY expression cannot be in descending order. Druid's segment generation code only supports ascending order. Instead of the fault, Druid now throws a query `ValidationException`.
+
+[14436](https://github.com/apache/druid/pull/14436)
+
+### SELECT query results
+
+SELECT queries executed using MSQ now generate only a subset of the results in the query reports.
+To fetch the complete result set, run the query using the native engine.
+
+[14370](https://github.com/apache/druid/pull/14370)
+
+
+### Other MSQ improvements
+
+- The same aggregator can now have two output names [14367](https://github.com/apache/druid/pull/14367)
+- Changed the default `clusterStatisticsMergeMode` to SEQUENTIAL (#14310)[https://github.com/apache/druid/pull/14310]
+- Added a query context parameter `MultiStageQueryContext` to determine whether the result of an MSQ select query is limited (#14476)[https://github.com/apache/druid/pull/14476]
+- Enabled using functions as inputs for `index` and `length` parameters (#14480)[https://github.com/apache/druid/pull/14480]
+
+## Ingestion
+
+### New property for task completion updates
+
+The new property `druid.indexer.queue.taskCompleteHandlerNumThreads` controls the number of threads used by the Overlord `TaskQueue` to handle task completion updates received from the workers.
+
+The following metrics have been added:
+* `task/status/queue/count`: Monitors the number of queued items
+* `task/status/updated/count`: Monitors the number of processed items
+
+[14533](https://github.com/apache/druid/pull/14533)
+
+### Improved response to max_allowed_packet limit
+
+If the Overlord fails to insert a task into the metadata because of a payload that exceeds the `max_allowed_packet` limit, the response now returns `400 Bad request` instead of `500 Internal server error`. This prevents an `index_parallel` task from retrying the insertion of a bad sub-task indefinitely and causes it to fail immediately.
+
+[14271](https://github.com/apache/druid/pull/14271)
+
+### Improved handling of mixed type arrays
+
+Druid now handles mixed type arrays such as `[["a", "b", "c"], {"x": 123}]` as `ARRAY<COMPLEX<json>` rather than throwing an incompatible type exception.
+
+[14438](https://github.com/apache/druid/pull/14438)
+
+### Other ingestion improvements
+
+* A negative streaming ingestion lag is no longer emitted as a result of stale offsets. [14292](https://github.com/apache/druid/pull/14292)
+* Removed double synchronization on simple map operations in Kubernetes task runner. [14435](https://github.com/apache/druid/pull/14435)
+* Kubernetes overlord extension now cleans up the job if the task pod fails to come up in time. [14425](https://github.com/apache/druid/pull/14425)
+
+## Querying
+
+### New function for regular expression replacement
+
+The new function `REGEXP_REPLACE` allows you to replace all instances of a pattern with a replacement string.
+
+[14460](https://github.com/apache/druid/pull/14460)
+
+### Query results directory
+
+Druid now supports a `query-results` directory in durable storage to store query results after the task finishes. The auto cleaner does not remove this directory unless the task ID is not known to the Overlord.
+
+[14446](https://github.com/apache/druid/pull/14446)
+
+### Limit for subquery results by memory usage
+
+Users can now add a guardrail to prevent subquery’s results from exceeding the set number of bytes by setting `druid.server.http.maxSubqueryRows` in the Broker's config or `maxSubqueryRows` in the query context. This feature is experimental for now and would default back to row-based limiting in case it fails to get the accurate size of the results consumed by the query.
+
+[13952](https://github.com/apache/druid/pull/13952)
+
+### HLL and Theta sketch estimates
+
+You can now use `HLL_SKETCH_ESTIMATE` and `THETA_SKETCH_ESTIMATE` as expressions. These estimates work on sketch columns and have the same behavior as `postAggs`.
+
+[14312](https://github.com/apache/druid/pull/14312)
+
+### String encoding parameter for HLL sketches
+
+HLL sketches now accept a new `stringEncoding` parameter to build sketches with UTF-8 bytes.
+
+[11201](https://github.com/apache/druid/pull/11201)
+
+### EARLIEST_BY and LATEST_BY signatures
+
+Updated EARLIEST_BY and LATEST_BY function signatures as follows:
+
+* Changed `EARLIEST(expr, timeColumn)` to `EARLIEST_BY(expr, timeColumn)`
+* Changed `LATEST(expr, timeColumn)` to `LATEST_BY(expr, timeColumn)`
+
+[14352](https://github.com/apache/druid/pull/14352)
+
+
+### Exposed Druid functions
+
+Exposed information about Druid functions in the `INFORMATION_SCHEMA.ROUTINES` table. The table contains the following columns:
+
+* ROUTINE_CATALOG: Name of the database. Always set to druid.
+* ROUTINE_SCHEMA: Name of the schema. Always set to INFORMATION_SCHEMA.
+* ROUTINE_NAME: Name of the function or operator. For example: APPROX_COUNT_DISTINCT_DS_THETA.
+* ROUTINE_TYPE: Type of routine. Always set to FUNCTION.
+* IS_AGGREGATOR: Determines if the routine is an aggregator. Returns YES for aggregator functions; NO for scalar functions.
+* SIGNATURES: Possible signatures for the routine as a string.
+
+[14378](https://github.com/apache/druid/pull/14378)
+
+### New Broker configuration for SQL schema migrations
+
+A new broker configuration, `druid.sql.planner.metadataColumnTypeMergePolicy`, adds configurable modes to how column types are computed for the SQL table schema when faced with differences between segments.
+
+The new `leastRestrictive` mode chooses the most appropriate type that data across all segments can best be coerced into and is the default behavior. This is a subtle behavior change around when segment driven schema migrations will take effect for the SQL schema:
+
+- With `leastRestrictive` mode, the schema only updates once all segments are reindexed to the new type.
+- With `latestInterval` mode, the SQL schema gets updated as soon as the first job with the new schema publishes segments in the latest time interval of the data.
+
+`leastRestrictive` can have better query time behavior and eliminates some query time errors that can occur when using `latestInterval`.
+
+[14319](https://github.com/apache/druid/pull/14319)
+
+### EXPLAIN PLAN improvements
+
+The EXPLAIN PLAN result includes a new column `ATTRIBUTES` that describes the attributes of a query.
+See [SQL query translation](docs/querying/sql-translation.md) for more information and examples.
+
+[14391](https://github.com/apache/druid/pull/14391)
+
+
+## Metrics and monitoring
+
+### Added a new SysMonitorOshi monitor
+
+Added a new monitor `SysMonitorOshi` to replace `SysMonitor`. The new monitor has a wider support for different machine architectures including ARM instances. We recommend switching to `SysMonitorOshi` as `SysMonitor` is now deprecated and will be removed in future releases.
+
+[14359](https://github.com/apache/druid/pull/14359)
+
+### Monitor for Overlord and Coordinator service health
+
+Added a new monitor `ServiceStatusMonitor` to monitor the service health of the Overlord and Coordinator.
+
+[14443](https://github.com/apache/druid/pull/14443)
+
+### New segment metrics
+
+The following metrics are now available for Brokers:
+
+|Metric|Description|Dimensions |Normal value|
+|------|-----------|------------|
+|`segment/metadatacache/refresh/count`|Number of segments to refresh in broker segment metadata cache. Emitted once per refresh per datasource.|`dataSource`| |
+|`segment/metadatacache/refresh/time`|Time taken to refresh segments in broker segment metadata cache. Emitted once per refresh per datasource.|`dataSource`| |
+| `segment/loadQueue/assigned` | Number of segments assigned for load or drop to the load queue of a server. |`dataSource`,`server` | Varies |
+| `segment/loadQueue/success` |Number of segment assignments that completed successfully.|`dataSource`, `server`|Varies|
+| `segment/loadQueue/cancelled` |Number of segment assignments that were canceled before completion. |`dataSource`,`server` | 0 |
+| `segment/loadQueue/failed` |Number of segment assignments that failed to complete.|`dataSource`, `server`|0|
+| `segment/assigned/broadcast` | |
+| `segment/dropped/broadcast` | |
+
+[14453](https://github.com/apache/druid/pull/14453)
+[13197](https://github.com/apache/druid/pull/13197)
+
+### New metrics for task completion updates
+
+The following metrics have been added:
+* `task/status/queue/count`: Monitors the number of queued items
+* `task/status/updated/count`: Monitors the number of processed items
+
+[14533](https://github.com/apache/druid/pull/14533)
+
+### Added `groupId` to Overlord task metrics
+
+Added `groupId` to task metrics emitted by the Overlord. This is helpful for grouping metrics like task/run/time by a single task group, such as a single compaction task or a single MSQ query.
+
+[14402](https://github.com/apache/druid/pull/14402)
+
+## Cluster management
+
+### Removed unused Coordinator dynamic configuration properties
+
+The following Coordinator dynamic configs have been removed:
+
+* `emitBalancingStats`: Stats for errors encountered while balancing will always be emitted. Other debugging stats will not be emitted but can be logged by setting the appropriate `debugDimensions`.
+* `useBatchedSegmentSampler` and `percentOfSegmentsToConsiderPerMove`: Batched segment sampling is now the standard and will always be on.
+
+[14524](https://github.com/apache/druid/pull/14524)
+
Review Comment:
Seems worth calling out. I had missed adding a release note to the PR description.
```suggestion
### Made Coordinator more resilient to leadership changes
The Coordinator is more resilient to leadership changes now and does not get stuck even if re-election happens while a coordinator run is in progress.
[14385](https://github.com/apache/druid/pull/14385)
```
##########
docs/do-not-merge.md:
##########
@@ -0,0 +1,377 @@
+<!--Intentionally, there's no Apache license so that the GHA fails it. This file is not meant to be merged.
+
+- https://github.com/apache/druid/pull/14266 - we removed input source security from 26 (https://github.com/apache/druid/pull/14003). Should we not include this in 27 release notes?
+
+-->
+
+Apache Druid 27.0.0 contains over $NUMBER_FEATURES new features, bug fixes, performance enhancements, documentation improvements, and additional test coverage from $NUMBER_OF_CONTRIBUTORS contributors.
+
+[See the complete set of changes for additional details]($LINK_TO_RELEASE_MILESTONE).
+
+Review the upgrade notes and incompatible changes before you upgrade to Druid 27.0.0.
+
+# Highlights
+
+<!-- HIGHLIGHTS H2. FOR EACH MAJOR FEATURE FOR THE RELEASE -->
+
+## Query from deep storage (experimental)
+
+Druid now supports querying segments that are stored only in deep storage. When you query from deep storage, you increase the data available for queries without necessarily having to scale your Historical processes to accommodate more data. To take advantage of the potential savings, make sure you configure your load rules to not load all your data onto Historical processes. Then, configure a corresponding drop rule to drop those segments.
+
+Note that at least one segment of a datasource must be loaded onto a Historical process so that Druid can plan the query. It can be any segment though.
+
+As part of this feature, there are new API endpoints available to interact with the query:
+
+- Submit queries using `POST /sql/statements`. The payload is the same as any other Druid SQL query with additional context parameter available.
+ - The required `executionMode: ASYNC` configures Druid to return the results asynchronously. Results are only available after the query completes.
+ - The optional context parameter `selectDestination:DURABLE_STORAGE` configures Druid to write the query results to durable storage.
+
+- Get the status of a query with `GET /sq/statements/:queryID`.
+
+- Get query results with `/sql/statements/:queryID`
+
+[14416](https://github.com/apache/druid/pull/14416)
+[14512](https://github.com/apache/druid/pull/14512)
+[14527](https://github.com/apache/druid/pull/14527)
+
+## Java 17 support
+
+Druid now fully supports Java 17.
+[14384](https://github.com/apache/druid/pull/14384)
+
+## Array column types
+
+Array column types are now generally available.
+
+For more information about this feature, see the [26.0.0 release notes for array column types](https://github.com/apache/druid/releases#26.0.0-highlights-auto-type-column-schema-%28experimental%29).
+
+## Schema auto-discovery
+
+Schema auto-discovery is now generally available.
+
+For more information about this feature, see the [26.0.0 release notes for Schema auto-discovery](https://github.com/apache/druid/releases#26.0.0-highlights-auto-type-column-schema-%28experimental%29-schema-auto-discovery-%28experimental%29).
+
+## Smart segment loading
+
+Revamped Coordinator to make it more stable and user-friendly. This is accompanied by several bug fixes, logging and metric improvements, and a whole new range of capabilities.
+
+Coordinator now supports a `smartSegmentLoading` mode, which is enabled by default. When enabled, users don't need to specify any of the following dynamic configs as they would be ignored by the coordinator. Instead, the coordinator computes the optimal values of the following configs at run time to best utilize coordinator runs:
+
+* `maxSegmentsInNodeLoadingQueue`
+* `maxSegmentsToMove`
+* `replicationThrottleLimit`
+* `useRoundRobinSegmentAssignment`
+* `useBatchedSegmentSampler`
+* `emitBalancingStats`
+
+These configs are now deprecated and will be removed in subsequent releases.
+
+Coordinator is now capable of prioritization and cancellation of items in segment load queues. This means that the coordinator now reacts faster to changes in the cluster and makes better segment assignment decisions.
+
+As part of this change, new metrics are available. For more information, see [New segment metrics](#new-segment-metrics).
+
+[13197](https://github.com/apache/druid/pull/13197)
+
+## Hadoop 2 deprecated
+
+Support for Hadoop 2 is now deprecated. It will be removed in a future release.
+
+# Additional features and improvements
+
+## MSQ task engine
+
+### `maxInputBytesPerWorker` context parameter
+
+The context parameter now denotes the estimated weighted size (in bytes) of the input to split on. The MSQ task engine now takes into account the input format and compression format instead of the actual file size reported by the file system.
+
+The default value for the context parameter has also been changed. It is now 512 MiB (previously 10 GiB).
+
+[14307](https://github.com/apache/druid/pull/14307)
+
+### Improved query planning behavior
+
+Druid now fails query planning if a CLUSTERED BY column contains descending order.
+Previously, queries would successfully plan if any CLUSTERED BY columns contained descending order.
+
+The MSQ fault, `InsertCannotOrderByDescending`, is deprecated. An INSERT or REPLACE query containing a CLUSTERED BY expression cannot be in descending order. Druid's segment generation code only supports ascending order. Instead of the fault, Druid now throws a query `ValidationException`.
+
+[14436](https://github.com/apache/druid/pull/14436)
+
+### SELECT query results
+
+SELECT queries executed using MSQ now generate only a subset of the results in the query reports.
+To fetch the complete result set, run the query using the native engine.
+
+[14370](https://github.com/apache/druid/pull/14370)
+
+
+### Other MSQ improvements
+
+- The same aggregator can now have two output names [14367](https://github.com/apache/druid/pull/14367)
+- Changed the default `clusterStatisticsMergeMode` to SEQUENTIAL (#14310)[https://github.com/apache/druid/pull/14310]
+- Added a query context parameter `MultiStageQueryContext` to determine whether the result of an MSQ select query is limited (#14476)[https://github.com/apache/druid/pull/14476]
+- Enabled using functions as inputs for `index` and `length` parameters (#14480)[https://github.com/apache/druid/pull/14480]
+
+## Ingestion
+
+### New property for task completion updates
+
+The new property `druid.indexer.queue.taskCompleteHandlerNumThreads` controls the number of threads used by the Overlord `TaskQueue` to handle task completion updates received from the workers.
+
+The following metrics have been added:
+* `task/status/queue/count`: Monitors the number of queued items
+* `task/status/updated/count`: Monitors the number of processed items
+
+[14533](https://github.com/apache/druid/pull/14533)
+
+### Improved response to max_allowed_packet limit
+
+If the Overlord fails to insert a task into the metadata because of a payload that exceeds the `max_allowed_packet` limit, the response now returns `400 Bad request` instead of `500 Internal server error`. This prevents an `index_parallel` task from retrying the insertion of a bad sub-task indefinitely and causes it to fail immediately.
+
+[14271](https://github.com/apache/druid/pull/14271)
+
+### Improved handling of mixed type arrays
+
+Druid now handles mixed type arrays such as `[["a", "b", "c"], {"x": 123}]` as `ARRAY<COMPLEX<json>` rather than throwing an incompatible type exception.
+
+[14438](https://github.com/apache/druid/pull/14438)
+
+### Other ingestion improvements
+
+* A negative streaming ingestion lag is no longer emitted as a result of stale offsets. [14292](https://github.com/apache/druid/pull/14292)
+* Removed double synchronization on simple map operations in Kubernetes task runner. [14435](https://github.com/apache/druid/pull/14435)
+* Kubernetes overlord extension now cleans up the job if the task pod fails to come up in time. [14425](https://github.com/apache/druid/pull/14425)
+
+## Querying
+
+### New function for regular expression replacement
+
+The new function `REGEXP_REPLACE` allows you to replace all instances of a pattern with a replacement string.
+
+[14460](https://github.com/apache/druid/pull/14460)
+
+### Query results directory
+
+Druid now supports a `query-results` directory in durable storage to store query results after the task finishes. The auto cleaner does not remove this directory unless the task ID is not known to the Overlord.
+
+[14446](https://github.com/apache/druid/pull/14446)
+
+### Limit for subquery results by memory usage
+
+Users can now add a guardrail to prevent subquery’s results from exceeding the set number of bytes by setting `druid.server.http.maxSubqueryRows` in the Broker's config or `maxSubqueryRows` in the query context. This feature is experimental for now and would default back to row-based limiting in case it fails to get the accurate size of the results consumed by the query.
+
+[13952](https://github.com/apache/druid/pull/13952)
+
+### HLL and Theta sketch estimates
+
+You can now use `HLL_SKETCH_ESTIMATE` and `THETA_SKETCH_ESTIMATE` as expressions. These estimates work on sketch columns and have the same behavior as `postAggs`.
+
+[14312](https://github.com/apache/druid/pull/14312)
+
+### String encoding parameter for HLL sketches
+
+HLL sketches now accept a new `stringEncoding` parameter to build sketches with UTF-8 bytes.
+
+[11201](https://github.com/apache/druid/pull/11201)
+
+### EARLIEST_BY and LATEST_BY signatures
+
+Updated EARLIEST_BY and LATEST_BY function signatures as follows:
+
+* Changed `EARLIEST(expr, timeColumn)` to `EARLIEST_BY(expr, timeColumn)`
+* Changed `LATEST(expr, timeColumn)` to `LATEST_BY(expr, timeColumn)`
+
+[14352](https://github.com/apache/druid/pull/14352)
+
+
+### Exposed Druid functions
+
+Exposed information about Druid functions in the `INFORMATION_SCHEMA.ROUTINES` table. The table contains the following columns:
+
+* ROUTINE_CATALOG: Name of the database. Always set to druid.
+* ROUTINE_SCHEMA: Name of the schema. Always set to INFORMATION_SCHEMA.
+* ROUTINE_NAME: Name of the function or operator. For example: APPROX_COUNT_DISTINCT_DS_THETA.
+* ROUTINE_TYPE: Type of routine. Always set to FUNCTION.
+* IS_AGGREGATOR: Determines if the routine is an aggregator. Returns YES for aggregator functions; NO for scalar functions.
+* SIGNATURES: Possible signatures for the routine as a string.
+
+[14378](https://github.com/apache/druid/pull/14378)
+
+### New Broker configuration for SQL schema migrations
+
+A new broker configuration, `druid.sql.planner.metadataColumnTypeMergePolicy`, adds configurable modes to how column types are computed for the SQL table schema when faced with differences between segments.
+
+The new `leastRestrictive` mode chooses the most appropriate type that data across all segments can best be coerced into and is the default behavior. This is a subtle behavior change around when segment driven schema migrations will take effect for the SQL schema:
+
+- With `leastRestrictive` mode, the schema only updates once all segments are reindexed to the new type.
+- With `latestInterval` mode, the SQL schema gets updated as soon as the first job with the new schema publishes segments in the latest time interval of the data.
+
+`leastRestrictive` can have better query time behavior and eliminates some query time errors that can occur when using `latestInterval`.
+
+[14319](https://github.com/apache/druid/pull/14319)
+
+### EXPLAIN PLAN improvements
+
+The EXPLAIN PLAN result includes a new column `ATTRIBUTES` that describes the attributes of a query.
+See [SQL query translation](docs/querying/sql-translation.md) for more information and examples.
+
+[14391](https://github.com/apache/druid/pull/14391)
+
+
+## Metrics and monitoring
+
+### Added a new SysMonitorOshi monitor
+
+Added a new monitor `SysMonitorOshi` to replace `SysMonitor`. The new monitor has a wider support for different machine architectures including ARM instances. We recommend switching to `SysMonitorOshi` as `SysMonitor` is now deprecated and will be removed in future releases.
+
+[14359](https://github.com/apache/druid/pull/14359)
+
+### Monitor for Overlord and Coordinator service health
+
+Added a new monitor `ServiceStatusMonitor` to monitor the service health of the Overlord and Coordinator.
+
+[14443](https://github.com/apache/druid/pull/14443)
+
+### New segment metrics
+
+The following metrics are now available for Brokers:
+
+|Metric|Description|Dimensions |Normal value|
+|------|-----------|------------|
+|`segment/metadatacache/refresh/count`|Number of segments to refresh in broker segment metadata cache. Emitted once per refresh per datasource.|`dataSource`| |
+|`segment/metadatacache/refresh/time`|Time taken to refresh segments in broker segment metadata cache. Emitted once per refresh per datasource.|`dataSource`| |
+| `segment/loadQueue/assigned` | Number of segments assigned for load or drop to the load queue of a server. |`dataSource`,`server` | Varies |
+| `segment/loadQueue/success` |Number of segment assignments that completed successfully.|`dataSource`, `server`|Varies|
+| `segment/loadQueue/cancelled` |Number of segment assignments that were canceled before completion. |`dataSource`,`server` | 0 |
+| `segment/loadQueue/failed` |Number of segment assignments that failed to complete.|`dataSource`, `server`|0|
+| `segment/assigned/broadcast` | |
+| `segment/dropped/broadcast` | |
+
+[14453](https://github.com/apache/druid/pull/14453)
+[13197](https://github.com/apache/druid/pull/13197)
+
+### New metrics for task completion updates
+
+The following metrics have been added:
+* `task/status/queue/count`: Monitors the number of queued items
+* `task/status/updated/count`: Monitors the number of processed items
+
+[14533](https://github.com/apache/druid/pull/14533)
+
+### Added `groupId` to Overlord task metrics
+
+Added `groupId` to task metrics emitted by the Overlord. This is helpful for grouping metrics like task/run/time by a single task group, such as a single compaction task or a single MSQ query.
+
+[14402](https://github.com/apache/druid/pull/14402)
+
+## Cluster management
+
+### Removed unused Coordinator dynamic configuration properties
+
+The following Coordinator dynamic configs have been removed:
+
+* `emitBalancingStats`: Stats for errors encountered while balancing will always be emitted. Other debugging stats will not be emitted but can be logged by setting the appropriate `debugDimensions`.
+* `useBatchedSegmentSampler` and `percentOfSegmentsToConsiderPerMove`: Batched segment sampling is now the standard and will always be on.
+
+[14524](https://github.com/apache/druid/pull/14524)
+
+### Improved segment balancing strategy
+
+The `cost` balancer strategy performs much better now and is capable of moving more segments in a single Coordinator run. These improvements were made by borrowing ideas from the `cachingCost` strategy.
+
+The `cachingCost` strategy itself has been known to cause issues and is now deprecated. It is likely to be removed in future Druid releases and must not be used on any cluster.
+
+[14484](https://github.com/apache/druid/pull/14484)
+
+### Enabled empty tiered replicants for load rules
+
+Druid now allows empty tiered replicants in load rules.
+
+[14432](https://github.com/apache/druid/pull/14432)
+
+### Changed Coordinator configs values
+
+The following Coordinator dynamic configs have new default values:
+
+* `maxsegmentsInNodeLoadingQueue` : 500, previously 100
+* `maxSegmentsToMove`: 100, previously 5
+* `replicationThrottleLimit`: 500, previously 10
+
+These new defaults can improve performance for most use cases.
+
+[14269](https://github.com/apache/druid/pull/14269)
+
Review Comment:
```suggestion
### Stabilized initialization of `HttpServerInventoryView`
The initialization of `HttpServerInventoryView` maintained by Brokers and Coordinator is now resilient to Historicals and Peons crashing. The crashed servers are simply marked as stopped and not waited upon during the initialization.
New metrics have been added to further monitor the sync status of `HttpServerInventoryView` with different servers.
[14517](https://github.com/apache/druid/pull/14517)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] DO NOT MERGE - 27.0.0 WIP release notes (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #14600:
URL: https://github.com/apache/druid/pull/14600#discussion_r1277103011
##########
docs/do-not-merge.md:
##########
@@ -53,20 +53,21 @@ For more information about this feature, see the [26.0.0 release notes for Schem
## Smart segment loading
-Revamped Coordinator to make it more stable and user-friendly. This is accompanied by several bug fixes, logging and metric improvements, and a whole new range of capabilities.
+The Coordinator is now more stable and user-friendly. This is accompanied by several bug fixes, logging and metric improvements, and a whole new range of capabilities.
Review Comment:
"The Coordinator" seems to have been repeated several times in this section, can we reduce it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] DO NOT MERGE - 27.0.0 WIP release notes (druid)
Posted by "abhishekagarwal87 (via GitHub)" <gi...@apache.org>.
abhishekagarwal87 commented on code in PR #14600:
URL: https://github.com/apache/druid/pull/14600#discussion_r1286606562
##########
docs/do-not-merge.md:
##########
@@ -182,163 +194,140 @@ Updated EARLIEST_BY and LATEST_BY function signatures as follows:
* Changed `EARLIEST(expr, timeColumn)` to `EARLIEST_BY(expr, timeColumn)`
* Changed `LATEST(expr, timeColumn)` to `LATEST_BY(expr, timeColumn)`
-[14352](https://github.com/apache/druid/pull/14352)
+[#14352](https://github.com/apache/druid/pull/14352)
+### INFORMATION_SCHEMA.ROUTINES TABLE
-### Exposed Druid functions
+Use the new `INFORMATION_SCHEMA.ROUTINES` to programmatically get information about the functions that Druid SQL supports.
-Exposed information about Druid functions in the `INFORMATION_SCHEMA.ROUTINES` table. The table contains the following columns:
+For more information, such as the available columns, see [ROUTINES table](https://druid.apache.org/docs/latest/querying/sql-metadata-tables.html#routines-table)
-* ROUTINE_CATALOG: Name of the database. Always set to druid.
-* ROUTINE_SCHEMA: Name of the schema. Always set to INFORMATION_SCHEMA.
-* ROUTINE_NAME: Name of the function or operator. For example: APPROX_COUNT_DISTINCT_DS_THETA.
-* ROUTINE_TYPE: Type of routine. Always set to FUNCTION.
-* IS_AGGREGATOR: Determines if the routine is an aggregator. Returns YES for aggregator functions; NO for scalar functions.
-* SIGNATURES: Possible signatures for the routine as a string.
-
-[14378](https://github.com/apache/druid/pull/14378)
+[#14378](https://github.com/apache/druid/pull/14378)
### New Broker configuration for SQL schema migrations
-A new broker configuration, `druid.sql.planner.metadataColumnTypeMergePolicy`, adds configurable modes to how column types are computed for the SQL table schema when faced with differences between segments.
+You can now better control how Druid reacts to schema changes between segments. This can make querying more resilient when newer segments introduce different types, such as if a column previously contained LONG values and newer segments contain STRING.
-The new `leastRestrictive` mode chooses the most appropriate type that data across all segments can best be coerced into and is the default behavior. This is a subtle behavior change around when segment driven schema migrations will take effect for the SQL schema:
+Use the new Broker configuration, `druid.sql.planner.metadataColumnTypeMergePolicy` to control how column types are computed for the SQL table schema when faced with differences between segments.
-- With `leastRestrictive` mode, the schema only updates once all segments are reindexed to the new type.
-- With `latestInterval` mode, the SQL schema gets updated as soon as the first job with the new schema publishes segments in the latest time interval of the data.
+Set it to one of the following:
+- `leastRestrictive`: the schema only updates once all segments are reindexed to the new type.
+- `latestInterval`: the SQL schema gets updated as soon as the first job with the new schema publishes segments in the latest time interval of the data.
+
`leastRestrictive` can have better query time behavior and eliminates some query time errors that can occur when using `latestInterval`.
-[14319](https://github.com/apache/druid/pull/14319)
+[#14319](https://github.com/apache/druid/pull/14319)
### EXPLAIN PLAN improvements
The EXPLAIN PLAN result includes a new column `ATTRIBUTES` that describes the attributes of a query.
-See [SQL query translation](docs/querying/sql-translation.md) for more information and examples.
-[14391](https://github.com/apache/druid/pull/14391)
+For more information, see [SQL translation](https://druid.apache.org/docs/latest/querying/sql-translation.html)
+[#14391](https://github.com/apache/druid/pull/14391)
-## Metrics and monitoring
+### New query filters
-### Added a new SysMonitorOshi monitor
+TBD
-Added a new monitor `SysMonitorOshi` to replace `SysMonitor`. The new monitor has a wider support for different machine architectures including ARM instances. We recommend switching to `SysMonitorOshi` as `SysMonitor` is now deprecated and will be removed in future releases.
+## Metrics and monitoring
-[14359](https://github.com/apache/druid/pull/14359)
### Monitor for Overlord and Coordinator service health
Added a new monitor `ServiceStatusMonitor` to monitor the service health of the Overlord and Coordinator.
-[14443](https://github.com/apache/druid/pull/14443)
+[#14443](https://github.com/apache/druid/pull/14443)
+
+### New Broker metrics
-### New segment metrics
+TBD for name changes
The following metrics are now available for Brokers:
-|Metric|Description|Dimensions|Normal value|
-|------|-----------|----------|------------|
+|Metric|Description|Dimensions|
+|------|-----------|----------|
|`segment/metadatacache/refresh/count`|Number of segments to refresh in broker segment metadata cache. Emitted once per refresh per datasource.|`dataSource`| |
|`segment/metadatacache/refresh/time`|Time taken to refresh segments in broker segment metadata cache. Emitted once per refresh per datasource.|`dataSource`| |
+
+[#14453](https://github.com/apache/druid/pull/14453)
+
+### New Coordinator metrics
+
+|Metric|Description|Dimensions|Normal value|
+|------|-----------|----------| --|
| `segment/loadQueue/assigned` | Number of segments assigned for load or drop to the load queue of a server. |`dataSource`,`server` | Varies |
| `segment/loadQueue/success` |Number of segment assignments that completed successfully.|`dataSource`, `server`|Varies|
| `segment/loadQueue/cancelled` |Number of segment assignments that were canceled before completion. |`dataSource`,`server` | 0 |
| `segment/loadQueue/failed` |Number of segment assignments that failed to complete.|`dataSource`, `server`|0|
-[14453](https://github.com/apache/druid/pull/14453)
-[13197](https://github.com/apache/druid/pull/13197)
+[#13197](https://github.com/apache/druid/pull/13197)
### New metrics for task completion updates
-The following metrics have been added:
-* `task/status/queue/count`: Monitors the number of queued items
-* `task/status/updated/count`: Monitors the number of processed items
+| Metric | Description | Normal value |
+| ------ | ----------- | ---------- |
+| `task/status/queue/count` | Monitors the number of queued items | Varies |
+| `task/status/updated/count` | Monitors the number of processed items | Varies |
-[14533](https://github.com/apache/druid/pull/14533)
+[#14533](https://github.com/apache/druid/pull/14533)
### Added `groupId` to Overlord task metrics
Added `groupId` to task metrics emitted by the Overlord. This is helpful for grouping metrics like task/run/time by a single task group, such as a single compaction task or a single MSQ query.
-[14402](https://github.com/apache/druid/pull/14402)
+[#14402](https://github.com/apache/druid/pull/14402)
### New metrics for monitoring sync status of `HttpServerInventoryView`
+TBD for name change
+
|Metric|Description|Dimensions|Normal value|
|------|-----------|-----------|----------|
-|`segment/serverview/sync/healthy`|Sync status of the Coordinator/Broker with a segment-loading server such as a Historical or Peon. Emitted by the Coordinator and Brokers only when [HTTP-based server view](../configuration/index.md#segment-management) is enabled. This metric can be used in conjunction with `segment/serverview/sync/unstableTime` to debug slow startup of the Coordinator.|`server`, `tier`|1 for fully synced servers, 0 otherwise|
-|`segment/serverview/sync/unstableTime`|Time in milliseconds for which the Coordinator/Broker has been failing to sync with a segment-loading server. Emitted by the Coordinator and Brokers only when [HTTP-based server view](../configuration/index.md#segment-management) is enabled.|`server`, `tier`|Not emitted for synced servers.|
-## Cluster management
-
-### Removed unused Coordinator dynamic configuration properties
+|`segment/serverview/sync/healthy`|Sync status of the Coordinator/Broker with a segment-loading server such as a Historical or Peon. Emitted by the Coordinator and Brokers only when HTTP-based server view is enabled. This metric can be used in conjunction with `segment/serverview/sync/unstableTime` to debug slow startup of the Coordinator.|`server`, `tier`|1 for fully synced servers, 0 otherwise|
+|`segment/serverview/sync/unstableTime`|Time in milliseconds for which the Coordinator/Broker has been failing to sync with a segment-loading server. Emitted by the Coordinator and Brokers only when HTTP-based server view is enabled.|`server`, `tier`|Not emitted for synced servers.|
Review Comment:
```suggestion
|`serverview/sync/healthy`|Sync status of the Coordinator/Broker with a segment-loading server such as a Historical or Peon. Emitted by the Coordinator and Brokers only when HTTP-based server view is enabled. This metric can be used in conjunction with `serverview/sync/unstableTime` to debug slow startup of the Coordinator.|`server`, `tier`|1 for fully synced servers, 0 otherwise|
|`serverview/sync/unstableTime`|Time in milliseconds for which the Coordinator/Broker has been failing to sync with a segment-loading server. Emitted by the Coordinator and Brokers only when HTTP-based server view is enabled.|`server`, `tier`|Not emitted for synced servers.|
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] DO NOT MERGE - 27.0.0 WIP release notes (druid)
Posted by "vogievetsky (via GitHub)" <gi...@apache.org>.
vogievetsky commented on code in PR #14600:
URL: https://github.com/apache/druid/pull/14600#discussion_r1282699063
##########
docs/do-not-merge.md:
##########
@@ -0,0 +1,466 @@
+<!--Intentionally, there's no Apache license so that the GHA fails it. This file is not meant to be merged.
+
+- https://github.com/apache/druid/pull/14266 - we removed input source security from 26 (https://github.com/apache/druid/pull/14003). Should we not include this in 27 release notes?
+
+-->
+
+# Still need
+
+search for TBD
+
+* Caveats for array column types
+* Caching cost for upgrade notes - is it this? [#14484](https://github.com/apache/druid/pull/14484)
+* User impact of https://github.com/apache/druid/pull/14048 for highlights
+* https://github.com/apache/druid/pull/14476 seems to only be documented here in release notes and the PR?
+* Vad's input for web console section
Review Comment:
How about:
## Web console
### Console now uses the new statements API for all MSQ interaction
The console uses the new async statements API for all `sql-msq-task` engine queries.
While this has relatively few impacts on the UX of the query view you are invited to peek under the hood and check out the new network requests being sent as a working example of the new API.
There are some fun UX benefits: you can now specify `durableStorage` as the result destination for `SELECT` queries (when durable storage is configured).
<img width="655" alt="image" src="https://github.com/apache/druid/assets/177816/9399dcbd-454a-451f-bb5a-6a655025fffe">
After running a `SELECT` query that wrote its resuls to `durableStorage` you can download the full, unlimited, result set directly from the broker.
https://github.com/apache/druid/pull/14540
https://github.com/apache/druid/pull/14669
https://github.com/apache/druid/pull/14712
### Added UI around data source with zero replica segments
This release of Druid supports having datasources with segments that are not replicated on any historicals, the console now had special handling for datasources like that.
<img width="506" alt="image" src="https://github.com/apache/druid/assets/177816/875014ab-a4f9-4672-bf3b-df2f21d8bdfa">
[https://github.com/apache/druid/pull/14540]
### Added a Explore view
The Explore view is a simple, stateless, SQL backed, data exploration view to the web console. It lets users explore data in Druid with point-and-click interaction and visualizations (instead of writing SQL and looking at a table). This can provide faster time-to-value for a user new to Druid and can allow a Druid veteran to quickly chart some data that they care about.
The Explore view is accessible from the more button in the header.
<img width="1115" alt="image" src="https://github.com/apache/druid/assets/177816/11bfd317-f2a9-4b39-b3f6-ab710c2cafe3">
https://github.com/apache/druid/pull/14602
### Added a dialog for viewing and setting the dynamic compaction config
[https://github.com/apache/druid/pull/14540]
### Other changes
- Replaced the **Ingestion** view with two views: **Supervisors** and **Tasks** [#14395](https://github.com/apache/druid/pull/14395)
- Added 'Start time' column to the Services tab [https://github.com/apache/druid/pull/13358]
- Added 'Replication factor' column to the Segments view [#14403](https://github.com/apache/druid/pull/14403)
- Added stateful filter URLs for all views [https://github.com/apache/druid/pull/14395]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] DO NOT MERGE - 27.0.0 WIP release notes (druid)
Posted by "abhishekrb19 (via GitHub)" <gi...@apache.org>.
abhishekrb19 commented on code in PR #14600:
URL: https://github.com/apache/druid/pull/14600#discussion_r1266235885
##########
docs/do-not-merge.md:
##########
@@ -0,0 +1,366 @@
+<!--Intentionally, there's no Apache license so that the GHA fails it. This file is not meant to be merged.
+
+- https://github.com/apache/druid/pull/14266 - we removed input source security from 26 (https://github.com/apache/druid/pull/14003). Should we not include this in 27 release notes?
+
+-->
+
+Apache Druid 27.0.0 contains over $NUMBER_FEATURES new features, bug fixes, performance enhancements, documentation improvements, and additional test coverage from $NUMBER_OF_CONTRIBUTORS contributors.
+
+[See the complete set of changes for additional details]($LINK_TO_RELEASE_MILESTONE).
+
+Review the upgrade notes and incompatible changes before you upgrade to Druid 27.0.0.
+
+# Highlights
+
+<!-- HIGHLIGHTS H2. FOR EACH MAJOR FEATURE FOR THE RELEASE -->
+
+## Query from deep storage
+
+TBD
+
+### New statements API
+
+Added a new API /druid/v2/sql/statements/ which allows users to fetch results in an asynchronous manner.
+
+[14416](https://github.com/apache/druid/pull/14416)
+
+### New and updated fields in API to get results
+
+The API response to get results now returns a `pages` field containing information on each page of the results status.
+The `numRows` and `sizeInBytes` fields are renamed to `numTotalRows` and `totalSizeInBytes`, respectively.
+
+[14512](https://github.com/apache/druid/pull/14512)
+
+
+### Durable storage for results
+
+MSQ can now write SELECT query results to durable storage. To do so, set the context flag `selectDestination:DURABLE_STORAGE` while issuing SELECT queries to MSQ.
+
+[14527](https://github.com/apache/druid/pull/14527)
+
+## Java 17 support
+
+Druid now fully supports Java 17.
+[14384](https://github.com/apache/druid/pull/14384)
+
+## Array column types
+
+GA + TBD
+
+## Schema auto-discovery
+
+GA + TBD
+
+## Smart segment loading
+
+[13197](https://github.com/apache/druid/pull/13197)
+
+## Hadoop 2 support dropped
+
+# Additional features and improvements
+
+## MSQ task engine
+
+### `maxInputBytesPerWorker` context parameter
+
+The context parameter now denotes the estimated weighted size (in bytes) of the input to split on. The MSQ task engine now takes into account the input format and compression format instead of the actual file size reported by the file system.
+
+The default value for the context parameter has also been changed. It is now 512 MiB (previously 10 GiB).
+
+[14307](https://github.com/apache/druid/pull/14307)
+
+### Improved query planning behavior
+
+Druid now fails query planning if a CLUSTERED BY column contains descending order.
+Previously, queries would successfully plan if any CLUSTERED BY columns contained descending order.
+
+The MSQ fault, `InsertCannotOrderByDescending`, is deprecated. An INSERT or REPLACE query containing a CLUSTERED BY expression cannot be in descending order. Druid's segment generation code only supports ascending order. Instead of the fault, Druid now throws a query `ValidationException`.
+
+[14436](https://github.com/apache/druid/pull/14436)
+
+### SELECT query results
+
+SELECT queries executed using MSQ now generate only a subset of the results in the query reports.
+To fetch the complete result set, run the query using the native engine.
+
+[14370](https://github.com/apache/druid/pull/14370)
+
+
+### Other MSQ improvements
+
+- The same aggregator can now have two output names [14367](https://github.com/apache/druid/pull/14367)
+- Changed the default `clusterStatisticsMergeMode` to SEQUENTIAL (#14310)[https://github.com/apache/druid/pull/14310]
+- Added a query context parameter `MultiStageQueryContext` to determine whether the result of an MSQ select query is limited (#14476)[https://github.com/apache/druid/pull/14476]
+- Enabled using functions as inputs for `index` and `length` parameters (#14480)[https://github.com/apache/druid/pull/14480]
+
+## Ingestion
+
+### New property for task completion updates
+
+The new property `druid.indexer.queue.taskCompleteHandlerNumThreads` controls the number of threads used by the Overlord `TaskQueue` to handle task completion updates received from the workers.
+
+The following metrics have been added:
+* `task/status/queue/count`: Monitors the number of queued items
+* `task/status/updated/count`: Monitors the number of processed items
+
+[14533](https://github.com/apache/druid/pull/14533)
+
+### Improved response to max_allowed_packet limit
+
+If the Overlord fails to insert a task into the metadata because of a payload that exceeds the `max_allowed_packet` limit, the response now returns `400 Bad request` instead of `500 Internal server error`. This prevents an `index_parallel` task from retrying the insertion of a bad sub-task indefinitely and causes it to fail immediately.
+
+[14271](https://github.com/apache/druid/pull/14271)
+
+### Improved handling of mixed type arrays
+
+Druid now handles mixed type arrays such as `[["a", "b", "c"], {"x": 123}]` as `ARRAY<COMPLEX<json>` rather than throwing an incompatible type exception.
+
+[14438](https://github.com/apache/druid/pull/14438)
+
+### Other ingestion improvements
+
+* A negative streaming ingestion lag is no longer emitted as a result of stale offsets. [14292](https://github.com/apache/druid/pull/14292)
+* Removed double synchronization on simple map operations in Kubernetes task runner. [14435](https://github.com/apache/druid/pull/14435)
+* Kubernetes overlord extension now cleans up the job if the task pod fails to come up in time. [14425](https://github.com/apache/druid/pull/14425)
+
+## Querying
+
+### New function for regular expression replacement
+
+The new function `REGEXP_REPLACE` allows you to replace all instances of a pattern with a replacement string.
+
+[14460](https://github.com/apache/druid/pull/14460)
+
+### Query results directory
+
+Druid now supports a `query-results` directory in durable storage to store query results after the task finishes. The auto cleaner does not remove this directory unless the task ID is not known to the Overlord.
+
+[14446](https://github.com/apache/druid/pull/14446)
+
+### Limit for subquery results by memory usage
+
+Users can now add a guardrail to prevent subquery’s results from exceeding the set number of bytes by setting `druid.server.http.maxSubqueryRows` in the Broker's config or `maxSubqueryRows` in the query context. This feature is experimental for now and would default back to row-based limiting in case it fails to get the accurate size of the results consumed by the query.
+
+[13952](https://github.com/apache/druid/pull/13952)
+
+### HLL and Theta sketch estimates
+
+You can now use HLL_SKETCH_ESTIMATE and THETA_SKETCH_ESTIMATE as expressions. These estimates work on sketch columns and have the same behavior as `postAggs`.
Review Comment:
```suggestion
You can now use `HLL_SKETCH_ESTIMATE` and `THETA_SKETCH_ESTIMATE` as expressions. These estimates work on sketch columns and have the same behavior as `postAggs`.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] DO NOT MERGE - 27.0.0 WIP release notes (druid)
Posted by "abhishekrb19 (via GitHub)" <gi...@apache.org>.
abhishekrb19 commented on code in PR #14600:
URL: https://github.com/apache/druid/pull/14600#discussion_r1266233222
##########
docs/do-not-merge.md:
##########
@@ -0,0 +1,366 @@
+<!--Intentionally, there's no Apache license so that the GHA fails it. This file is not meant to be merged.
+
+- https://github.com/apache/druid/pull/14266 - we removed input source security from 26 (https://github.com/apache/druid/pull/14003). Should we not include this in 27 release notes?
+
+-->
+
+Apache Druid 27.0.0 contains over $NUMBER_FEATURES new features, bug fixes, performance enhancements, documentation improvements, and additional test coverage from $NUMBER_OF_CONTRIBUTORS contributors.
+
+[See the complete set of changes for additional details]($LINK_TO_RELEASE_MILESTONE).
+
+Review the upgrade notes and incompatible changes before you upgrade to Druid 27.0.0.
+
+# Highlights
+
+<!-- HIGHLIGHTS H2. FOR EACH MAJOR FEATURE FOR THE RELEASE -->
+
+## Query from deep storage
+
+TBD
+
+### New statements API
+
+Added a new API /druid/v2/sql/statements/ which allows users to fetch results in an asynchronous manner.
+
+[14416](https://github.com/apache/druid/pull/14416)
+
+### New and updated fields in API to get results
+
+The API response to get results now returns a `pages` field containing information on each page of the results status.
+The `numRows` and `sizeInBytes` fields are renamed to `numTotalRows` and `totalSizeInBytes`, respectively.
+
+[14512](https://github.com/apache/druid/pull/14512)
+
+
+### Durable storage for results
+
+MSQ can now write SELECT query results to durable storage. To do so, set the context flag `selectDestination:DURABLE_STORAGE` while issuing SELECT queries to MSQ.
+
+[14527](https://github.com/apache/druid/pull/14527)
+
+## Java 17 support
+
+Druid now fully supports Java 17.
+[14384](https://github.com/apache/druid/pull/14384)
+
+## Array column types
+
+GA + TBD
+
+## Schema auto-discovery
+
+GA + TBD
+
+## Smart segment loading
+
+[13197](https://github.com/apache/druid/pull/13197)
+
+## Hadoop 2 support dropped
+
+# Additional features and improvements
+
+## MSQ task engine
+
+### `maxInputBytesPerWorker` context parameter
+
+The context parameter now denotes the estimated weighted size (in bytes) of the input to split on. The MSQ task engine now takes into account the input format and compression format instead of the actual file size reported by the file system.
+
+The default value for the context parameter has also been changed. It is now 512 MiB (previously 10 GiB).
+
+[14307](https://github.com/apache/druid/pull/14307)
+
+### Improved query planning behavior
+
+Druid now fails query planning if a CLUSTERED BY column contains descending order.
+Previously, queries would successfully plan if any CLUSTERED BY columns contained descending order.
+
+The MSQ fault, `InsertCannotOrderByDescending`, is deprecated. An INSERT or REPLACE query containing a CLUSTERED BY expression cannot be in descending order. Druid's segment generation code only supports ascending order. Instead of the fault, Druid now throws a query `ValidationException`.
+
+[14436](https://github.com/apache/druid/pull/14436)
+
+### SELECT query results
+
+SELECT queries executed using MSQ now generate only a subset of the results in the query reports.
+To fetch the complete result set, run the query using the native engine.
+
+[14370](https://github.com/apache/druid/pull/14370)
+
+
+### Other MSQ improvements
+
+- The same aggregator can now have two output names [14367](https://github.com/apache/druid/pull/14367)
+- Changed the default `clusterStatisticsMergeMode` to SEQUENTIAL (#14310)[https://github.com/apache/druid/pull/14310]
+- Added a query context parameter `MultiStageQueryContext` to determine whether the result of an MSQ select query is limited (#14476)[https://github.com/apache/druid/pull/14476]
+- Enabled using functions as inputs for `index` and `length` parameters (#14480)[https://github.com/apache/druid/pull/14480]
+
+## Ingestion
+
+### New property for task completion updates
+
+The new property `druid.indexer.queue.taskCompleteHandlerNumThreads` controls the number of threads used by the Overlord `TaskQueue` to handle task completion updates received from the workers.
+
+The following metrics have been added:
+* `task/status/queue/count`: Monitors the number of queued items
+* `task/status/updated/count`: Monitors the number of processed items
+
+[14533](https://github.com/apache/druid/pull/14533)
+
+### Improved response to max_allowed_packet limit
+
+If the Overlord fails to insert a task into the metadata because of a payload that exceeds the `max_allowed_packet` limit, the response now returns `400 Bad request` instead of `500 Internal server error`. This prevents an `index_parallel` task from retrying the insertion of a bad sub-task indefinitely and causes it to fail immediately.
+
+[14271](https://github.com/apache/druid/pull/14271)
+
+### Improved handling of mixed type arrays
+
+Druid now handles mixed type arrays such as `[["a", "b", "c"], {"x": 123}]` as `ARRAY<COMPLEX<json>` rather than throwing an incompatible type exception.
+
+[14438](https://github.com/apache/druid/pull/14438)
+
+### Other ingestion improvements
+
+* A negative streaming ingestion lag is no longer emitted as a result of stale offsets. [14292](https://github.com/apache/druid/pull/14292)
+* Removed double synchronization on simple map operations in Kubernetes task runner. [14435](https://github.com/apache/druid/pull/14435)
+* Kubernetes overlord extension now cleans up the job if the task pod fails to come up in time. [14425](https://github.com/apache/druid/pull/14425)
+
+## Querying
+
+### New function for regular expression replacement
+
+The new function `REGEXP_REPLACE` allows you to replace all instances of a pattern with a replacement string.
+
+[14460](https://github.com/apache/druid/pull/14460)
+
+### Query results directory
+
+Druid now supports a `query-results` directory in durable storage to store query results after the task finishes. The auto cleaner does not remove this directory unless the task ID is not known to the Overlord.
+
+[14446](https://github.com/apache/druid/pull/14446)
+
+### Limit for subquery results by memory usage
+
+Users can now add a guardrail to prevent subquery’s results from exceeding the set number of bytes by setting `druid.server.http.maxSubqueryRows` in the Broker's config or `maxSubqueryRows` in the query context. This feature is experimental for now and would default back to row-based limiting in case it fails to get the accurate size of the results consumed by the query.
+
+[13952](https://github.com/apache/druid/pull/13952)
+
+### HLL and Theta sketch estimates
+
+You can now use HLL_SKETCH_ESTIMATE and THETA_SKETCH_ESTIMATE as expressions. These estimates work on sketch columns and have the same behavior as `postAggs`.
+
+[14312](https://github.com/apache/druid/pull/14312)
+
+### String encoding parameter for HLL sketches
+
+HLL sketches now accept a new `stringEncoding` parameter to build sketches with UTF-8 bytes.
+
+[11201](https://github.com/apache/druid/pull/11201)
+
+### EARLIEST_BY and LATEST_BY signatures
+
+Updated EARLIEST_BY and LATEST_BY function signatures as follows:
+
+* Changed `EARLIEST(expr, timeColumn)` to `EARLIEST_BY(expr, timeColumn)`
+* Changed `LATEST(expr, timeColumn)` to `LATEST_BY(expr, timeColumn)`
+
+[14352](https://github.com/apache/druid/pull/14352)
+
+
+### Exposed Druid functions
+
+Exposed information about Druid functions in the `INFORMATION_SCHEMA.ROUTINES` table. The table contains the following columns:
+
+* ROUTINE_CATALOG: Name of the database. Always set to druid.
+* ROUTINE_SCHEMA: Name of the schema. Always set to INFORMATION_SCHEMA.
+* ROUTINE_NAME: Name of the function or operator. For example: APPROX_COUNT_DISTINCT_DS_THETA.
+* ROUTINE_TYPE: Type of routine. Always set to FUNCTION.
+* IS_AGGREGATOR: Determines if the routine is an aggregator. Returns YES for aggregator functions; NO for scalar functions.
+* SIGNATURES: Possible signatures for the routine as a string.
+
+[14378](https://github.com/apache/druid/pull/14378)
+
+### New Broker configuration for SQL schema migrations
+
+A new broker configuration, `druid.sql.planner.metadataColumnTypeMergePolicy`, adds configurable modes to how column types are computed for the SQL table schema when faced with differences between segments.
+
+The new `leastRestrictive` mode chooses the most appropriate type that data across all segments can best be coerced into and is the default behavior. This is a subtle behavior change around when segment driven schema migrations will take effect for the SQL schema:
+
+- With `leastRestrictive` mode, the schema only updates once all segments are reindexed to the new type.
+- With `latestInterval` mode, the SQL schema gets updated as soon as the first job with the new schema publishes segments in the latest time interval of the data.
+
+`leastRestrictive` can have better query time behavior and eliminates some query time errors that can occur when using `latestInterval`.
+
+[14319](https://github.com/apache/druid/pull/14319)
+
+### EXPLAIN PLAN improvements
+
+The EXPLAIN PLAN result includes a new column `ATTRIBUTES` that describes the attributes of a query.
+See [SQL query translation](docs/querying/sql-translation.md) for more information and examples.
+
+[14391](https://github.com/apache/druid/pull/14391)
+
+
+## Metrics and monitoring
+
+### Added a new SysMonitorOshi monitor
+
+Added a new monitor `SysMonitorOshi` to replace `SysMonitor`. The new monitor has a wider support for different machine architectures including ARM instances. We recommend switching to `SysMonitorOshi` as `SysMonitor` is now deprecated and will be removed in future releases.
+
+[14359](https://github.com/apache/druid/pull/14359)
+
+### Monitor for Overlord and Coordinator service health
+
+Added a new monitor `ServiceStatusMonitor` to monitor the service health of the Overlord and Coordinator.
+
+[14443](https://github.com/apache/druid/pull/14443)
+
+### New segment metrics
+
+The following metrics are now available for Brokers:
+
+|Metric|Description|Normal value|
+|------|-----------|------------|
+|`segment/metadatacache/refresh/count`|Number of segments to refresh in broker segment metadata cache. Emitted once per refresh per datasource.|`dataSource`|
+|`segment/metadatacache/refresh/time`|Time taken to refresh segments in broker segment metadata cache. Emitted once per refresh per datasource.|`dataSource`|
+
+[14453](https://github.com/apache/druid/pull/14453)
+
+### New metrics for task completion updates
+
+The following metrics have been added:
+* `task/status/queue/count`: Monitors the number of queued items
+* `task/status/updated/count`: Monitors the number of processed items
+
+[14533](https://github.com/apache/druid/pull/14533)
+
+### Added `groupId` to Overlord task metrics
+
+Added `groupId` to task metrics emitted by the Overlord. This is helpful for grouping metrics like task/run/time by a single task group, such as a single compaction task or a single MSQ query.
+
+[14402](https://github.com/apache/druid/pull/14402)
+
+## Cluster management
+
+### Enabled cancellation and prioritization of load queue items
+
+Revamped Coordinator to make it more stable and user-friendly. This is accompanied by several bug fixes, logging and metric improvements, and a whole new range of capabilities.
+
+Coordinator now supports a `smartSegmentLoading` mode, which is enabled by default. When enabled, users don't need to specify any of the following dynamic configs as they would be ignored by the coordinator. Instead, the coordinator computes the optimal values of the following configs at run time to best utilize coordinator runs:
+
+* `maxSegmentsInNodeLoadingQueue`
+* `maxSegmentsToMove`
+* `replicationThrottleLimit`
+* `useRoundRobinSegmentAssignment`
+* `useBatchedSegmentSampler`
+* `emitBalancingStats`
+
+These configs are now deprecated and will be removed in subsequent releases.
+
+Coordinator is now capable of prioritization and cancellation of items in segment load queues. This means that the coordinator now reacts faster to changes in the cluster and makes better segment assignment decisions.
+
+[13197](https://github.com/apache/druid/pull/13197)
+
+### Removed unused Coordinator dynamic configuration properties
+
+The following Coordinator dynamic configs have been removed:
+
+* `emitBalancingStats`: Stats for errors encountered while balancing will always be emitted. Other debugging stats will not be emitted but can be logged by setting the appropriate `debugDimensions`.
+* `useBatchedSegmentSampler` and `percentOfSegmentsToConsiderPerMove`: Batched segment sampling is now the standard and will always be on.
+
+[14524](https://github.com/apache/druid/pull/14524)
+
+### Improved segment balancing strategy
+
+The `cost` balancer strategy performs much better now and is capable of moving more segments in a single Coordinator run. These improvements were made by borrowing ideas from the `cachingCost` strategy.
+
+The `cachingCost` strategy itself has been known to cause issues and is now deprecated. It is likely to be removed in future Druid releases and must not be used on any cluster.
+
+[14484](https://github.com/apache/druid/pull/14484)
+
+### Enabled empty tiered replicants for load rules
+
+Druid now allows empty tiered replicants in load rules.
+
+[14432](https://github.com/apache/druid/pull/14432)
+
+### Changed Coordinator configs values
+
+The following Coordinator dynamic configs have new default values:
+
+* `maxsegmentsInNodeLoadingQueue` : 500, previously 100
+* `maxSegmentsToMove`: 100, previously 5
+* `replicationThrottleLimit`: 500, previously 10
+
+These new defaults can improve performance for most use cases.
+
+[14269](https://github.com/apache/druid/pull/14269)
+
+### Improved error messages
+
+Introduced a new unified exception, `DruidException`, for surfacing errors. It is partially compatible with the old way of reporting error messages. Response codes remain the same, all fields that previously existed on the response will continue to exist and be populated, including `errorMessage`. Some error messages have changed to be more consumable by humans and some cases have the message restructured. There should be no impact to the response codes.
+
+[14004](https://github.com/apache/druid/pull/14004)
+
+## Web console
+
+### Supervisors and Tasks
+
+Replaced the **Ingestion** view with two views: **Supervisors** and **Tasks**.
+
+[14395](https://github.com/apache/druid/pull/14395)
+
+### Added replication factor column
+
+Added a new virtual column `replication_factor` to the `sys.segments` table. This returns the total number of replicants of the segment across all tiers. The column is set to -1 if the information is not available.
+
+[14403](https://github.com/apache/druid/pull/14403)
+
+### Other web console improvements
+
+## Extensions
+
+### Improved segment metadata for Kafka emitter extension
+
+The Kafka emitter extension has been improved. You can now publish events related to segments and their metadata to Kafka.
+You can set the new properties such as in the following example:
+
+```
Review Comment:
nit: for syntax highlighting:
```suggestion
```properties
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] DO NOT MERGE - 27.0.0 WIP release notes (druid)
Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm commented on code in PR #14600:
URL: https://github.com/apache/druid/pull/14600#discussion_r1268857953
##########
docs/do-not-merge.md:
##########
@@ -0,0 +1,366 @@
+<!--Intentionally, there's no Apache license so that the GHA fails it. This file is not meant to be merged.
+
+- https://github.com/apache/druid/pull/14266 - we removed input source security from 26 (https://github.com/apache/druid/pull/14003). Should we not include this in 27 release notes?
+
+-->
+
+Apache Druid 27.0.0 contains over $NUMBER_FEATURES new features, bug fixes, performance enhancements, documentation improvements, and additional test coverage from $NUMBER_OF_CONTRIBUTORS contributors.
+
+[See the complete set of changes for additional details]($LINK_TO_RELEASE_MILESTONE).
+
+Review the upgrade notes and incompatible changes before you upgrade to Druid 27.0.0.
+
+# Highlights
+
+<!-- HIGHLIGHTS H2. FOR EACH MAJOR FEATURE FOR THE RELEASE -->
+
+## Query from deep storage
+
+TBD
+
+### New statements API
+
+Added a new API /druid/v2/sql/statements/ which allows users to fetch results in an asynchronous manner.
+
+[14416](https://github.com/apache/druid/pull/14416)
+
+### New and updated fields in API to get results
+
+The API response to get results now returns a `pages` field containing information on each page of the results status.
+The `numRows` and `sizeInBytes` fields are renamed to `numTotalRows` and `totalSizeInBytes`, respectively.
+
+[14512](https://github.com/apache/druid/pull/14512)
+
+
+### Durable storage for results
+
+MSQ can now write SELECT query results to durable storage. To do so, set the context flag `selectDestination:DURABLE_STORAGE` while issuing SELECT queries to MSQ.
+
+[14527](https://github.com/apache/druid/pull/14527)
+
+## Java 17 support
+
+Druid now fully supports Java 17.
+[14384](https://github.com/apache/druid/pull/14384)
+
+## Array column types
+
+GA + TBD
+
+## Schema auto-discovery
+
+GA + TBD
+
+## Smart segment loading
+
+[13197](https://github.com/apache/druid/pull/13197)
+
+## Hadoop 2 support dropped
Review Comment:
It is not being dropped in this release, but we did decide to deprecate it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] DO NOT MERGE - 27.0.0 WIP release notes (druid)
Posted by "317brian (via GitHub)" <gi...@apache.org>.
317brian commented on code in PR #14600:
URL: https://github.com/apache/druid/pull/14600#discussion_r1281297145
##########
docs/do-not-merge.md:
##########
@@ -4,6 +4,23 @@
-->
+# Still need
+
+search for TBD
+
+* Caveats for array column types
+* More info about Hadoop 2 deprecation
+* Info for parameter execution highlight
+* Caching cost for upgrade notes - is it this? [#14484](https://github.com/apache/druid/pull/14484)
+* User impact of https://github.com/apache/druid/pull/14048 for highlights
+* https://github.com/apache/druid/pull/14476 seems to only be documented here in release notes and the PR?
+* Vad's input for web console section
+* PR for Temporary storage as a runtime parameter. MSQ will start honoring changes to the Middle Manager.
+* What to do about input source security. There are no docs for it.
+* New query filters
+* Confirm the correct naming in https://github.com/apache/druid/pull/14359 (OSHI sys mon change). The changed files say one thing and the PR description says another
+* Community extensions. What do we want to say about iceberg and the other one (what was the other one?)
Review Comment:
Also SQL compatability mdoe mention somewhere?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] DO NOT MERGE - 27.0.0 WIP release notes (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #14600:
URL: https://github.com/apache/druid/pull/14600#discussion_r1277103309
##########
docs/do-not-merge.md:
##########
@@ -264,6 +263,12 @@ Added `groupId` to task metrics emitted by the Overlord. This is helpful for gro
[14402](https://github.com/apache/druid/pull/14402)
+### New metrics for monitoring sync status of `HttpServerInventoryView`
+
+|Metric|Description|Dimensions|Normal value|
+|------|-----------|-----------|----------|
+|`segment/serverview/sync/healthy`|Sync status of the Coordinator/Broker with a segment-loading server such as a Historical or Peon. Emitted by the Coordinator and Brokers only when [HTTP-based server view](../configuration/index.md#segment-management) is enabled. This metric can be used in conjunction with `segment/serverview/sync/unstableTime` to debug slow startup of the Coordinator.|`server`, `tier`|1 for fully synced servers, 0 otherwise|
+|`segment/serverview/sync/unstableTime`|Time in milliseconds for which the Coordinator/Broker has been failing to sync with a segment-loading server. Emitted by the Coordinator and Brokers only when [HTTP-based server view](../configuration/index.md#segment-management) is enabled.|`server`, `tier`|Not emitted for synced servers.|```
Review Comment:
```suggestion
|`segment/serverview/sync/unstableTime`|Time in milliseconds for which the Coordinator/Broker has been failing to sync with a segment-loading server. Emitted by the Coordinator and Brokers only when [HTTP-based server view](../configuration/index.md#segment-management) is enabled.|`server`, `tier`|Not emitted for synced servers.|
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] DO NOT MERGE - 27.0.0 WIP release notes (druid)
Posted by "317brian (via GitHub)" <gi...@apache.org>.
317brian closed pull request #14600: DO NOT MERGE - 27.0.0 WIP release notes
URL: https://github.com/apache/druid/pull/14600
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] DO NOT MERGE - 27.0.0 WIP release notes (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #14600:
URL: https://github.com/apache/druid/pull/14600#discussion_r1280123562
##########
docs/do-not-merge.md:
##########
@@ -30,29 +32,37 @@ As part of this feature, there are new API endpoints available to interact with
- Get query results with `GET /sql/statements/:queryID`
-[14416](https://github.com/apache/druid/pull/14416)
-[14512](https://github.com/apache/druid/pull/14512)
-[14527](https://github.com/apache/druid/pull/14527)
+[#14416](https://github.com/apache/druid/pull/14416)
-## Java 17 support
+[#14512](https://github.com/apache/druid/pull/14512)
-Druid now fully supports Java 17.
-[14384](https://github.com/apache/druid/pull/14384)
+[#14527](https://github.com/apache/druid/pull/14527)
-## Array column types
+ADD LINK TO DOCS
-Array column types are now generally available.
-
-For more information about this feature, see the [26.0.0 release notes for array column types](https://github.com/apache/druid/releases#26.0.0-highlights-auto-type-column-schema-%28experimental%29).
+WILL TO JAZZ IT UP
## Schema auto-discovery
Schema auto-discovery is now generally available.
For more information about this feature, see the [26.0.0 release notes for Schema auto-discovery](https://github.com/apache/druid/releases#26.0.0-highlights-auto-type-column-schema-%28experimental%29-schema-auto-discovery-%28experimental%29).
+## Array column types - COMBINE WITH SCHEMA AUTO-DISCOVERY
+
+As part of Schema auto-discovery, array column types are now generally available.
+
+- THERE ARE CAVEATS MUST USE AUTO TYPE COLUMN INDEXER FROM SCHEMA AUTO-DISCOVERY
+- GET CAVEATS FROM CLINT
+
+
+For more information about this feature, see the [26.0.0 release notes for array column types](https://github.com/apache/druid/releases#26.0.0-highlights-auto-type-column-schema-%28experimental%29).
+
+
## Smart segment loading
+INSTEAD OF PRIORITIZING THINGS EQUALLY; WE PRIORITIZE NEW SEGMENTS THAT ARE UNDERREPLICATED
Review Comment:
**Suggested content for this section:**
The Coordinator is now much more stable and user-friendly. In the new `smartSegmentLoading` mode, it dynamically computes values for several configs which maximize performance.
The Coordinator can now prioritize load of more recent segments and segments that are completely unavailable over load of segments that already have some replicas loaded in the cluster. It can also re-evaluate decisions taken in previous runs and cancel operations that are not needed anymore. Moreoever, move operations started by segment balancing do not compete with the load of unavailable segments thus reducing the reaction time for changes in the cluster and speeding up segment assignment decisions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] DO NOT MERGE - 27.0.0 WIP release notes (druid)
Posted by "317brian (via GitHub)" <gi...@apache.org>.
317brian commented on PR #14600:
URL: https://github.com/apache/druid/pull/14600#issuecomment-1670388252
I've cleaned up all my notes and included more information. @abhishekagarwal87, @AmatyaAvadhanula , and @2bethere can y'all double check the deprecation list to make sure I have everything.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] DO NOT MERGE - 27.0.0 WIP release notes (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #14600:
URL: https://github.com/apache/druid/pull/14600#discussion_r1275735442
##########
docs/do-not-merge.md:
##########
@@ -0,0 +1,377 @@
+<!--Intentionally, there's no Apache license so that the GHA fails it. This file is not meant to be merged.
+
+- https://github.com/apache/druid/pull/14266 - we removed input source security from 26 (https://github.com/apache/druid/pull/14003). Should we not include this in 27 release notes?
+
+-->
+
+Apache Druid 27.0.0 contains over $NUMBER_FEATURES new features, bug fixes, performance enhancements, documentation improvements, and additional test coverage from $NUMBER_OF_CONTRIBUTORS contributors.
+
+[See the complete set of changes for additional details]($LINK_TO_RELEASE_MILESTONE).
+
+Review the upgrade notes and incompatible changes before you upgrade to Druid 27.0.0.
+
+# Highlights
+
+<!-- HIGHLIGHTS H2. FOR EACH MAJOR FEATURE FOR THE RELEASE -->
+
+## Query from deep storage (experimental)
+
+Druid now supports querying segments that are stored only in deep storage. When you query from deep storage, you increase the data available for queries without necessarily having to scale your Historical processes to accommodate more data. To take advantage of the potential savings, make sure you configure your load rules to not load all your data onto Historical processes. Then, configure a corresponding drop rule to drop those segments.
+
+Note that at least one segment of a datasource must be loaded onto a Historical process so that Druid can plan the query. It can be any segment though.
+
+As part of this feature, there are new API endpoints available to interact with the query:
+
+- Submit queries using `POST /sql/statements`. The payload is the same as any other Druid SQL query with additional context parameter available.
+ - The required `executionMode: ASYNC` configures Druid to return the results asynchronously. Results are only available after the query completes.
+ - The optional context parameter `selectDestination:DURABLE_STORAGE` configures Druid to write the query results to durable storage.
+
+- Get the status of a query with `GET /sq/statements/:queryID`.
+
+- Get query results with `/sql/statements/:queryID`
+
+[14416](https://github.com/apache/druid/pull/14416)
+[14512](https://github.com/apache/druid/pull/14512)
+[14527](https://github.com/apache/druid/pull/14527)
+
+## Java 17 support
+
+Druid now fully supports Java 17.
+[14384](https://github.com/apache/druid/pull/14384)
+
+## Array column types
+
+Array column types are now generally available.
+
+For more information about this feature, see the [26.0.0 release notes for array column types](https://github.com/apache/druid/releases#26.0.0-highlights-auto-type-column-schema-%28experimental%29).
+
+## Schema auto-discovery
+
+Schema auto-discovery is now generally available.
+
+For more information about this feature, see the [26.0.0 release notes for Schema auto-discovery](https://github.com/apache/druid/releases#26.0.0-highlights-auto-type-column-schema-%28experimental%29-schema-auto-discovery-%28experimental%29).
+
+## Smart segment loading
+
+Revamped Coordinator to make it more stable and user-friendly. This is accompanied by several bug fixes, logging and metric improvements, and a whole new range of capabilities.
+
+Coordinator now supports a `smartSegmentLoading` mode, which is enabled by default. When enabled, users don't need to specify any of the following dynamic configs as they would be ignored by the coordinator. Instead, the coordinator computes the optimal values of the following configs at run time to best utilize coordinator runs:
+
+* `maxSegmentsInNodeLoadingQueue`
+* `maxSegmentsToMove`
+* `replicationThrottleLimit`
+* `useRoundRobinSegmentAssignment`
+* `useBatchedSegmentSampler`
+* `emitBalancingStats`
+
+These configs are now deprecated and will be removed in subsequent releases.
+
+Coordinator is now capable of prioritization and cancellation of items in segment load queues. This means that the coordinator now reacts faster to changes in the cluster and makes better segment assignment decisions.
+
+As part of this change, new metrics are available. For more information, see [New segment metrics](#new-segment-metrics).
+
+[13197](https://github.com/apache/druid/pull/13197)
+
+## Hadoop 2 deprecated
+
+Support for Hadoop 2 is now deprecated. It will be removed in a future release.
+
+# Additional features and improvements
+
+## MSQ task engine
+
+### `maxInputBytesPerWorker` context parameter
+
+The context parameter now denotes the estimated weighted size (in bytes) of the input to split on. The MSQ task engine now takes into account the input format and compression format instead of the actual file size reported by the file system.
+
+The default value for the context parameter has also been changed. It is now 512 MiB (previously 10 GiB).
+
+[14307](https://github.com/apache/druid/pull/14307)
+
+### Improved query planning behavior
+
+Druid now fails query planning if a CLUSTERED BY column contains descending order.
+Previously, queries would successfully plan if any CLUSTERED BY columns contained descending order.
+
+The MSQ fault, `InsertCannotOrderByDescending`, is deprecated. An INSERT or REPLACE query containing a CLUSTERED BY expression cannot be in descending order. Druid's segment generation code only supports ascending order. Instead of the fault, Druid now throws a query `ValidationException`.
+
+[14436](https://github.com/apache/druid/pull/14436)
+
+### SELECT query results
+
+SELECT queries executed using MSQ now generate only a subset of the results in the query reports.
+To fetch the complete result set, run the query using the native engine.
+
+[14370](https://github.com/apache/druid/pull/14370)
+
+
+### Other MSQ improvements
+
+- The same aggregator can now have two output names [14367](https://github.com/apache/druid/pull/14367)
+- Changed the default `clusterStatisticsMergeMode` to SEQUENTIAL (#14310)[https://github.com/apache/druid/pull/14310]
+- Added a query context parameter `MultiStageQueryContext` to determine whether the result of an MSQ select query is limited (#14476)[https://github.com/apache/druid/pull/14476]
+- Enabled using functions as inputs for `index` and `length` parameters (#14480)[https://github.com/apache/druid/pull/14480]
+
+## Ingestion
+
+### New property for task completion updates
+
+The new property `druid.indexer.queue.taskCompleteHandlerNumThreads` controls the number of threads used by the Overlord `TaskQueue` to handle task completion updates received from the workers.
+
+The following metrics have been added:
+* `task/status/queue/count`: Monitors the number of queued items
+* `task/status/updated/count`: Monitors the number of processed items
+
+[14533](https://github.com/apache/druid/pull/14533)
+
+### Improved response to max_allowed_packet limit
+
+If the Overlord fails to insert a task into the metadata because of a payload that exceeds the `max_allowed_packet` limit, the response now returns `400 Bad request` instead of `500 Internal server error`. This prevents an `index_parallel` task from retrying the insertion of a bad sub-task indefinitely and causes it to fail immediately.
+
+[14271](https://github.com/apache/druid/pull/14271)
+
+### Improved handling of mixed type arrays
+
+Druid now handles mixed type arrays such as `[["a", "b", "c"], {"x": 123}]` as `ARRAY<COMPLEX<json>` rather than throwing an incompatible type exception.
+
+[14438](https://github.com/apache/druid/pull/14438)
+
+### Other ingestion improvements
+
+* A negative streaming ingestion lag is no longer emitted as a result of stale offsets. [14292](https://github.com/apache/druid/pull/14292)
+* Removed double synchronization on simple map operations in Kubernetes task runner. [14435](https://github.com/apache/druid/pull/14435)
+* Kubernetes overlord extension now cleans up the job if the task pod fails to come up in time. [14425](https://github.com/apache/druid/pull/14425)
+
+## Querying
+
+### New function for regular expression replacement
+
+The new function `REGEXP_REPLACE` allows you to replace all instances of a pattern with a replacement string.
+
+[14460](https://github.com/apache/druid/pull/14460)
+
+### Query results directory
+
+Druid now supports a `query-results` directory in durable storage to store query results after the task finishes. The auto cleaner does not remove this directory unless the task ID is not known to the Overlord.
+
+[14446](https://github.com/apache/druid/pull/14446)
+
+### Limit for subquery results by memory usage
+
+Users can now add a guardrail to prevent subquery’s results from exceeding the set number of bytes by setting `druid.server.http.maxSubqueryRows` in the Broker's config or `maxSubqueryRows` in the query context. This feature is experimental for now and would default back to row-based limiting in case it fails to get the accurate size of the results consumed by the query.
+
+[13952](https://github.com/apache/druid/pull/13952)
+
+### HLL and Theta sketch estimates
+
+You can now use `HLL_SKETCH_ESTIMATE` and `THETA_SKETCH_ESTIMATE` as expressions. These estimates work on sketch columns and have the same behavior as `postAggs`.
+
+[14312](https://github.com/apache/druid/pull/14312)
+
+### String encoding parameter for HLL sketches
+
+HLL sketches now accept a new `stringEncoding` parameter to build sketches with UTF-8 bytes.
+
+[11201](https://github.com/apache/druid/pull/11201)
+
+### EARLIEST_BY and LATEST_BY signatures
+
+Updated EARLIEST_BY and LATEST_BY function signatures as follows:
+
+* Changed `EARLIEST(expr, timeColumn)` to `EARLIEST_BY(expr, timeColumn)`
+* Changed `LATEST(expr, timeColumn)` to `LATEST_BY(expr, timeColumn)`
+
+[14352](https://github.com/apache/druid/pull/14352)
+
+
+### Exposed Druid functions
+
+Exposed information about Druid functions in the `INFORMATION_SCHEMA.ROUTINES` table. The table contains the following columns:
+
+* ROUTINE_CATALOG: Name of the database. Always set to druid.
+* ROUTINE_SCHEMA: Name of the schema. Always set to INFORMATION_SCHEMA.
+* ROUTINE_NAME: Name of the function or operator. For example: APPROX_COUNT_DISTINCT_DS_THETA.
+* ROUTINE_TYPE: Type of routine. Always set to FUNCTION.
+* IS_AGGREGATOR: Determines if the routine is an aggregator. Returns YES for aggregator functions; NO for scalar functions.
+* SIGNATURES: Possible signatures for the routine as a string.
+
+[14378](https://github.com/apache/druid/pull/14378)
+
+### New Broker configuration for SQL schema migrations
+
+A new broker configuration, `druid.sql.planner.metadataColumnTypeMergePolicy`, adds configurable modes to how column types are computed for the SQL table schema when faced with differences between segments.
+
+The new `leastRestrictive` mode chooses the most appropriate type that data across all segments can best be coerced into and is the default behavior. This is a subtle behavior change around when segment driven schema migrations will take effect for the SQL schema:
+
+- With `leastRestrictive` mode, the schema only updates once all segments are reindexed to the new type.
+- With `latestInterval` mode, the SQL schema gets updated as soon as the first job with the new schema publishes segments in the latest time interval of the data.
+
+`leastRestrictive` can have better query time behavior and eliminates some query time errors that can occur when using `latestInterval`.
+
+[14319](https://github.com/apache/druid/pull/14319)
+
+### EXPLAIN PLAN improvements
+
+The EXPLAIN PLAN result includes a new column `ATTRIBUTES` that describes the attributes of a query.
+See [SQL query translation](docs/querying/sql-translation.md) for more information and examples.
+
+[14391](https://github.com/apache/druid/pull/14391)
+
+
+## Metrics and monitoring
+
+### Added a new SysMonitorOshi monitor
+
+Added a new monitor `SysMonitorOshi` to replace `SysMonitor`. The new monitor has a wider support for different machine architectures including ARM instances. We recommend switching to `SysMonitorOshi` as `SysMonitor` is now deprecated and will be removed in future releases.
+
+[14359](https://github.com/apache/druid/pull/14359)
+
+### Monitor for Overlord and Coordinator service health
+
+Added a new monitor `ServiceStatusMonitor` to monitor the service health of the Overlord and Coordinator.
+
+[14443](https://github.com/apache/druid/pull/14443)
+
+### New segment metrics
+
+The following metrics are now available for Brokers:
+
+|Metric|Description|Dimensions |Normal value|
+|------|-----------|------------|
Review Comment:
```suggestion
|Metric|Description|Dimensions|Normal value|
|------|-----------|----------|----------|
```
##########
docs/do-not-merge.md:
##########
@@ -0,0 +1,377 @@
+<!--Intentionally, there's no Apache license so that the GHA fails it. This file is not meant to be merged.
+
+- https://github.com/apache/druid/pull/14266 - we removed input source security from 26 (https://github.com/apache/druid/pull/14003). Should we not include this in 27 release notes?
+
+-->
+
+Apache Druid 27.0.0 contains over $NUMBER_FEATURES new features, bug fixes, performance enhancements, documentation improvements, and additional test coverage from $NUMBER_OF_CONTRIBUTORS contributors.
+
+[See the complete set of changes for additional details]($LINK_TO_RELEASE_MILESTONE).
+
+Review the upgrade notes and incompatible changes before you upgrade to Druid 27.0.0.
+
+# Highlights
+
+<!-- HIGHLIGHTS H2. FOR EACH MAJOR FEATURE FOR THE RELEASE -->
+
+## Query from deep storage (experimental)
+
+Druid now supports querying segments that are stored only in deep storage. When you query from deep storage, you increase the data available for queries without necessarily having to scale your Historical processes to accommodate more data. To take advantage of the potential savings, make sure you configure your load rules to not load all your data onto Historical processes. Then, configure a corresponding drop rule to drop those segments.
+
+Note that at least one segment of a datasource must be loaded onto a Historical process so that Druid can plan the query. It can be any segment though.
+
+As part of this feature, there are new API endpoints available to interact with the query:
+
+- Submit queries using `POST /sql/statements`. The payload is the same as any other Druid SQL query with additional context parameter available.
+ - The required `executionMode: ASYNC` configures Druid to return the results asynchronously. Results are only available after the query completes.
+ - The optional context parameter `selectDestination:DURABLE_STORAGE` configures Druid to write the query results to durable storage.
+
+- Get the status of a query with `GET /sq/statements/:queryID`.
+
+- Get query results with `/sql/statements/:queryID`
+
+[14416](https://github.com/apache/druid/pull/14416)
+[14512](https://github.com/apache/druid/pull/14512)
+[14527](https://github.com/apache/druid/pull/14527)
+
+## Java 17 support
+
+Druid now fully supports Java 17.
+[14384](https://github.com/apache/druid/pull/14384)
+
+## Array column types
+
+Array column types are now generally available.
+
+For more information about this feature, see the [26.0.0 release notes for array column types](https://github.com/apache/druid/releases#26.0.0-highlights-auto-type-column-schema-%28experimental%29).
+
+## Schema auto-discovery
+
+Schema auto-discovery is now generally available.
+
+For more information about this feature, see the [26.0.0 release notes for Schema auto-discovery](https://github.com/apache/druid/releases#26.0.0-highlights-auto-type-column-schema-%28experimental%29-schema-auto-discovery-%28experimental%29).
+
+## Smart segment loading
+
+Revamped Coordinator to make it more stable and user-friendly. This is accompanied by several bug fixes, logging and metric improvements, and a whole new range of capabilities.
+
+Coordinator now supports a `smartSegmentLoading` mode, which is enabled by default. When enabled, users don't need to specify any of the following dynamic configs as they would be ignored by the coordinator. Instead, the coordinator computes the optimal values of the following configs at run time to best utilize coordinator runs:
+
+* `maxSegmentsInNodeLoadingQueue`
+* `maxSegmentsToMove`
+* `replicationThrottleLimit`
+* `useRoundRobinSegmentAssignment`
+* `useBatchedSegmentSampler`
+* `emitBalancingStats`
+
+These configs are now deprecated and will be removed in subsequent releases.
+
+Coordinator is now capable of prioritization and cancellation of items in segment load queues. This means that the coordinator now reacts faster to changes in the cluster and makes better segment assignment decisions.
+
+As part of this change, new metrics are available. For more information, see [New segment metrics](#new-segment-metrics).
+
+[13197](https://github.com/apache/druid/pull/13197)
+
+## Hadoop 2 deprecated
+
+Support for Hadoop 2 is now deprecated. It will be removed in a future release.
+
+# Additional features and improvements
+
+## MSQ task engine
+
+### `maxInputBytesPerWorker` context parameter
+
+The context parameter now denotes the estimated weighted size (in bytes) of the input to split on. The MSQ task engine now takes into account the input format and compression format instead of the actual file size reported by the file system.
+
+The default value for the context parameter has also been changed. It is now 512 MiB (previously 10 GiB).
+
+[14307](https://github.com/apache/druid/pull/14307)
+
+### Improved query planning behavior
+
+Druid now fails query planning if a CLUSTERED BY column contains descending order.
+Previously, queries would successfully plan if any CLUSTERED BY columns contained descending order.
+
+The MSQ fault, `InsertCannotOrderByDescending`, is deprecated. An INSERT or REPLACE query containing a CLUSTERED BY expression cannot be in descending order. Druid's segment generation code only supports ascending order. Instead of the fault, Druid now throws a query `ValidationException`.
+
+[14436](https://github.com/apache/druid/pull/14436)
+
+### SELECT query results
+
+SELECT queries executed using MSQ now generate only a subset of the results in the query reports.
+To fetch the complete result set, run the query using the native engine.
+
+[14370](https://github.com/apache/druid/pull/14370)
+
+
+### Other MSQ improvements
+
+- The same aggregator can now have two output names [14367](https://github.com/apache/druid/pull/14367)
+- Changed the default `clusterStatisticsMergeMode` to SEQUENTIAL (#14310)[https://github.com/apache/druid/pull/14310]
+- Added a query context parameter `MultiStageQueryContext` to determine whether the result of an MSQ select query is limited (#14476)[https://github.com/apache/druid/pull/14476]
+- Enabled using functions as inputs for `index` and `length` parameters (#14480)[https://github.com/apache/druid/pull/14480]
+
+## Ingestion
+
+### New property for task completion updates
+
+The new property `druid.indexer.queue.taskCompleteHandlerNumThreads` controls the number of threads used by the Overlord `TaskQueue` to handle task completion updates received from the workers.
+
+The following metrics have been added:
+* `task/status/queue/count`: Monitors the number of queued items
+* `task/status/updated/count`: Monitors the number of processed items
+
+[14533](https://github.com/apache/druid/pull/14533)
+
+### Improved response to max_allowed_packet limit
+
+If the Overlord fails to insert a task into the metadata because of a payload that exceeds the `max_allowed_packet` limit, the response now returns `400 Bad request` instead of `500 Internal server error`. This prevents an `index_parallel` task from retrying the insertion of a bad sub-task indefinitely and causes it to fail immediately.
+
+[14271](https://github.com/apache/druid/pull/14271)
+
+### Improved handling of mixed type arrays
+
+Druid now handles mixed type arrays such as `[["a", "b", "c"], {"x": 123}]` as `ARRAY<COMPLEX<json>` rather than throwing an incompatible type exception.
+
+[14438](https://github.com/apache/druid/pull/14438)
+
+### Other ingestion improvements
+
+* A negative streaming ingestion lag is no longer emitted as a result of stale offsets. [14292](https://github.com/apache/druid/pull/14292)
+* Removed double synchronization on simple map operations in Kubernetes task runner. [14435](https://github.com/apache/druid/pull/14435)
+* Kubernetes overlord extension now cleans up the job if the task pod fails to come up in time. [14425](https://github.com/apache/druid/pull/14425)
+
+## Querying
+
+### New function for regular expression replacement
+
+The new function `REGEXP_REPLACE` allows you to replace all instances of a pattern with a replacement string.
+
+[14460](https://github.com/apache/druid/pull/14460)
+
+### Query results directory
+
+Druid now supports a `query-results` directory in durable storage to store query results after the task finishes. The auto cleaner does not remove this directory unless the task ID is not known to the Overlord.
+
+[14446](https://github.com/apache/druid/pull/14446)
+
+### Limit for subquery results by memory usage
+
+Users can now add a guardrail to prevent subquery’s results from exceeding the set number of bytes by setting `druid.server.http.maxSubqueryRows` in the Broker's config or `maxSubqueryRows` in the query context. This feature is experimental for now and would default back to row-based limiting in case it fails to get the accurate size of the results consumed by the query.
+
+[13952](https://github.com/apache/druid/pull/13952)
+
+### HLL and Theta sketch estimates
+
+You can now use `HLL_SKETCH_ESTIMATE` and `THETA_SKETCH_ESTIMATE` as expressions. These estimates work on sketch columns and have the same behavior as `postAggs`.
+
+[14312](https://github.com/apache/druid/pull/14312)
+
+### String encoding parameter for HLL sketches
+
+HLL sketches now accept a new `stringEncoding` parameter to build sketches with UTF-8 bytes.
+
+[11201](https://github.com/apache/druid/pull/11201)
+
+### EARLIEST_BY and LATEST_BY signatures
+
+Updated EARLIEST_BY and LATEST_BY function signatures as follows:
+
+* Changed `EARLIEST(expr, timeColumn)` to `EARLIEST_BY(expr, timeColumn)`
+* Changed `LATEST(expr, timeColumn)` to `LATEST_BY(expr, timeColumn)`
+
+[14352](https://github.com/apache/druid/pull/14352)
+
+
+### Exposed Druid functions
+
+Exposed information about Druid functions in the `INFORMATION_SCHEMA.ROUTINES` table. The table contains the following columns:
+
+* ROUTINE_CATALOG: Name of the database. Always set to druid.
+* ROUTINE_SCHEMA: Name of the schema. Always set to INFORMATION_SCHEMA.
+* ROUTINE_NAME: Name of the function or operator. For example: APPROX_COUNT_DISTINCT_DS_THETA.
+* ROUTINE_TYPE: Type of routine. Always set to FUNCTION.
+* IS_AGGREGATOR: Determines if the routine is an aggregator. Returns YES for aggregator functions; NO for scalar functions.
+* SIGNATURES: Possible signatures for the routine as a string.
+
+[14378](https://github.com/apache/druid/pull/14378)
+
+### New Broker configuration for SQL schema migrations
+
+A new broker configuration, `druid.sql.planner.metadataColumnTypeMergePolicy`, adds configurable modes to how column types are computed for the SQL table schema when faced with differences between segments.
+
+The new `leastRestrictive` mode chooses the most appropriate type that data across all segments can best be coerced into and is the default behavior. This is a subtle behavior change around when segment driven schema migrations will take effect for the SQL schema:
+
+- With `leastRestrictive` mode, the schema only updates once all segments are reindexed to the new type.
+- With `latestInterval` mode, the SQL schema gets updated as soon as the first job with the new schema publishes segments in the latest time interval of the data.
+
+`leastRestrictive` can have better query time behavior and eliminates some query time errors that can occur when using `latestInterval`.
+
+[14319](https://github.com/apache/druid/pull/14319)
+
+### EXPLAIN PLAN improvements
+
+The EXPLAIN PLAN result includes a new column `ATTRIBUTES` that describes the attributes of a query.
+See [SQL query translation](docs/querying/sql-translation.md) for more information and examples.
+
+[14391](https://github.com/apache/druid/pull/14391)
+
+
+## Metrics and monitoring
+
+### Added a new SysMonitorOshi monitor
+
+Added a new monitor `SysMonitorOshi` to replace `SysMonitor`. The new monitor has a wider support for different machine architectures including ARM instances. We recommend switching to `SysMonitorOshi` as `SysMonitor` is now deprecated and will be removed in future releases.
+
+[14359](https://github.com/apache/druid/pull/14359)
+
+### Monitor for Overlord and Coordinator service health
+
+Added a new monitor `ServiceStatusMonitor` to monitor the service health of the Overlord and Coordinator.
+
+[14443](https://github.com/apache/druid/pull/14443)
+
+### New segment metrics
+
+The following metrics are now available for Brokers:
+
+|Metric|Description|Dimensions |Normal value|
+|------|-----------|------------|
Review Comment:
```suggestion
|Metric|Description|Dimensions|Normal value|
|------|-----------|----------|------------|
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] DO NOT MERGE - 27.0.0 WIP release notes (druid)
Posted by "317brian (via GitHub)" <gi...@apache.org>.
317brian commented on code in PR #14600:
URL: https://github.com/apache/druid/pull/14600#discussion_r1272731278
##########
docs/do-not-merge.md:
##########
@@ -0,0 +1,377 @@
+<!--Intentionally, there's no Apache license so that the GHA fails it. This file is not meant to be merged.
+
+- https://github.com/apache/druid/pull/14266 - we removed input source security from 26 (https://github.com/apache/druid/pull/14003). Should we not include this in 27 release notes?
+
+-->
+
+Apache Druid 27.0.0 contains over $NUMBER_FEATURES new features, bug fixes, performance enhancements, documentation improvements, and additional test coverage from $NUMBER_OF_CONTRIBUTORS contributors.
+
+[See the complete set of changes for additional details]($LINK_TO_RELEASE_MILESTONE).
+
+Review the upgrade notes and incompatible changes before you upgrade to Druid 27.0.0.
+
+# Highlights
+
+<!-- HIGHLIGHTS H2. FOR EACH MAJOR FEATURE FOR THE RELEASE -->
+
+## Query from deep storage
+
+TBD
+
+### New statements API
+
+Added a new API /druid/v2/sql/statements/ which allows users to fetch results in an asynchronous manner.
+
+[14416](https://github.com/apache/druid/pull/14416)
+
+### New and updated fields in API to get results
+
+The API response to get results now returns a `pages` field containing information on each page of the results status.
+The `numRows` and `sizeInBytes` fields are renamed to `numTotalRows` and `totalSizeInBytes`, respectively.
+
+[14512](https://github.com/apache/druid/pull/14512)
+
+
+### Durable storage for results
+
+MSQ can now write SELECT query results to durable storage. To do so, set the context flag `selectDestination:DURABLE_STORAGE` while issuing SELECT queries to MSQ.
+
+[14527](https://github.com/apache/druid/pull/14527)
+
+## Java 17 support
+
+Druid now fully supports Java 17.
+[14384](https://github.com/apache/druid/pull/14384)
+
+## Array column types
+
+GA + TBD
+
+## Schema auto-discovery
+
+GA + TBD
+
+## Smart segment loading
+
+[13197](https://github.com/apache/druid/pull/13197)
+
+## Hadoop 2 deprecated
+
+Hadoop 2 is deprecated. It will be removed in a future release.
+
+# Additional features and improvements
+
+## MSQ task engine
+
+### `maxInputBytesPerWorker` context parameter
+
+The context parameter now denotes the estimated weighted size (in bytes) of the input to split on. The MSQ task engine now takes into account the input format and compression format instead of the actual file size reported by the file system.
+
+The default value for the context parameter has also been changed. It is now 512 MiB (previously 10 GiB).
+
+[14307](https://github.com/apache/druid/pull/14307)
+
+### Improved query planning behavior
+
+Druid now fails query planning if a CLUSTERED BY column contains descending order.
+Previously, queries would successfully plan if any CLUSTERED BY columns contained descending order.
+
+The MSQ fault, `InsertCannotOrderByDescending`, is deprecated. An INSERT or REPLACE query containing a CLUSTERED BY expression cannot be in descending order. Druid's segment generation code only supports ascending order. Instead of the fault, Druid now throws a query `ValidationException`.
+
+[14436](https://github.com/apache/druid/pull/14436)
+
+### SELECT query results
+
+SELECT queries executed using MSQ now generate only a subset of the results in the query reports.
+To fetch the complete result set, run the query using the native engine.
+
+[14370](https://github.com/apache/druid/pull/14370)
+
+
+### Other MSQ improvements
+
+- The same aggregator can now have two output names [14367](https://github.com/apache/druid/pull/14367)
+- Changed the default `clusterStatisticsMergeMode` to SEQUENTIAL (#14310)[https://github.com/apache/druid/pull/14310]
+- Added a query context parameter `MultiStageQueryContext` to determine whether the result of an MSQ select query is limited (#14476)[https://github.com/apache/druid/pull/14476]
+- Enabled using functions as inputs for `index` and `length` parameters (#14480)[https://github.com/apache/druid/pull/14480]
+
+## Ingestion
+
+### New property for task completion updates
+
+The new property `druid.indexer.queue.taskCompleteHandlerNumThreads` controls the number of threads used by the Overlord `TaskQueue` to handle task completion updates received from the workers.
+
+The following metrics have been added:
+* `task/status/queue/count`: Monitors the number of queued items
+* `task/status/updated/count`: Monitors the number of processed items
+
+[14533](https://github.com/apache/druid/pull/14533)
+
+### Improved response to max_allowed_packet limit
+
+If the Overlord fails to insert a task into the metadata because of a payload that exceeds the `max_allowed_packet` limit, the response now returns `400 Bad request` instead of `500 Internal server error`. This prevents an `index_parallel` task from retrying the insertion of a bad sub-task indefinitely and causes it to fail immediately.
+
+[14271](https://github.com/apache/druid/pull/14271)
+
+### Improved handling of mixed type arrays
+
+Druid now handles mixed type arrays such as `[["a", "b", "c"], {"x": 123}]` as `ARRAY<COMPLEX<json>` rather than throwing an incompatible type exception.
+
+[14438](https://github.com/apache/druid/pull/14438)
+
+### Other ingestion improvements
+
+* A negative streaming ingestion lag is no longer emitted as a result of stale offsets. [14292](https://github.com/apache/druid/pull/14292)
+* Removed double synchronization on simple map operations in Kubernetes task runner. [14435](https://github.com/apache/druid/pull/14435)
+* Kubernetes overlord extension now cleans up the job if the task pod fails to come up in time. [14425](https://github.com/apache/druid/pull/14425)
+
+## Querying
+
+### New function for regular expression replacement
+
+The new function `REGEXP_REPLACE` allows you to replace all instances of a pattern with a replacement string.
+
+[14460](https://github.com/apache/druid/pull/14460)
+
+### Query results directory
+
+Druid now supports a `query-results` directory in durable storage to store query results after the task finishes. The auto cleaner does not remove this directory unless the task ID is not known to the Overlord.
+
+[14446](https://github.com/apache/druid/pull/14446)
+
+### Limit for subquery results by memory usage
+
+Users can now add a guardrail to prevent subquery’s results from exceeding the set number of bytes by setting `druid.server.http.maxSubqueryRows` in the Broker's config or `maxSubqueryRows` in the query context. This feature is experimental for now and would default back to row-based limiting in case it fails to get the accurate size of the results consumed by the query.
+
+[13952](https://github.com/apache/druid/pull/13952)
+
+### HLL and Theta sketch estimates
+
+You can now use `HLL_SKETCH_ESTIMATE` and `THETA_SKETCH_ESTIMATE` as expressions. These estimates work on sketch columns and have the same behavior as `postAggs`.
+
+[14312](https://github.com/apache/druid/pull/14312)
+
+### String encoding parameter for HLL sketches
+
+HLL sketches now accept a new `stringEncoding` parameter to build sketches with UTF-8 bytes.
+
+[11201](https://github.com/apache/druid/pull/11201)
+
+### EARLIEST_BY and LATEST_BY signatures
+
+Updated EARLIEST_BY and LATEST_BY function signatures as follows:
+
+* Changed `EARLIEST(expr, timeColumn)` to `EARLIEST_BY(expr, timeColumn)`
+* Changed `LATEST(expr, timeColumn)` to `LATEST_BY(expr, timeColumn)`
+
+[14352](https://github.com/apache/druid/pull/14352)
+
+
+### Exposed Druid functions
+
+Exposed information about Druid functions in the `INFORMATION_SCHEMA.ROUTINES` table. The table contains the following columns:
+
+* ROUTINE_CATALOG: Name of the database. Always set to druid.
+* ROUTINE_SCHEMA: Name of the schema. Always set to INFORMATION_SCHEMA.
+* ROUTINE_NAME: Name of the function or operator. For example: APPROX_COUNT_DISTINCT_DS_THETA.
+* ROUTINE_TYPE: Type of routine. Always set to FUNCTION.
+* IS_AGGREGATOR: Determines if the routine is an aggregator. Returns YES for aggregator functions; NO for scalar functions.
+* SIGNATURES: Possible signatures for the routine as a string.
+
+[14378](https://github.com/apache/druid/pull/14378)
+
+### New Broker configuration for SQL schema migrations
+
+A new broker configuration, `druid.sql.planner.metadataColumnTypeMergePolicy`, adds configurable modes to how column types are computed for the SQL table schema when faced with differences between segments.
+
+The new `leastRestrictive` mode chooses the most appropriate type that data across all segments can best be coerced into and is the default behavior. This is a subtle behavior change around when segment driven schema migrations will take effect for the SQL schema:
+
+- With `leastRestrictive` mode, the schema only updates once all segments are reindexed to the new type.
+- With `latestInterval` mode, the SQL schema gets updated as soon as the first job with the new schema publishes segments in the latest time interval of the data.
+
+`leastRestrictive` can have better query time behavior and eliminates some query time errors that can occur when using `latestInterval`.
+
+[14319](https://github.com/apache/druid/pull/14319)
+
+### EXPLAIN PLAN improvements
+
+The EXPLAIN PLAN result includes a new column `ATTRIBUTES` that describes the attributes of a query.
+See [SQL query translation](docs/querying/sql-translation.md) for more information and examples.
+
+[14391](https://github.com/apache/druid/pull/14391)
+
+
+## Metrics and monitoring
+
+### Added a new SysMonitorOshi monitor
+
+Added a new monitor `SysMonitorOshi` to replace `SysMonitor`. The new monitor has a wider support for different machine architectures including ARM instances. We recommend switching to `SysMonitorOshi` as `SysMonitor` is now deprecated and will be removed in future releases.
+
+[14359](https://github.com/apache/druid/pull/14359)
+
+### Monitor for Overlord and Coordinator service health
+
+Added a new monitor `ServiceStatusMonitor` to monitor the service health of the Overlord and Coordinator.
+
+[14443](https://github.com/apache/druid/pull/14443)
+
+### New segment metrics
+
+The following metrics are now available for Brokers:
+
+|Metric|Description|Dimensions |Normal value|
+|------|-----------|------------|
+|`segment/metadatacache/refresh/count`|Number of segments to refresh in broker segment metadata cache. Emitted once per refresh per datasource.|`dataSource`| |
+|`segment/metadatacache/refresh/time`|Time taken to refresh segments in broker segment metadata cache. Emitted once per refresh per datasource.|`dataSource`| |
+| `segment/loadQueue/assigned` | Number of segments assigned for load or drop to the load queue of a server. |`dataSource`,`server` | Varies |
+| `segment/loadQueue/success` |Number of segment assignments that completed successfully.|`dataSource`, `server`|Varies|
+| `segment/loadQueue/cancelled` |Number of segment assignments that were canceled before completion. |`dataSource`,`server` | 0 |
+| `segment/loadQueue/failed` |Number of segment assignments that failed to complete.|`dataSource`, `server`|0|
+| `segment/assigned/broadcast` | |
Review Comment:
@kfaraz It looks like the 2 broadcast metrics from 13197 didn't get added to https://github.com/apache/druid/blob/master/docs/operations/metrics.md
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] DO NOT MERGE - 27.0.0 WIP release notes (druid)
Posted by "317brian (via GitHub)" <gi...@apache.org>.
317brian commented on code in PR #14600:
URL: https://github.com/apache/druid/pull/14600#discussion_r1287727569
##########
docs/do-not-merge.md:
##########
@@ -182,163 +194,140 @@ Updated EARLIEST_BY and LATEST_BY function signatures as follows:
* Changed `EARLIEST(expr, timeColumn)` to `EARLIEST_BY(expr, timeColumn)`
* Changed `LATEST(expr, timeColumn)` to `LATEST_BY(expr, timeColumn)`
-[14352](https://github.com/apache/druid/pull/14352)
+[#14352](https://github.com/apache/druid/pull/14352)
+### INFORMATION_SCHEMA.ROUTINES TABLE
-### Exposed Druid functions
+Use the new `INFORMATION_SCHEMA.ROUTINES` to programmatically get information about the functions that Druid SQL supports.
-Exposed information about Druid functions in the `INFORMATION_SCHEMA.ROUTINES` table. The table contains the following columns:
+For more information, such as the available columns, see [ROUTINES table](https://druid.apache.org/docs/latest/querying/sql-metadata-tables.html#routines-table)
-* ROUTINE_CATALOG: Name of the database. Always set to druid.
-* ROUTINE_SCHEMA: Name of the schema. Always set to INFORMATION_SCHEMA.
-* ROUTINE_NAME: Name of the function or operator. For example: APPROX_COUNT_DISTINCT_DS_THETA.
-* ROUTINE_TYPE: Type of routine. Always set to FUNCTION.
-* IS_AGGREGATOR: Determines if the routine is an aggregator. Returns YES for aggregator functions; NO for scalar functions.
-* SIGNATURES: Possible signatures for the routine as a string.
-
-[14378](https://github.com/apache/druid/pull/14378)
+[#14378](https://github.com/apache/druid/pull/14378)
### New Broker configuration for SQL schema migrations
-A new broker configuration, `druid.sql.planner.metadataColumnTypeMergePolicy`, adds configurable modes to how column types are computed for the SQL table schema when faced with differences between segments.
+You can now better control how Druid reacts to schema changes between segments. This can make querying more resilient when newer segments introduce different types, such as if a column previously contained LONG values and newer segments contain STRING.
-The new `leastRestrictive` mode chooses the most appropriate type that data across all segments can best be coerced into and is the default behavior. This is a subtle behavior change around when segment driven schema migrations will take effect for the SQL schema:
+Use the new Broker configuration, `druid.sql.planner.metadataColumnTypeMergePolicy` to control how column types are computed for the SQL table schema when faced with differences between segments.
-- With `leastRestrictive` mode, the schema only updates once all segments are reindexed to the new type.
-- With `latestInterval` mode, the SQL schema gets updated as soon as the first job with the new schema publishes segments in the latest time interval of the data.
+Set it to one of the following:
+- `leastRestrictive`: the schema only updates once all segments are reindexed to the new type.
+- `latestInterval`: the SQL schema gets updated as soon as the first job with the new schema publishes segments in the latest time interval of the data.
+
`leastRestrictive` can have better query time behavior and eliminates some query time errors that can occur when using `latestInterval`.
-[14319](https://github.com/apache/druid/pull/14319)
+[#14319](https://github.com/apache/druid/pull/14319)
### EXPLAIN PLAN improvements
The EXPLAIN PLAN result includes a new column `ATTRIBUTES` that describes the attributes of a query.
-See [SQL query translation](docs/querying/sql-translation.md) for more information and examples.
-[14391](https://github.com/apache/druid/pull/14391)
+For more information, see [SQL translation](https://druid.apache.org/docs/latest/querying/sql-translation.html)
+[#14391](https://github.com/apache/druid/pull/14391)
-## Metrics and monitoring
+### New query filters
-### Added a new SysMonitorOshi monitor
+TBD
-Added a new monitor `SysMonitorOshi` to replace `SysMonitor`. The new monitor has a wider support for different machine architectures including ARM instances. We recommend switching to `SysMonitorOshi` as `SysMonitor` is now deprecated and will be removed in future releases.
+## Metrics and monitoring
-[14359](https://github.com/apache/druid/pull/14359)
### Monitor for Overlord and Coordinator service health
Added a new monitor `ServiceStatusMonitor` to monitor the service health of the Overlord and Coordinator.
-[14443](https://github.com/apache/druid/pull/14443)
+[#14443](https://github.com/apache/druid/pull/14443)
+
+### New Broker metrics
-### New segment metrics
+TBD for name changes
The following metrics are now available for Brokers:
-|Metric|Description|Dimensions|Normal value|
-|------|-----------|----------|------------|
+|Metric|Description|Dimensions|
+|------|-----------|----------|
|`segment/metadatacache/refresh/count`|Number of segments to refresh in broker segment metadata cache. Emitted once per refresh per datasource.|`dataSource`| |
|`segment/metadatacache/refresh/time`|Time taken to refresh segments in broker segment metadata cache. Emitted once per refresh per datasource.|`dataSource`| |
+
+[#14453](https://github.com/apache/druid/pull/14453)
+
+### New Coordinator metrics
+
+|Metric|Description|Dimensions|Normal value|
+|------|-----------|----------| --|
| `segment/loadQueue/assigned` | Number of segments assigned for load or drop to the load queue of a server. |`dataSource`,`server` | Varies |
| `segment/loadQueue/success` |Number of segment assignments that completed successfully.|`dataSource`, `server`|Varies|
| `segment/loadQueue/cancelled` |Number of segment assignments that were canceled before completion. |`dataSource`,`server` | 0 |
| `segment/loadQueue/failed` |Number of segment assignments that failed to complete.|`dataSource`, `server`|0|
-[14453](https://github.com/apache/druid/pull/14453)
-[13197](https://github.com/apache/druid/pull/13197)
+[#13197](https://github.com/apache/druid/pull/13197)
### New metrics for task completion updates
-The following metrics have been added:
-* `task/status/queue/count`: Monitors the number of queued items
-* `task/status/updated/count`: Monitors the number of processed items
+| Metric | Description | Normal value |
+| ------ | ----------- | ---------- |
+| `task/status/queue/count` | Monitors the number of queued items | Varies |
+| `task/status/updated/count` | Monitors the number of processed items | Varies |
-[14533](https://github.com/apache/druid/pull/14533)
+[#14533](https://github.com/apache/druid/pull/14533)
### Added `groupId` to Overlord task metrics
Added `groupId` to task metrics emitted by the Overlord. This is helpful for grouping metrics like task/run/time by a single task group, such as a single compaction task or a single MSQ query.
-[14402](https://github.com/apache/druid/pull/14402)
+[#14402](https://github.com/apache/druid/pull/14402)
### New metrics for monitoring sync status of `HttpServerInventoryView`
+TBD for name change
+
|Metric|Description|Dimensions|Normal value|
|------|-----------|-----------|----------|
-|`segment/serverview/sync/healthy`|Sync status of the Coordinator/Broker with a segment-loading server such as a Historical or Peon. Emitted by the Coordinator and Brokers only when [HTTP-based server view](../configuration/index.md#segment-management) is enabled. This metric can be used in conjunction with `segment/serverview/sync/unstableTime` to debug slow startup of the Coordinator.|`server`, `tier`|1 for fully synced servers, 0 otherwise|
-|`segment/serverview/sync/unstableTime`|Time in milliseconds for which the Coordinator/Broker has been failing to sync with a segment-loading server. Emitted by the Coordinator and Brokers only when [HTTP-based server view](../configuration/index.md#segment-management) is enabled.|`server`, `tier`|Not emitted for synced servers.|
-## Cluster management
-
-### Removed unused Coordinator dynamic configuration properties
+|`segment/serverview/sync/healthy`|Sync status of the Coordinator/Broker with a segment-loading server such as a Historical or Peon. Emitted by the Coordinator and Brokers only when HTTP-based server view is enabled. This metric can be used in conjunction with `segment/serverview/sync/unstableTime` to debug slow startup of the Coordinator.|`server`, `tier`|1 for fully synced servers, 0 otherwise|
+|`segment/serverview/sync/unstableTime`|Time in milliseconds for which the Coordinator/Broker has been failing to sync with a segment-loading server. Emitted by the Coordinator and Brokers only when HTTP-based server view is enabled.|`server`, `tier`|Not emitted for synced servers.|
Review Comment:
Manually updated in https://github.com/apache/druid/pull/14600/commits/0959b288c5a56b65cd0394f4d6f4be1b810a31ff
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] DO NOT MERGE - 27.0.0 WIP release notes (druid)
Posted by "abhishekrb19 (via GitHub)" <gi...@apache.org>.
abhishekrb19 commented on code in PR #14600:
URL: https://github.com/apache/druid/pull/14600#discussion_r1266232044
##########
docs/do-not-merge.md:
##########
@@ -0,0 +1,366 @@
+<!--Intentionally, there's no Apache license so that the GHA fails it. This file is not meant to be merged.
+
+- https://github.com/apache/druid/pull/14266 - we removed input source security from 26 (https://github.com/apache/druid/pull/14003). Should we not include this in 27 release notes?
+
+-->
+
+Apache Druid 27.0.0 contains over $NUMBER_FEATURES new features, bug fixes, performance enhancements, documentation improvements, and additional test coverage from $NUMBER_OF_CONTRIBUTORS contributors.
+
+[See the complete set of changes for additional details]($LINK_TO_RELEASE_MILESTONE).
+
+Review the upgrade notes and incompatible changes before you upgrade to Druid 27.0.0.
+
+# Highlights
+
+<!-- HIGHLIGHTS H2. FOR EACH MAJOR FEATURE FOR THE RELEASE -->
+
+## Query from deep storage
+
+TBD
+
+### New statements API
+
+Added a new API /druid/v2/sql/statements/ which allows users to fetch results in an asynchronous manner.
+
+[14416](https://github.com/apache/druid/pull/14416)
+
+### New and updated fields in API to get results
+
+The API response to get results now returns a `pages` field containing information on each page of the results status.
+The `numRows` and `sizeInBytes` fields are renamed to `numTotalRows` and `totalSizeInBytes`, respectively.
+
+[14512](https://github.com/apache/druid/pull/14512)
+
+
+### Durable storage for results
+
+MSQ can now write SELECT query results to durable storage. To do so, set the context flag `selectDestination:DURABLE_STORAGE` while issuing SELECT queries to MSQ.
+
+[14527](https://github.com/apache/druid/pull/14527)
+
+## Java 17 support
+
+Druid now fully supports Java 17.
+[14384](https://github.com/apache/druid/pull/14384)
+
+## Array column types
+
+GA + TBD
+
+## Schema auto-discovery
+
+GA + TBD
+
+## Smart segment loading
+
+[13197](https://github.com/apache/druid/pull/13197)
+
+## Hadoop 2 support dropped
+
+# Additional features and improvements
+
+## MSQ task engine
+
+### `maxInputBytesPerWorker` context parameter
+
+The context parameter now denotes the estimated weighted size (in bytes) of the input to split on. The MSQ task engine now takes into account the input format and compression format instead of the actual file size reported by the file system.
+
+The default value for the context parameter has also been changed. It is now 512 MiB (previously 10 GiB).
+
+[14307](https://github.com/apache/druid/pull/14307)
+
+### Improved query planning behavior
+
+Druid now fails query planning if a CLUSTERED BY column contains descending order.
+Previously, queries would successfully plan if any CLUSTERED BY columns contained descending order.
+
+The MSQ fault, `InsertCannotOrderByDescending`, is deprecated. An INSERT or REPLACE query containing a CLUSTERED BY expression cannot be in descending order. Druid's segment generation code only supports ascending order. Instead of the fault, Druid now throws a query `ValidationException`.
+
+[14436](https://github.com/apache/druid/pull/14436)
+
+### SELECT query results
+
+SELECT queries executed using MSQ now generate only a subset of the results in the query reports.
+To fetch the complete result set, run the query using the native engine.
+
+[14370](https://github.com/apache/druid/pull/14370)
+
+
+### Other MSQ improvements
+
+- The same aggregator can now have two output names [14367](https://github.com/apache/druid/pull/14367)
+- Changed the default `clusterStatisticsMergeMode` to SEQUENTIAL (#14310)[https://github.com/apache/druid/pull/14310]
+- Added a query context parameter `MultiStageQueryContext` to determine whether the result of an MSQ select query is limited (#14476)[https://github.com/apache/druid/pull/14476]
+- Enabled using functions as inputs for `index` and `length` parameters (#14480)[https://github.com/apache/druid/pull/14480]
+
+## Ingestion
+
+### New property for task completion updates
+
+The new property `druid.indexer.queue.taskCompleteHandlerNumThreads` controls the number of threads used by the Overlord `TaskQueue` to handle task completion updates received from the workers.
+
+The following metrics have been added:
+* `task/status/queue/count`: Monitors the number of queued items
+* `task/status/updated/count`: Monitors the number of processed items
+
+[14533](https://github.com/apache/druid/pull/14533)
+
+### Improved response to max_allowed_packet limit
+
+If the Overlord fails to insert a task into the metadata because of a payload that exceeds the `max_allowed_packet` limit, the response now returns `400 Bad request` instead of `500 Internal server error`. This prevents an `index_parallel` task from retrying the insertion of a bad sub-task indefinitely and causes it to fail immediately.
+
+[14271](https://github.com/apache/druid/pull/14271)
+
+### Improved handling of mixed type arrays
+
+Druid now handles mixed type arrays such as `[["a", "b", "c"], {"x": 123}]` as `ARRAY<COMPLEX<json>` rather than throwing an incompatible type exception.
+
+[14438](https://github.com/apache/druid/pull/14438)
+
+### Other ingestion improvements
+
+* A negative streaming ingestion lag is no longer emitted as a result of stale offsets. [14292](https://github.com/apache/druid/pull/14292)
+* Removed double synchronization on simple map operations in Kubernetes task runner. [14435](https://github.com/apache/druid/pull/14435)
+* Kubernetes overlord extension now cleans up the job if the task pod fails to come up in time. [14425](https://github.com/apache/druid/pull/14425)
+
+## Querying
+
+### New function for regular expression replacement
+
+The new function `REGEXP_REPLACE` allows you to replace all instances of a pattern with a replacement string.
+
+[14460](https://github.com/apache/druid/pull/14460)
+
+### Query results directory
+
+Druid now supports a `query-results` directory in durable storage to store query results after the task finishes. The auto cleaner does not remove this directory unless the task ID is not known to the Overlord.
+
+[14446](https://github.com/apache/druid/pull/14446)
+
+### Limit for subquery results by memory usage
+
+Users can now add a guardrail to prevent subquery’s results from exceeding the set number of bytes by setting `druid.server.http.maxSubqueryRows` in the Broker's config or `maxSubqueryRows` in the query context. This feature is experimental for now and would default back to row-based limiting in case it fails to get the accurate size of the results consumed by the query.
+
+[13952](https://github.com/apache/druid/pull/13952)
+
+### HLL and Theta sketch estimates
+
+You can now use HLL_SKETCH_ESTIMATE and THETA_SKETCH_ESTIMATE as expressions. These estimates work on sketch columns and have the same behavior as `postAggs`.
+
+[14312](https://github.com/apache/druid/pull/14312)
+
+### String encoding parameter for HLL sketches
+
+HLL sketches now accept a new `stringEncoding` parameter to build sketches with UTF-8 bytes.
+
+[11201](https://github.com/apache/druid/pull/11201)
+
+### EARLIEST_BY and LATEST_BY signatures
+
+Updated EARLIEST_BY and LATEST_BY function signatures as follows:
+
+* Changed `EARLIEST(expr, timeColumn)` to `EARLIEST_BY(expr, timeColumn)`
+* Changed `LATEST(expr, timeColumn)` to `LATEST_BY(expr, timeColumn)`
+
+[14352](https://github.com/apache/druid/pull/14352)
+
+
+### Exposed Druid functions
+
+Exposed information about Druid functions in the `INFORMATION_SCHEMA.ROUTINES` table. The table contains the following columns:
+
+* ROUTINE_CATALOG: Name of the database. Always set to druid.
+* ROUTINE_SCHEMA: Name of the schema. Always set to INFORMATION_SCHEMA.
+* ROUTINE_NAME: Name of the function or operator. For example: APPROX_COUNT_DISTINCT_DS_THETA.
+* ROUTINE_TYPE: Type of routine. Always set to FUNCTION.
+* IS_AGGREGATOR: Determines if the routine is an aggregator. Returns YES for aggregator functions; NO for scalar functions.
+* SIGNATURES: Possible signatures for the routine as a string.
+
+[14378](https://github.com/apache/druid/pull/14378)
+
+### New Broker configuration for SQL schema migrations
+
+A new broker configuration, `druid.sql.planner.metadataColumnTypeMergePolicy`, adds configurable modes to how column types are computed for the SQL table schema when faced with differences between segments.
+
+The new `leastRestrictive` mode chooses the most appropriate type that data across all segments can best be coerced into and is the default behavior. This is a subtle behavior change around when segment driven schema migrations will take effect for the SQL schema:
+
+- With `leastRestrictive` mode, the schema only updates once all segments are reindexed to the new type.
+- With `latestInterval` mode, the SQL schema gets updated as soon as the first job with the new schema publishes segments in the latest time interval of the data.
+
+`leastRestrictive` can have better query time behavior and eliminates some query time errors that can occur when using `latestInterval`.
+
+[14319](https://github.com/apache/druid/pull/14319)
+
+### EXPLAIN PLAN improvements
+
+The EXPLAIN PLAN result includes a new column `ATTRIBUTES` that describes the attributes of a query.
+See [SQL query translation](docs/querying/sql-translation.md) for more information and examples.
+
+[14391](https://github.com/apache/druid/pull/14391)
+
+
+## Metrics and monitoring
+
+### Added a new SysMonitorOshi monitor
+
+Added a new monitor `SysMonitorOshi` to replace `SysMonitor`. The new monitor has a wider support for different machine architectures including ARM instances. We recommend switching to `SysMonitorOshi` as `SysMonitor` is now deprecated and will be removed in future releases.
+
+[14359](https://github.com/apache/druid/pull/14359)
+
+### Monitor for Overlord and Coordinator service health
+
+Added a new monitor `ServiceStatusMonitor` to monitor the service health of the Overlord and Coordinator.
+
+[14443](https://github.com/apache/druid/pull/14443)
+
+### New segment metrics
+
+The following metrics are now available for Brokers:
+
+|Metric|Description|Normal value|
+|------|-----------|------------|
+|`segment/metadatacache/refresh/count`|Number of segments to refresh in broker segment metadata cache. Emitted once per refresh per datasource.|`dataSource`|
+|`segment/metadatacache/refresh/time`|Time taken to refresh segments in broker segment metadata cache. Emitted once per refresh per datasource.|`dataSource`|
+
+[14453](https://github.com/apache/druid/pull/14453)
+
+### New metrics for task completion updates
+
+The following metrics have been added:
+* `task/status/queue/count`: Monitors the number of queued items
+* `task/status/updated/count`: Monitors the number of processed items
+
+[14533](https://github.com/apache/druid/pull/14533)
+
+### Added `groupId` to Overlord task metrics
+
+Added `groupId` to task metrics emitted by the Overlord. This is helpful for grouping metrics like task/run/time by a single task group, such as a single compaction task or a single MSQ query.
+
+[14402](https://github.com/apache/druid/pull/14402)
+
+## Cluster management
+
+### Enabled cancellation and prioritization of load queue items
+
+Revamped Coordinator to make it more stable and user-friendly. This is accompanied by several bug fixes, logging and metric improvements, and a whole new range of capabilities.
+
+Coordinator now supports a `smartSegmentLoading` mode, which is enabled by default. When enabled, users don't need to specify any of the following dynamic configs as they would be ignored by the coordinator. Instead, the coordinator computes the optimal values of the following configs at run time to best utilize coordinator runs:
+
+* `maxSegmentsInNodeLoadingQueue`
+* `maxSegmentsToMove`
+* `replicationThrottleLimit`
+* `useRoundRobinSegmentAssignment`
+* `useBatchedSegmentSampler`
+* `emitBalancingStats`
+
+These configs are now deprecated and will be removed in subsequent releases.
+
+Coordinator is now capable of prioritization and cancellation of items in segment load queues. This means that the coordinator now reacts faster to changes in the cluster and makes better segment assignment decisions.
+
+[13197](https://github.com/apache/druid/pull/13197)
+
+### Removed unused Coordinator dynamic configuration properties
+
+The following Coordinator dynamic configs have been removed:
+
+* `emitBalancingStats`: Stats for errors encountered while balancing will always be emitted. Other debugging stats will not be emitted but can be logged by setting the appropriate `debugDimensions`.
+* `useBatchedSegmentSampler` and `percentOfSegmentsToConsiderPerMove`: Batched segment sampling is now the standard and will always be on.
+
+[14524](https://github.com/apache/druid/pull/14524)
+
+### Improved segment balancing strategy
+
+The `cost` balancer strategy performs much better now and is capable of moving more segments in a single Coordinator run. These improvements were made by borrowing ideas from the `cachingCost` strategy.
+
+The `cachingCost` strategy itself has been known to cause issues and is now deprecated. It is likely to be removed in future Druid releases and must not be used on any cluster.
+
+[14484](https://github.com/apache/druid/pull/14484)
+
+### Enabled empty tiered replicants for load rules
+
+Druid now allows empty tiered replicants in load rules.
+
+[14432](https://github.com/apache/druid/pull/14432)
+
+### Changed Coordinator configs values
+
+The following Coordinator dynamic configs have new default values:
+
+* `maxsegmentsInNodeLoadingQueue` : 500, previously 100
+* `maxSegmentsToMove`: 100, previously 5
+* `replicationThrottleLimit`: 500, previously 10
+
+These new defaults can improve performance for most use cases.
+
+[14269](https://github.com/apache/druid/pull/14269)
+
+### Improved error messages
+
+Introduced a new unified exception, `DruidException`, for surfacing errors. It is partially compatible with the old way of reporting error messages. Response codes remain the same, all fields that previously existed on the response will continue to exist and be populated, including `errorMessage`. Some error messages have changed to be more consumable by humans and some cases have the message restructured. There should be no impact to the response codes.
+
+[14004](https://github.com/apache/druid/pull/14004)
+
+## Web console
+
+### Supervisors and Tasks
+
+Replaced the **Ingestion** view with two views: **Supervisors** and **Tasks**.
+
+[14395](https://github.com/apache/druid/pull/14395)
+
+### Added replication factor column
+
+Added a new virtual column `replication_factor` to the `sys.segments` table. This returns the total number of replicants of the segment across all tiers. The column is set to -1 if the information is not available.
+
+[14403](https://github.com/apache/druid/pull/14403)
+
+### Other web console improvements
+
+## Extensions
+
+### Improved segment metadata for Kafka emitter extension
+
+The Kafka emitter extension has been improved. You can now publish events related to segments and their metadata to Kafka.
+You can set the new properties such as in the following example:
+
+```
+druid.emitter.kafka.event.types=["metrics", "alerts", "segment_metadata"]
+druid.emitter.kafka.segmentMetadata.topic=foo
+```
+
+[14281](https://github.com/apache/druid/pull/14281)
+
+## Dependency updates
Review Comment:
Similar to the 26.0 release, should we include a "Bug fixes" section with a pointer to the list:
https://github.com/apache/druid/issues?q=is%3Aclosed+milestone%3A27.0+-label%3ABackport+label%3ABug+
For the 27.0.0 release, though, the bugs list looks relatively small - from a cursory scan of the merged PRs in 27.0, it seems like some of them weren't tagged "Bug".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org