You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2021/09/03 18:56:06 UTC
[GitHub] [druid] clintropolis opened a new issue #11657: [Draft] 0.22.0 Release Notes
clintropolis opened a new issue #11657:
URL: https://github.com/apache/druid/issues/11657
Apache Druid 0.22.0 contains over 400 new features, bug fixes, performance enhancements, documentation improvements, and additional test coverage from k contributors. [See the complete set of changes for additional details](https://github.com/apache/druid/milestone/43?closed=1).
# <a name="22-new-features" href="#22-new-features">#</a> New features
## <a name="22-query" href="#22-query">#</a> Query engine
### <a name="22-multi-distinct" href="#22-multi-distinct">#</a> Support for multiple distinct aggregators in same query
_todo_
https://github.com/apache/druid/pull/11014
### <a name="22-expr-agg" href="#22-expr-agg">#</a> Expression aggregator
_todo_
https://github.com/apache/druid/pull/11104
### <a name="22-array-agg" href="#22-array-agg">#</a> SQL STRING_AGG and ARRAY_AGG aggregator functions
_todo_
https://github.com/apache/druid/pull/11157
https://github.com/apache/druid/pull/11241
### <a name="22-bitwise" href="#22-bitwise">#</a> Bitwise math function expressions and aggregators
_todo_
https://github.com/apache/druid/pull/10605
https://github.com/apache/druid/pull/10823
https://github.com/apache/druid/pull/11280
### <a name="22-sql-routing" href="#22-sql-routing">#</a> SQL query routing improvements
_todo_
https://github.com/apache/druid/pull/11566
https://github.com/apache/druid/pull/11495
### <a name="22-proto-jdbc" href="#22-proto-jdbc">#</a> Avatica protobuf JDBC Support
_todo_
https://github.com/apache/druid/pull/10543
### <a name="22-query-error-log" href="#22-query-error-log">#</a> Improved query error logging
_todo_
https://github.com/apache/druid/pull/11519
## <a name="22-streaming" href="#22-streaming">#</a> Streaming Ingestion
### <a name="22-streaming-task-autoscaling" href="#22-streaming-task-autoscaling">#</a> Task autoscaling for Kafka and Kinesis streaming ingestion
_todo_
https://github.com/apache/druid/pull/10524
https://github.com/apache/druid/pull/10985
### <a name="22-avro-proto-stream" href="#22-avro-proto-stream">#</a> Avro and Protobuf streaming InputFormat and Confluent Schema Registry Support
_todo_
https://github.com/apache/druid/pull/11040
https://github.com/apache/druid/pull/11018
https://github.com/apache/druid/pull/10314
https://github.com/apache/druid/pull/10839
### <a name="22-kafka-groupid" href="#22-kafka-groupid">#</a> Kafka ingestion support for specifying group.id
_todo_
https://github.com/apache/druid/pull/11147
## <a name="22-batch" href="#22-batch">#</a> Native Batch Ingestion
### <a name="22-shuffle-deep-storage" href="#22-shuffle-deep-storage">#</a> Support for using deep storage for intermediary shuffle data
_todo_
https://github.com/apache/druid/pull/11507
### <a name="22-batch-memory" href="#22-batch-memory">#</a> Improved native batch ingestion task memory usage
_todo_
https://github.com/apache/druid/pull/11123
https://github.com/apache/druid/pull/11294
### <a name="22-batch-wait-for-handoff" href="#22-batch-wait-for-handoff">#</a> Allow batch tasks to wait until segment handoff before completion
_todo_
https://github.com/apache/druid/pull/10676
## <a name="22-compact" href="#22-compact">#</a> Data lifecycle management
### <a name="22-compact-granularity" href="#22-compact-granularity">#</a> Support managing segment and query granularity for auto-compaction
_todo_
https://github.com/apache/druid/pull/10843
https://github.com/apache/druid/pull/10856
https://github.com/apache/druid/pull/10900
https://github.com/apache/druid/pull/10912
https://github.com/apache/druid/pull/11009
### <a name="22-compact-drop" href="#22-compact-drop">#</a> Make dropExisting flag for Compaction configurable and add warning documentations
_todo_
https://github.com/apache/druid/pull/11070
### <a name="22-compact-skip-lock" href="#22-compact-skip-lock">#</a> Allow compaction to temporarily skip locked intervals
_todo_
https://github.com/apache/druid/pull/11190
### <a name="22-config-cleanup" href="#22-config-cleanup">#</a> Support for additional automatic metadata cleanup
_todo_
https://github.com/apache/druid/pull/11078
https://github.com/apache/druid/pull/11084
https://github.com/apache/druid/pull/11164
https://github.com/apache/druid/pull/11200
https://github.com/apache/druid/pull/11227
https://github.com/apache/druid/pull/11232
https://github.com/apache/druid/pull/11245
### <a name="22-data-cleanup" href="#22-data-cleanup">#</a> Dropping data
_todo_
https://github.com/apache/druid/pull/11025
https://github.com/apache/druid/pull/11501
## <a name="22-coordinator" href="#22-coordinator">#</a> Coordinator
### <a name="22-coord-timeout" href="#22-coord-timeout">#</a> Control over load/drop timeouts with Apache Zookeeper based segment management
_todo_
https://github.com/apache/druid/pull/10213
### <a name="22-coord-balance" href="#22-coord-balance">#</a> Faster coordinator segment balancing
_todo_
https://github.com/apache/druid/pull/11257
### <a name="22-coord-replication" href="#22-coord-replication">#</a> Improved loadstatus API to optionally compute under-replication based on cluster size
_todo_
https://github.com/apache/druid/pull/11056
### <a name="22-coord-rep-limit" href="#22-coord-rep-limit">#</a> Optional limits on the number of non-primary replicants loaded per coordination cycle
_todo_
https://github.com/apache/druid/pull/11135
## <a name="22-web" href="#22-web">#</a> Web Console
### <a name="22-web-general" href="#22-web-general">#</a> General improvements
_todo_
https://github.com/apache/druid/pull/10951
https://github.com/apache/druid/pull/11365
https://github.com/apache/druid/pull/10795
### <a name="22-web-query" href="#22-web-query">#</a> Query view
_todo_
https://github.com/apache/druid/pull/11158
https://github.com/apache/druid/pull/11128
https://github.com/apache/druid/pull/11203
### <a name="22-web-data" href="#22-web-data">#</a> Data management
_todo_
https://github.com/apache/druid/pull/10909
https://github.com/apache/druid/pull/11359
https://github.com/apache/druid/pull/11620
## <a name="22-metrics" href="#22-metrics">#</a> Metrics
### <a name="22-metrics-prometheus" href="#22-metrics-prometheus">#</a> Prometheus metric emitter
_todo_
https://github.com/apache/druid/pull/10412
### <a name="22-metrics-queue" href="#22-metrics-queue">#</a> New metric ingest/notices/queueSize for supervisor monitoring
_todo_
https://github.com/apache/druid/pull/11417
### <a name="22-metrics-count" href="#22-metrics-count">#</a> New metric query/segments/count for the number of segments which participate in a query
_todo_
https://github.com/apache/druid/pull/11394
## <a name="22-clouds" href="#22-clouds">#</a> Cloud integrations
### <a name="22-clouds-web-ident" href="#22-clouds-web-ident">#</a> AWS Web Identity / IRSA Support
_todo_
https://github.com/apache/druid/pull/10541
### <a name="22-clouds-assume-role" href="#22-clouds-assume-role">#</a> S3 ingestion support for assuming a role
_todo_
https://github.com/apache/druid/pull/10995
### <a name="22-clouds-gcs-lookup" href="#22-clouds-gcs-lookup">#</a> Google Cloud Storage support for URI lookups
_todo_
https://github.com/apache/druid/pull/11026
## <a name="22-odds-and-ends" href="#22-odds-and-ends">#</a> Other changes
### <a name="22-avro-union-by-type" href="#22-avro-union-by-type">#</a> Extracting Avro union fields by type
_todo_
https://github.com/apache/druid/pull/10505
### <a name="22-mysql-mariadb" href="#22-mysql-mariadb">#</a> Support using MariaDb connector with MySQL extensions
_todo_
https://github.com/apache/druid/pull/11402
### <a name="22-dynamic-config" href="#22-dynamic-config">#</a> Add Environment Variable DynamicConfigProvider
_todo_
https://github.com/apache/druid/pull/11377
### <a name="22-dynamic-config-schema-registry" href="#22-dynamic-config-schema-registry">#</a> Add DynamicConfigProvider for Schema Registry
_todo_
https://github.com/apache/druid/pull/11362
# <a name="22-security" href="#22-security">#</a> Bug fixes
## <a name="22-security-protos" href="#22-security-protos">#</a> Control of allowed protocols for HTTP and HDFS input sources
_todo_
https://github.com/apache/druid/pull/10830
## <a name="22-security-ldap" href="#22-security-ldap">#</a> Fix expiration logic for LDAP internal credential cache
_todo_
https://github.com/apache/druid/pull/11395
# <a name="22-perf-improvements" href="#22-perf-improvements">#</a> Performance improvements
## <a name="22-perf-query" href="#22-perf-query">#</a> Query engine
### <a name="22-perf-general" href="#22-perf-general">#</a> General performance
_todo_
https://github.com/apache/druid/pull/11171
https://github.com/apache/druid/pull/10904
### <a name="22-perf-grouper" href="#22-perf-grouper">#</a> SQL group-by query will now use query granularity when possible
_todo_
https://github.com/apache/druid/pull/11379
### <a name="22-perf-joins" href="#22-perf-joins">#</a> JOIN query enhacements
_todo_
https://github.com/apache/druid/pull/11068
https://github.com/apache/druid/pull/10697
https://github.com/apache/druid/pull/10947
### <a name="22-perf-metadata" href="#22-perf-metadata">#</a> Segment metadata query improvements
_todo_
https://github.com/apache/druid/pull/10892
## <a name="22-perf-vector" href="#22-perf-vector">#</a> Vectorized query engine
### <a name="22-vector-aggs" href="#22-vector-aggs">#</a> DataSketches theta and quantiles aggregators
_todo_
https://github.com/apache/druid/pull/11183
https://github.com/apache/druid/pull/10767
### <a name="22-vector-aggs" href="#22-vector-aggs">#</a> Cardinality aggregator
_todo_
https://github.com/apache/druid/pull/11182
### <a name="22-vector-expr" href="#22-vector-expr">#</a> Expression processing improvements
_todo_
https://github.com/apache/druid/pull/10613
https://github.com/apache/druid/pull/11010
https://github.com/apache/druid/pull/11213
### <a name="22-vector-longs" href="#22-vector-longs">#</a> Improved LONG column scan speeds, especially for 'auto' encoded columns
_todo_
https://github.com/apache/druid/pull/11004
https://github.com/apache/druid/pull/11039
## <a name="22-perf-sql" href="#22-perf-sql">#</a> SQL
### <a name="22-jdbc-frame" href="#22-jdbc-frame">#</a> Druid JDBC server result batch size configuration
_todo_
https://github.com/apache/druid/pull/10880
### <a name="22-sql-parse" href="#22-sql-parse">#</a> SQL parsing improvements
_todo_
https://github.com/apache/druid/pull/11041
### <a name="22-sys-seg" href="#22-sys-seg">#</a> Improved SYS.SEGMENTS performance
_todo_
https://github.com/apache/druid/pull/11008
### <a name="22-sql-broker" href="#22-sql-broker">#</a> Improve broker SQL schema building
_todo_
https://github.com/apache/druid/pull/11457
# <a name="22-bug-fixes" href="#22-bug-fixes">#</a> Bug fixes
Druid 0.22.0 contains over 80 bug fixes, you can see [the complete list here](https://github.com/apache/druid/pulls?q=label%3ABug+milestone%3A0.22.0+is%3Aclosed+).
# <a name="22-upgrading-to-22" href="#22-upgrading-to-22">#</a> Upgrading to 0.22.0
Consider the following changes and updates when upgrading from Druid 0.21.x to 0.22.0. If you're updating from an earlier version than 0.21.0, see the release notes of the relevant intermediate versions.
### <a name="22-dropped-zk-3.4" href="#22-dropped-zk-3.4">#</a> Dropped support for Apache ZooKeeper 3.4
Following up to 0.21, which officially deprecated support for Zookeeper 3.4, [which has been end-of-life for a while](https://mail-archives.apache.org/mod_mbox/zookeeper-user/202004.mbox/%3C41A7EC67-D8F4-4C3A-B2DB-C2741C2EECA3%40apache.org%3E), support for ZooKeeper 3.4 is now removed in 0.22.0. Be sure to upgrade your Zookeeper cluster prior to upgrading your Druid cluster to 0.22.0.
https://github.com/apache/druid/issues/10780
https://github.com/apache/druid/pull/11073
DruidInputSource: Fix issues in column projection, timestamp handling.
https://github.com/apache/druid/pull/10267
Fix idempotence of segment allocation and task report apis in native batch ingestion
https://github.com/apache/druid/pull/11189
SQL timeseries no longer skip empty buckets with all granularity
https://github.com/apache/druid/pull/11188
Web console: Remove support for IE11 and other older browsers
https://github.com/apache/druid/pull/11357
Update default maxSegmentsInNodeLoadingQueue
https://github.com/apache/druid/pull/11540
# <a name="22-dev-notes" href="#22-dev-notes">#</a> Developer notices
enrich expression cache key information to support expressions which depend on external state
https://github.com/apache/druid/pull/11358
Replace Processing ExecutorService with QueryProcessingPool
https://github.com/apache/druid/pull/11382
Make SegmentLoader extensible and customizable
https://github.com/apache/druid/pull/11398
# <a name="22-known-issues" href="#22-known-issues">#</a> Known issues
For a full list of open issues, please see https://github.com/apache/druid/labels/Bug.
# <a name="22-credits" href="#22-credits">#</a> Credits
Thanks to everyone who contributed to this release!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] clintropolis commented on issue #11657: [Draft] 0.22.0 Release Notes
Posted by GitBox <gi...@apache.org>.
clintropolis commented on issue #11657:
URL: https://github.com/apache/druid/issues/11657#issuecomment-925323301
>The linked docs and the linked PRs make it look like only manual compaction supports changing query granularity. Either this sentence is wrong, or the docs should be updated (and more PRs should be linked). cc @maytasm who did most of the linked PRs.
Good catch, this was my mistake when doing the writeup :+1:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] glasser commented on issue #11657: [Draft] 0.22.0 Release Notes
Posted by GitBox <gi...@apache.org>.
glasser commented on issue #11657:
URL: https://github.com/apache/druid/issues/11657#issuecomment-924390303
> Druid manual and automatic compaction can now be configured to change segment and query granularity.
The linked docs and the linked PRs make it look like only manual compaction supports changing query granularity. Either this sentence is wrong, or the docs should be updated (and more PRs should be linked). cc @maytasm who did most of the linked PRs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] clintropolis closed issue #11657: [Draft] 0.22.0 Release Notes
Posted by GitBox <gi...@apache.org>.
clintropolis closed issue #11657:
URL: https://github.com/apache/druid/issues/11657
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org