You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2019/02/22 03:09:54 UTC
[GitHub] jon-wei opened a new issue #7126: [WIP] 0.14.0-incubating release
notes
jon-wei opened a new issue #7126: [WIP] 0.14.0-incubating release notes
URL: https://github.com/apache/incubator-druid/issues/7126
Druid 0.14.0-incubating contains over 200 new features, performance/stability/documentation improvements, and bug fixes from 56 contributors. Major new features and improvements include:
- New web console
- Kinesis indexing service
- Maintenance mode for Historicals
- Published segment cache in Broker
- Bloom filter aggregator and expression
- Force push down option for nested GroupBy queries
- Better segment handoff and drop rule handling
- Automatically kill MapReduce jobs when Hadoop ingestion tasks are killed
- DogStatsD tag support for statsd emitter
- New API for retrieving all lookup specs
- New compaction options
- More efficient cachingCost segment balancing strategy
The full list of changes is here: https://github.com/apache/incubator-druid/pulls?q=is%3Apr+is%3Amerged+milestone%3A0.14.0
Documentation for this release is at: http://druid.io/docs/0.14.0-incubating/
# Highlights
## New web console
Druid has a new web console that provides functionality that was previously split between the coordinator and overlord consoles.
The new console allows the user to manage datasources, segments, tasks, data processes (Historicals and MiddleManagers), and coordinator dynamic configuration. The user can also run SQL and native Druid queries within the console.
For more details, please see http://druid.io/docs/0.14.0-incubating/operations/web-consoles.html
Added by @vogievetsky in #6923.
## Kinesis indexing service
Druid now supports ingestion from Kinesis streams, provided by the new `druid-kinesis-indexing-service` core extension.
Please see http://druid.io/docs/latest/development/extensions-core/kinesis-ingestion.html for details.
Added by @jsun98 in #6431.
## Maintenance mode for Historicals
Historical processes can now be put into a "maintenance" mode, where the coordinator will no longer consider the Historical process as a target for segment replication. The coordinator will also move segments off the process in maintenance.
This is controlled via Coordinator dynamic configuration. For more details, please see http://druid.io/docs/0.14.0-incubating/configuration/index.html#dynamic-configuration.
Added by @egor-ryashin in #6349.
## Published segment cache on Broker
The Druid Broker now has the ability to maintain a cache of published segments via polling the Coordinator, which can significantly improve response time for metadata queries on the `sys.segments` system table.
Please see http://druid.io/docs/0.14.0-incubating/querying/sql.html#retrieving-metadata for details.
Added by @surekasaharan in #6901
## Bloom filter aggregator and expression
A new aggregator for constructing Bloom filters at query time and support for performing Bloom filter checks within Druid expressions have been added to the `druid-bloom-filter` extension.
Please see http://druid.io/docs/0.14.0-incubating/development/extensions-core/bloom-filter.html
Added by @clintropolis in #6904 and #6397
## Force push down option for nested GroupBy queries
Outer query execution for nested GroupBy queries can now be pushed down to Historical processes; previously, the outer queries would always be executed on the Broker.
Please see https://github.com/apache/incubator-druid/pull/5471 for details.
Added by @samarthjain in #5471.
## Better segment handoff and retention rule handling
Segment handoff will now ignore segments that would be dropped by a datasource's retention rules, avoiding ingestion failures caused by issue #5868.
Period load rules will now include the future by default.
A new "Period Drop Before" rule has been added. Please see http://druid.io/docs/0.14.0-incubating/operations/rule-configuration.html#period-drop-before-rule for details.
Added by @QiuMM in #6676, #6414, and 6415.
## Automatically kill MapReduce jobs when Hadoop ingestion tasks are killed
Druid will now automatically terminate MapReduce jobs created by Hadoop batch ingestion tasks when the ingestion task is killed.
Added by @ankit0811 in #6828.
## DogStatsD tag support for statsd-emitter
The `statsd-emitter` extension now supports DogStatsD-style tags. Please see http://druid.io/docs/0.14.0-incubating/development/extensions-contrib/statsd.md
Added by @deiwin in #6605, with support for constant tags added by @glasser in #6791.
## New API for retrieving all lookup specs
A new API for retrieving all lookup specs for all tiers has been added. Please see http://druid.io/docs/0.14.0-incubating/querying/lookups#get-all-lookups for details.
Added by @jihoonson in #7025.
## New compaction options
Auto-compaction now supports the `maxRowsPerSegment` option. Please see http://druid.io/docs/0.14.0-incubating/design/coordinator.html#compacting-segments for details.
The compaction task now supports a new `segmentGranularity` option, deprecating the older `keepSegmentGranularity` option for controlling the segment granularity of compacted segments. Please see the `segmentGranularity` table in http://druid.io/docs/0.14.0-incubating/ingestion/compaction.html for more information on these properties.
Added by @jihoonson in #6758 and #6780.
## More efficient cachingCost segment balancing strategy
The `cachingCost` Coordinator segment balancing strategy will now only consider Historical processes for balancing decisions. Previously the strategy would unnecessarily consider active worker tasks as well, which are not targets for segment replication.
Added by @QiuMM in #6879.
## New metrics:
- New allocation rate metric `jvm/heapAlloc/bytes`, added by @egor-ryashin in #6710.
- New query count metric ``query/count`, added by @QiuMM in #6473
## New interfaces for extension developers
### RequestLogEvent
It is now possible to control the fields in `RequestLogEvent`, emitted by `EmittingRequestLogger`. Please see #6477 for details. Added by @leventov.
### Custom TLS certificate checks
An extension point for custom TLS certificate checks has been added. Please see http://druid.io/docs/0.14.0-incubating/operations/tls-support.html#custom-tls-certificate-checks for details. Added by @jon-wei in #6432.
## SQL Enhancements
- Enhancements to dsql. https://github.com/apache/incubator-druid/pull/6929
- SQL: Add support for queries with project-after-semijoin. https://github.com/apache/incubator-druid/pull/6756
- SQL: Support for selecting multi-value dimensions. https://github.com/apache/incubator-druid/pull/6462
- SQL: Support AVG on system tables. https://github.com/apache/incubator-druid/pull/6601
- SQL: Add "POSITION" function. https://github.com/apache/incubator-druid/pull/6596
- SQL: Set INFORMATION_SCHEMA catalog name to "druid". https://github.com/apache/incubator-druid/pull/6595
- SQL: Fix ordering of sort, sortProject in DruidSemiJoin. https://github.com/apache/incubator-druid/pull/6769
- Add SQL id, request logs, and metrics https://github.com/apache/incubator-druid/pull/6302
- Add more sketch aggregator support in Druid SQL https://github.com/apache/incubator-druid/pull/6951
- bloom filter sql https://github.com/apache/incubator-druid/pull/6502
## Kafka Indexing Service no longer experimental
# Updating from 0.13.0-incubating and earlier
- overhaul 'druid-parquet-extensions' module, promoting from 'contrib' to 'core' https://github.com/apache/incubator-druid/pull/6360
druid-parquet-extensions: delete old extension from "extensions-contrib" before restarting with new version
## Behavior changes
- tasks tables in metadata storage are not cleared #6592
- autosize processing buffers based on direct memory sizing by default https://github.com/apache/incubator-druid/pull/6588
- update insert pending segments logic to synchronous https://github.com/apache/incubator-druid/pull/6336
- Period load/drop/broadcast rule include the future by default https://github.com/apache/incubator-druid/pull/6414
## Property changes
change propertyBase in ServerViewModule #6774
move parquet extension input formats #6727
fix missing property in JsonTypeInfo of SegmentWriteOutMediumFactory #6656
## Deprecations
Deprecate IntervalChunkingQueryRunner #6591
ApproxHistogram, HLL (link to doc page)
keepSegmentGranularity compaction
## Interface changes for extension developers
Introduce SegmentId class #6370
combine druid-api, druid-common, java-util into druid-core #6443
# Credits
Thanks to everyone who contributed to this release!
@a2l007
@AlexanderSaydakov
@anantmf
@ankit0811
@asdf2014
@awelsh93
@benhopp
@Caroline1000
@clintropolis
@dclim
@deiwin
@DiegoEliasCosta
@drcrallen
@dyf6372
@Dylan1312
@egor-ryashin
@elloooooo
@evans
@FaxianZhao
@gaodayue
@gianm
@glasser
@Guadrado
@gvsmirnov
@hate13
@hoesler
@hpandeycodeit
@janeklb
@jihoonson
@jon-wei
@jorbay-au
@jsun98
@justinborromeo
@kamaci
@leventov
@lxqfy
@navkumar
@niketh
@patelh
@pzhdfy
@QiuMM
@rcgarcia74
@richardstartin
@robertervin
@samarthjain
@seoeun25
@Shimi
@surekhasaharan
@taiii
@thomask
@TwojaWina
@VincentNewkirk
@vogievetsky
@vvararu
@yunwan
@zhaojiandong
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org