You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2019/08/21 21:53:13 UTC
[GitHub] [incubator-druid] clintropolis opened a new issue #8369:
0.16.0-incubating release notes
clintropolis opened a new issue #8369: 0.16.0-incubating release notes
URL: https://github.com/apache/incubator-druid/issues/8369
Apache Druid 0.16.0-incubating contains of TBD new features, performance enhancements, bug fixes, and major documentation improvements from TBD contributors.
# Highlights
## Performance
### GroupBy array-based result rows
groupBy v2 queries now use an array-based representation of result rows, rather than the map-based representation used by prior versions of Druid. This provides faster generation and processing of result sets. Out of the box this change is invisible and backwards-compatible; you will not have to change any configuration to reap the benefits of this more efficient format, and it will have no impact on cached results. Internally this format will _always_ be utilized automatically by the broker in the queries that it issues to historicals. By default the results will be translated back to the existing 'map' based format at the broker before sending them back to the client.
However, if you would like to avoid the overhead of this translation, and get even faster results,`resultAsArray` may be set on the query context to directly pass through the new array based result row format. The schema is as follows, in order:
* Timestamp (optional; only if granularity != ALL)
* Dimensions (in order)
* Aggregators (in order)
* Post-aggregators (optional; in order, if present)
See also:
* https://github.com/apache/incubator-druid/issues/8118
* https://github.com/apache/incubator-druid/pull/8196
### 'Vectorized' Query Processing
An experimental 'vectorized' query execution engine is new in 0.16.0, which can provide a speed increase in the range of 1.3-3x for timeseries and group by v2 queries. It operates on the principle of batching operations on rows instead of processing a single row at a time, e.g. iterating bitmaps in batches instead of per row, reading column values in batches, filtering in batches, aggregating values in batches, and so on. This results in significantly fewer method calls, better memory locality, and increased cache efficiency.
This is an experimental feature, but we view it as the path forward for Druid query processing and are excited for feedback as we continue to improve and fill out missing features in upcoming releases.
* Only timeseries and groupBy have vectorized engines.
* GroupBy doesn't handle multi-value dimensions or granularity other than "all" yet.
* Vector cursors cannot handle virtual columns or descending order.
* Expressions are not supported anywhere: not as inputs to aggregators, in virtual functions, or in filters.
* Only some aggregators have vectorized implementations: "count", "doubleSum", "floatSum", "longSum", "hyperUnique", and "filtered".
* Only some filters have vectorized matchers: "selector", "bound", "in", "like", "regex", "search", "and", "or", and "not".
* Dimension specs other than "default" don't work yet (no extraction functions or filtered dimension specs).
The feature can be enabled by setting `vectorize` to true in your query context (default is `false`). This works both for Druid SQL and for native queries. When set to `true`, vectorization will be used if possible; otherwise, Druid will fall back to its non-vectorized query engine. You can also set it to `"force"`, which will return an error if the query cannot be fully vectorized. This is helpful for confirming that vectorization is indeed being used.
You can control the block size during execution by setting the `vectorSize` query context parameter (default is `1000`).
See also:
* https://github.com/apache/incubator-druid/issues/7093
* https://github.com/apache/incubator-druid/pull/6794
## "Minor" compaction
Users of the Kafka indexing service and compaction and who get a trickle of late data, can find a _huge_ improvement in the form of a new concept called 'minor' compaction. Enabled by internal changes to how data segments are versioned, minor compaction is based on the idea of 'segment' based locking at indexing time instead of the current Druid locking behavior (which is now referred to as 'time chunk' locking). Segment locking as you might expect allows only the segments which are being compacted to be locked, while still allowing new 'appending' indexing tasks (like Kafka indexing tasks) to continue to run and create new segments, simulataneously. This is a big deal if you get a lot of late data, because the current behavior results in compaction tasks starving as higher priority realtime tasks hog the locks. This prevention of compaction tasks from optimizing the datasources segment sizes results in reduced overall performance.
To enable segment locking, you will need to set `forceTimeChunkLock` to `false` in the task context, or set `druid.indexer.tasklock.forceTimeChunkLock=false` in the Overlord configuration. However, *beware*, after enabling this feature, due to the changes in segment versioning, there is **no rollback** path built in, so once you upgrade to 0.16, you cannot downgrade to an older version of Druid. Because of this, we *highly recommend* confirming that Druid 0.16 is stable in your cluster before enabling this feature.
It has a humble name, but the changes of minor compaction run deep, and it is not possible to adequately describe the mechanisms that drive this in these release notes, so check out the proposal and PR for more details.
See also:
* https://github.com/apache/incubator-druid/issues/7491
* https://github.com/apache/incubator-druid/pull/7547
## Druid "indexer" process
The new Indexer process is an alternative to the MiddleManager + Peon task execution system. Instead of forking a separate JVM process per-task, the Indexer runs tasks as separate threads within a single JVM process. The Indexer is designed to be easier to configure and deploy compared to the MiddleManager + Peon system and to better enable resource sharing across tasks.
The advantage of the Indexer is that it allows query processing resources, lookups, cached authentication/authorization information, and much more to be shared between all running indexing task threads, giving each individual task access to a larger pool of resources and far fewer redundant actions done than is possible with the Peon model of execution where each task is isolated in its own process.
Using Indexer does come with one downside: the loss of process isolation provided by Peon processes means that a single task can potentially affect all running indexing tasks on that Indexer. The `druid.worker.globalIngestionHeapLimitBytes` and `druid.worker.numConcurrentMerges` configurations are meant to help minimize this.
You can start using indexing by supplying `server indexer` as the command-line argument to `org.apache.druid.cli.Main` when starting the service. To use Indexer in place of a MiddleManager and Peon, you should be able to adapt values from the configuration into the Indexer configuration, lifting `druid.indexer.fork.property.` configurations directly to the Indexer, and sizing heap and direct memory based on the Peon sizes multiplied by the number of task slots (unlike a MiddleManager, it does not accept the configurations `druid.indexer.runner.javaOpts` or `druid.indexer.runner.javaOptsArray`). See the [indexer documentation](TBD) for details.
See also:
https://github.com/apache/incubator-druid/pull/8107
## Native parallel batch indexing with shuffle
In 0.16.0, Druid's `index_parallel ` native parallel batch indexing task now supports 'perfect' rollup with the implementation of a 2 stage shuffle process.
Tasks in stage 1 perform a secondary partitioning of rows on top of the standard time based partitioning of segment granularity, creating an intermediary data segment for each partition. Stage 2 tasks are each assigned a set of the partitionings created during stage 1, and will collect and combine the set of intermediary data segments which belong to that partitioning, allowing it to achieve complete rollup when building the final segments. At this time, only hash-based partitioning is supported.
This can be enabled by setting `forceGuaranteedRollup` to `true` in the` tuningConfig`; `numShards` in `partitionsSpec` and `intervals` in `granularitySpec` must also be set.
The Druid MiddleManager (or the new Indexer) processes have a new responsibility for these indexing tasks, serving the intermediary partition segments output of stage 1 into the stage 2 tasks, so depending on configuration and cluster size, the MiddleManager jvm configuration might need to be adjusted to increase heap allocation and http threads. These numbers are expected to scale with cluster size, as all MiddleManager or Indexer processes involved in a shuffle will need the ability to communicate with each other, but we do not expect the footprint to be significantly larger than it is currently. Optimistically we suggest trying with your existing configurations, and bumping up heap and http thread count only if issues are encountered.
See also:
https://github.com/apache/incubator-druid/issues/8061
## Web Console
TBD
## SQL enchancements
TBD
### TIMESTAMPDIFF
To complement `TIMESTAMPADD` which can modify timestamps by time units, a `TIMESTAMPDIFF` which can compute the signed number of time units between two timestamps. For syntax information, check out the [SQL documentation](link TBD)
See also:
* https://github.com/apache/incubator-druid/pull/7695
### NVL
To increase permissiveness and SQL compatibility with users coming to Druid with experience in other databases, an alias for the `COALESCE` function has been added in the form of `NVL`, which some SQL dialects use instead. [See SQL documentation for details](link TBD)
See also:
* https://github.com/apache/incubator-druid/pull/7965
### TIME_CEIL
`TIME_CEIL`, a time specific, more flexible version of the `CEIL` function, has also been added to Druid 0.16.0. This function can round a timestamp up by an ISO8601 period, like P3M (quarters) or PT12H (half-days), optionally for a specific timezone. [See SQL documentation](link TBD) for additional information.
See also:
* https://github.com/apache/incubator-druid/pull/8027
### IPv4
Druid 0.16.0 also adds specialized SQL operators and native expressions for dealing with IPv4 internet addresses in dotted-decimal string or integer format. The new operators are `IPV4_MATCH(address, subnet)`, `IPV4_PARSE(address)`, and `IPV4_STRINGIFY(address)`, which can match IP addresses to subnets in CIDR notation, translate dotted-decimal string format to integer format, and translate integer format into dotted-decimal string format, respectively. [See SQL documentation](link TBD) for details.
See also:
* https://github.com/apache/incubator-druid/pull/8223
TBD:
multi-value string column support for expressions
https://github.com/apache/incubator-druid/pull/7588
## Official Apache Druid Docker image
TBD
https://jira.apache.org/jira/browse/INFRA-18887
## Extensions
### druid-datasketches
The `druid-datasketches` extension, built on top of Apache Datasketches (incubating), has been expanded with 3 new post aggregators, `quantilesDoublesSketchToRank` which computes an approximation to the rank of a given value that is the fraction of the distribution less than that value, and `quantilesDoublesSketchToCDF` which computes an approximation to the Cumulative Distribution Function given an array of split points that define the edges of the bins.
Another post aggregation, `thetaSketchToString` which will print a summary of sketch has been added to assist in debugging. See [Datasketches extension documentation](https://druid.apache.org/docs/latest/development/extensions-core/datasketches-extension.html) to learn more about this and other features.
See also:
* https://github.com/apache/incubator-druid/pull/7550
* https://github.com/apache/incubator-druid/pull/7937
The `HLLSketch` aggregator has been improved with a query-time only `round` option to support rounding values into whole numbers, to give it feature parity with the built-in `cardinality` and `hyperUnique` aggregators.
See also:
* https://github.com/apache/incubator-druid/pull/8023
Finally, users of `HllSketch` should also see a performance improvement due to some changes made which allow Druid to precompute an empty sketch and copy that into the aggregation buffers, greatly decreasing time to initialize the aggregator during query processing.
See also:
* https://github.com/apache/incubator-druid/pull/8194
### druid-stats
The `druid-stats` core extension has been enhanced with SQL support, exposing `VAR_POP` and `VAR_SAMP` to compute variance population and sample with the variance aggregator, as well as `STDDEV_POP` and `STDDEV_SAMP`to compute standard deviation population and sample using the standard deviation post aggregator. Additionally, `VARIANCE` and `STDDEV` functions are added as aliases for `VAR_SAMP` and `STDDEV_SAMP` respectively. See [SQL documentation](https://druid.apache.org/docs/latest/querying/sql.html#aggregation-functions) and [stats extension documentation](https://druid.apache.org/docs/latest/development/extensions-core/stats.html) for more details.
See also:
* https://github.com/apache/incubator-druid/pull/7801
### New extension: druid-tdigestsketch
A new set of approximate sketch aggregators for computing quantiles and the like and based on [t-digest](https://github.com/tdunning/t-digest) has been added in Druid 0.16. T-digest was designed for parallel programming use cases like distributed aggregations or map reduce jobs by making combining two intermediate t-digests easy and efficient. It serves to complement existing algorithms provided by the Apache Datasketches extension and moments sketch extension. See [the extension documentation](link TBD) for more details.
See also:
* https://github.com/apache/incubator-druid/issues/7303
* https://github.com/apache/incubator-druid/pull/7331
### New extension: druid-influxdb-emitter
A new Druid emitter extension to allow sending Druid metrics to [influxdb](https://github.com/influxdata/influxdb) over HTTP has also been added in 0.16. Currently this emitter only emits service metric events to InfluxDB (See [Druid metrics](https://druid.apache.org/docs/latest/operations/metrics.html) for a list of metrics). When a metric event is fired it is added to a queue of events. After a configurable amount of time, the events on the queue are transformed to InfluxDB's line protocol and POSTed to the InfluxDB HTTP API. The entire queue is flushed at this point. The queue is also flushed as the emitter is shutdown. See [the extension docs](link TBD) for details.
See also:
* https://github.com/apache/incubator-druid/pull/7717
### druid-histogram
TBD
Add finalizeAsBase64Binary option to FixedBucketsHistogramAggregatorFactory
https://github.com/apache/incubator-druid/pull/7784
## Fine tuning your workloads
Besides the experimental vectorized query engine and new indexer process type, 0.16 also has some additional features available to allow potentially fine tuning indexing and query performance via experimentation.
### Control Indexing Intermediary Segment Compression
First up, the ability to independently control what compression is used (or disable it) when persisting intermediary segments during indexing. This configuration available to the `indexSpec` property, and can be added to `tuningConfig` as:
```
"indexSpecForIntermediatePersists": {
"dimensionCompression": "uncompressed",
"metricCompression": "none"
}
```
for example to disable compression entirely for intermediary segments. One potential reason to consider 'uncompressed' intermediary segments is to ease up on the amount of Java 'direct' memory required to perform the final merge of intermediary segments before they are published and pushed to deep storage, as reading data from uncompressed columns does not require the 64kb direct buffers which are used to decode lz4 and other encoded columns. Of course this is a trade-off of storage space and page cache footprint, so we recommend experimenting with this before settling on a configuration to use for your production workloads.
See also:
* https://github.com/apache/incubator-druid/pull/7919
### Control Filter Bitmap Index Utilization
Bitmap indexes are usually a huge performance boost for Druid, but under some scenarios can result in _slower_ query speeds, particularly in cases of computationally expensive filters on very high cardinality dimensions. In Druid 0.16, a new mechanism to provide some manual control over when bitmap indexes are utilized, and when a filter will be done as a row scan are now in place, and available on a per filter, per query basis. Most filters will accept a new property, `filterTuning`, which might look something like this:
```
"filterTuning": {
"useIndex": true,
"minCardinalityToUseBitmapIndex": 0,
"maxCardinalityToUseBitmapIndex": 1000
}
```
`useIndex` if set to false will disallow a filter to utilize bitmap indexes, while `minCardinalityToUseBitmapIndex` and `maxCardinalityToUseBitmapIndex` allow using column cardinality to determine if bitmap indexes should be used per segment queried. All properties are optional, and default behavior if `filterTuning` is not supplied remains unchanged. Note that this feature is _not_ documented in user facing documentation, considered experimental, and subject to change in any future release.
See also:
* https://github.com/apache/incubator-druid/pull/8209
## Request Logging
If you would have liked to enable Druid request logging, but use Druid SQL and find them a bit too chatty due to all the metadata queries, you are luck with 0.16 due to a new configuration option that allows selectively muting specific types of queries from request logging. The option, `druid.request.logging.mutedQueryTypes`, accepts a list of "queryType" strings as defined by Druid's [native JSON query API](http://druid.apache.org/docs/latest/querying/querying.html), and defaults to an empty list (so nothing is ignored). For example,
```
druid.request.logging.mutedQueryTypes=["segmentMetadata", "timeBoundary"]
```
would mute all request logs for `segmentMetadata` and `timeBoundary` queries.
See also:
* https://github.com/apache/incubator-druid/pull/7562
## Refreshed website documentation
TBD
# Upgrading to Druid 0.16.0
TBD
Setting ACL on S3 task logs on similar lines as that of data segments pushed to S3
https://github.com/apache/incubator-druid/pull/7907
Enable SQL by default #7808
https://github.com/apache/incubator-druid/pull/7808
Use PartitionsSpec for all task types
https://github.com/apache/incubator-druid/pull/8141
remove deprecated standalone realtime node
https://github.com/apache/incubator-druid/pull/7915
https://github.com/apache/incubator-druid/pull/8020
Remove keepSegmentGranularity option for compaction
https://github.com/apache/incubator-druid/pull/7747
Remove LegacyKafkaIndexTaskRunner
https://github.com/apache/incubator-druid/pull/7735
After this PR, rolling update from a version < 0.12.0 is not supported.
Refactor SQLMetadataSegmentManager; Change contract of REST methods in DataSourcesResource
https://github.com/apache/incubator-druid/pull/7653
Remove Apache Pig from the tests
https://github.com/apache/incubator-druid/pull/7810
Remove obsolete isExcluded config from Kerberos authenticator
https://github.com/apache/incubator-druid/pull/7745
allow sql lookup function to take advantage of injective lookups
https://github.com/apache/incubator-druid/pull/7655
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org