You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2022/03/23 04:35:48 UTC
[GitHub] [druid] kfaraz commented on a change in pull request #12350: Improved docs for range partitioning.
kfaraz commented on a change in pull request #12350:
URL: https://github.com/apache/druid/pull/12350#discussion_r832838513
##########
File path: docs/ingestion/native-batch.md
##########
@@ -387,19 +400,44 @@ them to create the final segments. Finally, they push the final segments to the
> the task may fail if the input changes in between the two passes.
#### Multi-dimension range partitioning
-> Multiple dimension (multi-dimension) range partitioning is an experimental feature. Multi-dimension range partitioning is currently not supported in the sequential mode of the Parallel task.
-When you use multi-dimension partitioning for your data, Druid is able to distribute segment sizes more evenly than with single dimension partitioning.
+> Multiple dimension (multi-dimension) range partitioning is an experimental feature.
+> Multi-dimension range partitioning is currently not supported in the sequential mode of the
+> Parallel task.
-For segment pruning to be effective and translate into better query performance, you must include the first of your `partitionDimensions` in the `WHERE` clause at query time. For example, given the following `partitionDimensions`:
-```
- "partitionsSpec": {
- "type": "range",
- "partitionDimensions":["coutryName","cityName"],
- "targetRowsPerSegment" : 5000
+When you use multi-dimension range partitioning for your data, Druid is able to distribute segment
+sizes more evenly than with single dimension partitioning.
+
+Range partitioning has several benefits:
+
+1. Lower storage footprint due to combining similar data into the same segments, which improves compressibility.
+2. Better query performance due to Broker-level segment pruning, which removes segments from
+ consideration when they cannot possibly contain data matching the query filter.
+
+For Broker-level segment pruning to be effective, you must include a set of `partitionDimensions`,
+starting from the left, in the `WHERE` clause at query time using filters that support pruning.
+Filters that support pruning include:
+
+- Equality on literals that match the type of the column, like `x = 'foo'` and `x IN ('foo', 'bar')`
Review comment:
Isn't type of the dimensions always string (atleast for now)?
Also, the pruning happens assuming a RangeSet of String using string comparisons only, even for equality.
Please correct me if I am missing something.
##########
File path: docs/ingestion/native-batch.md
##########
@@ -387,19 +400,44 @@ them to create the final segments. Finally, they push the final segments to the
> the task may fail if the input changes in between the two passes.
#### Multi-dimension range partitioning
-> Multiple dimension (multi-dimension) range partitioning is an experimental feature. Multi-dimension range partitioning is currently not supported in the sequential mode of the Parallel task.
-When you use multi-dimension partitioning for your data, Druid is able to distribute segment sizes more evenly than with single dimension partitioning.
+> Multiple dimension (multi-dimension) range partitioning is an experimental feature.
+> Multi-dimension range partitioning is currently not supported in the sequential mode of the
+> Parallel task.
-For segment pruning to be effective and translate into better query performance, you must include the first of your `partitionDimensions` in the `WHERE` clause at query time. For example, given the following `partitionDimensions`:
-```
- "partitionsSpec": {
- "type": "range",
- "partitionDimensions":["coutryName","cityName"],
- "targetRowsPerSegment" : 5000
+When you use multi-dimension range partitioning for your data, Druid is able to distribute segment
+sizes more evenly than with single dimension partitioning.
+
+Range partitioning has several benefits:
+
+1. Lower storage footprint due to combining similar data into the same segments, which improves compressibility.
+2. Better query performance due to Broker-level segment pruning, which removes segments from
+ consideration when they cannot possibly contain data matching the query filter.
+
+For Broker-level segment pruning to be effective, you must include a set of `partitionDimensions`,
+starting from the left, in the `WHERE` clause at query time using filters that support pruning.
+Filters that support pruning include:
+
+- Equality on literals that match the type of the column, like `x = 'foo'` and `x IN ('foo', 'bar')`
+ where `x` is a string; or `x = 3` and `x IN (2, 3)` where `x` is numeric.
+- Comparison between string columns and string literals, like `x < 'foo'` or other comparisons
+ involving `<`, `>`, `<=`, or `>=`. Comparisons involving numbers do not support pruning.
Review comment:
This part is probably not needed, if the above comment regarding type of dimensions always being string holds true.
```suggestion
involving `<`, `>`, `<=`, or `>=`.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org