You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2018/10/04 07:12:11 UTC
[GitHub] jihoonson commented on a change in pull request #6326: Add support
hash partitioning by a subset of dimensions to indexTask
jihoonson commented on a change in pull request #6326: Add support hash partitioning by a subset of dimensions to indexTask
URL: https://github.com/apache/incubator-druid/pull/6326#discussion_r222558067
##########
File path: docs/content/ingestion/native_tasks.md
##########
@@ -475,6 +475,7 @@ The tuningConfig is optional and default parameters will be used if no tuningCon
|maxBytesInMemory|Used in determining when intermediate persists to disk should occur. Normally this is computed internally and user does not need to set it. This value represents number of bytes to aggregate in heap memory before persisting. This is based on a rough estimate of memory usage and not actual usage. The maximum heap memory usage for indexing is maxBytesInMemory * (2 + maxPendingPersists)|1/6 of max JVM memory|no|
|maxTotalRows|Total number of rows in segments waiting for being pushed. Used in determining when intermediate pushing should occur.|20000000|no|
|numShards|Directly specify the number of shards to create. If this is specified and 'intervals' is specified in the granularitySpec, the index task can skip the determine intervals/partitions pass through the data. numShards cannot be specified if targetPartitionSize is set.|null|no|
+|partitionDimensions|The dimensions to partition on. Leave blank to select all dimensions. Only used with numShards > 1, will be ignored when targetPartitionSize or maxTotalRows is set.|null|no|
Review comment:
@himanshug @dclim thanks for the review. I believe that you guys want to add documentation about Druid's partition support rather than this specific configuration. That sounds helpful and I definitely agree with that. But, hash partitioning itself is not added in this PR. This PR just adds support for specifying partition keys. If you don't mind, I'd like to do in a follow-up PR. Does it make sense?
> Wouldn't it get ignored if forceGuaranteedRollup is false?
@dclim this is true, thanks. But, `forceGuaranteedRollup` is useful only with `numShards > 1`. If `forceGuaranteedRollup` is set, all other configurations like `maxTotalRows` or `targetPartitionSize` get ignored. This is only because we don't support range partitioning yet which can be used with `targetPartitionSize` together.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org