You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by ppadma <gi...@git.apache.org> on 2018/02/23 05:32:55 UTC
[GitHub] drill pull request #1129: DRILL-6180: Use System Option "output_batch_size" ...
GitHub user ppadma opened a pull request:
https://github.com/apache/drill/pull/1129
DRILL-6180: Use System Option "output_batch_size" for External Sort
External Sort has boot time configuration for output batch size "drill.exec.sort.external.spill.merge_batch_size" which is defaulted to 16M.
To make batch sizing configuration uniform across all operators, change this to use new system option that is added "drill.exec.memory.operator.output_batch_size".
This option has default value of 32M. Changed it to 16M.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ppadma/drill DRILL-6180
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/drill/pull/1129.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1129
----
commit 5121663c1fac618d0374667c97c20570197b7455
Author: Padma Penumarthy <pp...@...>
Date: 2018-02-23T00:41:47Z
DRILL-6180: Use System Option "output_batch_size" for External Sort
----
---
[GitHub] drill issue #1129: DRILL-6180: Use System Option "output_batch_size" for Ext...
Posted by ppadma <gi...@git.apache.org>.
Github user ppadma commented on the issue:
https://github.com/apache/drill/pull/1129
@paul-rogers Made the change you suggested. Please take a look when you get a chance.
---
[GitHub] drill pull request #1129: DRILL-6180: Use System Option "output_batch_size" ...
Posted by paul-rogers <gi...@git.apache.org>.
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/1129#discussion_r171314892
--- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/xsort/managed/SortConfig.java ---
@@ -71,8 +72,8 @@
private final int mSortBatchSize;
- public SortConfig(DrillConfig config) {
-
+ public SortConfig(FragmentContext context) {
+ DrillConfig config = context.getConfig();
--- End diff --
Suggestion: pass in the original `DrillConfig` plus an option manager rather than the fragment context. The suggestion minimizes undesired dependencies.
---
[GitHub] drill pull request #1129: DRILL-6180: Use System Option "output_batch_size" ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/drill/pull/1129
---
[GitHub] drill pull request #1129: DRILL-6180: Use System Option "output_batch_size" ...
Posted by paul-rogers <gi...@git.apache.org>.
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/1129#discussion_r171315310
--- Diff: exec/java-exec/src/main/resources/drill-module.conf ---
@@ -421,7 +416,7 @@ drill.exec.options: {
drill.exec.storage.implicit.fqn.column.label: "fqn",
drill.exec.storage.implicit.suffix.column.label: "suffix",
drill.exec.testing.controls: "{}",
- drill.exec.memory.operator.output_batch_size : 33554432, # 32 MB
+ drill.exec.memory.operator.output_batch_size : 16777216, # 16 MB
--- End diff --
Thanks for making this adjustment.
---