You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@drill.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2016/12/13 23:12:58 UTC
[jira] [Commented] (DRILL-5104) Foreman sets external sort memory
allocation even for a physical plan
[ https://issues.apache.org/jira/browse/DRILL-5104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15746568#comment-15746568 ]
ASF GitHub Bot commented on DRILL-5104:
---------------------------------------
GitHub user paul-rogers opened a pull request:
https://github.com/apache/drill/pull/694
DRILL-5104: Foreman should not alter physical plan memory
Foreman should not set external sort memory for a physical plan
Physical plans include a plan for memory allocations. However, the code
path in Foreman replans external sort memory, even for a physical plan.
This makes it impossible to use a physical plan to test memory
configuration.
This change avoids changing memory settings in a physical plan; while
preserving the adjustments for logical plans or SQL queries.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/paul-rogers/drill DRILL-5104
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/drill/pull/694.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #694
----
commit 9c99c453cf4e7eeef4ca61a127caf8b79cdb3fb9
Author: Paul Rogers <pr...@maprtech.com>
Date: 2016-12-13T22:36:42Z
DRILL-5104: Foreman should not set external sort memory for a physical plan
Physical plans include a plan for memory allocations. However, the code
path in Foreman replans external sort memory, even for a physical plan.
This makes it impossible to use a physical plan to test memory
configuration.
This change avoids changing memory settings in a physical plan; while
preserving the adjustments for logical plans or SQL queries.
----
> Foreman sets external sort memory allocation even for a physical plan
> ---------------------------------------------------------------------
>
> Key: DRILL-5104
> URL: https://issues.apache.org/jira/browse/DRILL-5104
> Project: Apache Drill
> Issue Type: Sub-task
> Affects Versions: 1.8.0
> Reporter: Paul Rogers
> Assignee: Paul Rogers
>
> Consider the (disabled) unit test {{TestSimpleExternalSort.outOfMemoryExternalSort}} which uses the physical plan {{xsort/oom_sort_test.json}} that contains a setting for the amount of memory to allocate:
> {code}
> {
> ...
> pop:"external-sort",
> ...
> initialAllocation: 1000000,
> maxAllocation: 30000000
> },
> {code}
> When run, the amount of memory is set to 715827882. The reason is that code was added to {{Foreman}} to compute the memory to allocate to the external sort:
> {code}
> private void runPhysicalPlan(final PhysicalPlan plan) throws ExecutionSetupException {
> validatePlan(plan);
> MemoryAllocationUtilities.setupSortMemoryAllocations(plan, queryContext);
> {code}
> The problem is that a physical plan should execute as provided to enable detailed testing.
> To solve this problem, move the sort memory setup to the path taken by SQL queries, but not via physical plans.
> This change is necessary to re-enable the previously-disabled external sort tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)