You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@drill.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2017/08/25 23:42:00 UTC
[jira] [Commented] (DRILL-5741) During startup Drill should not
exceed the available memory
[ https://issues.apache.org/jira/browse/DRILL-5741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16142446#comment-16142446 ]
ASF GitHub Bot commented on DRILL-5741:
---------------------------------------
GitHub user kkhatua opened a pull request:
https://github.com/apache/drill/pull/922
DRILL-5741: During startup Drill should not exceed the available memory
Providing an environment variable - DRILLBIT_MAX_PROC_MEM to ensure that a Drillbit's max memory parameters, cumulatively, don't exceed the value specified.
The variable can be defined in KB, MB, or GB; similar in syntax to how the JVM MaxHeap is specified.
e.g.
```
DRILLBIT_MAX_PROC_MEM=13G
DRILLBIT_MAX_PROC_MEM=8192m
DRILLBIT_MAX_PROC_MEM=4194304K
```
In addition, you can specify it as a percent of the available free memory prior to the Drillbit starting up:
`DRILLBIT_MAX_PROC_MEM=40%
`
For a system with with 28GB free memory, when set to 40% the Drillbit (with settings defined in drill-env.sh) fails startup with the following message:
```
2017-08-25 14:58:57 [ERROR] Unable to start Drillbit due to memory constraint violations
Max Memory Permitted : 40% (11 of 28 GB free memory)
Total Memory Requested : 26 GB
Check the following settings to possibly modify (or increase the Max Memory Permitted):
-Xmx8g
-XX:MaxDirectMemorySize=16g
-XX:ReservedCodeCacheSize=1G
-XX:MaxPermSize=512M
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/kkhatua/drill DRILL-5741
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/drill/pull/922.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #922
----
commit 04b04d8e4b0c24fef053bbbd6ecebfaebad99d9c
Author: Kunal Khatua <kk...@maprtech.com>
Date: 2017-08-25T23:35:37Z
DRILL-5741: During startup Drill should not exceed the available memory
Providing an environment variable - DRILLBIT_MAX_PROC_MEM to ensure that a Drillbit's max memory parameters, cumulatively, don't exceed the value specified.
The variable can be defined in KB, MB, or GB; similar in syntax to how the JVM MaxHeap is specified.
e.g.
```
DRILLBIT_MAX_PROC_MEM=13G
DRILLBIT_MAX_PROC_MEM=8192m
DRILLBIT_MAX_PROC_MEM=4194304K
```
In addition, you can specify it as a percent of the available free memory prior to the Drillbit starting up:
`DRILLBIT_MAX_PROC_MEM=40%
`
For a system with with 28GB free memory, when set to 40% the Drillbit (with settings defined in drill-env.sh) fails startup with the following message:
```
2017-08-25 14:58:57 [ERROR] Unable to start Drillbit due to memory constraint violations
Max Memory Permitted : 40% (11 of 28 GB free memory)
Total Memory Requested : 26 GB
Check the following settings to possibly modify (or increase the Max Memory Permitted):
-Xmx8g
-XX:MaxDirectMemorySize=16g
-XX:ReservedCodeCacheSize=1G
-XX:MaxPermSize=512M
```
----
> During startup Drill should not exceed the available memory
> -----------------------------------------------------------
>
> Key: DRILL-5741
> URL: https://issues.apache.org/jira/browse/DRILL-5741
> Project: Apache Drill
> Issue Type: Improvement
> Components: Server
> Affects Versions: 1.11.0
> Reporter: Kunal Khatua
> Assignee: Kunal Khatua
> Fix For: 1.12.0
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> Currently, during startup, a Drillbit can be assigned large values for the following:
> * Xmx (Heap)
> * XX:MaxDirectMemorySize
> * XX:ReservedCodeCacheSize
> * XX:MaxPermSize
> All of this, potentially, can exceed the available memory on a system when a Drillbit is under heavy load. It would be good to have the Drillbit ensure during startup itself that the cumulative value of these parameters does not exceed a pre-defined upper limit for the Drill process.
> The proposal is to have the [runbit|https://github.com/apache/drill/blob/master/distribution/src/resources/runbit] script look for an additional environment variable:
> {{DRILLBIT_MAX_PROC_MEM}}
> The parameter can specify the maximum in GB/MB (similar in syntax to how the Java's MaxHeap is defined), or in terms of percentage of available memory (not to exceed 95%).
> The [runbit|https://github.com/apache/drill/blob/master/distribution/src/resources/runbit] script will perform the calculation of the sum of memory required by the memory spaces (heap, direct, etc) and ensure that it is within the limit defined by the {{DRILLBIT_MAX_PROC_MEM}} env variable.
> In the absence of this parameter, there will be no restriction. A node admin can then define this variable in the default terminal's environment (e.g. {{/root/.bashrc}} ) files.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)