You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@drill.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2018/01/03 22:40:00 UTC
[jira] [Commented] (DRILL-5741) Automatically manage memory
allocations during startup
[ https://issues.apache.org/jira/browse/DRILL-5741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310408#comment-16310408 ]
ASF GitHub Bot commented on DRILL-5741:
---------------------------------------
GitHub user kkhatua opened a pull request:
https://github.com/apache/drill/pull/1082
DRILL-5741: Automatically manage memory allocations during startup
Providing an environment variable - DRILLBIT_MAX_PROC_MEM to ensure that a Drillbit's max memory parameters, cumulatively, don't exceed the value specified.
The variable can be defined in KB, MB, or GB; similar in syntax to how the JVM MaxHeap is specified. For e.g.
```
DRILLBIT_MAX_PROC_MEM=13G
DRILLBIT_MAX_PROC_MEM=8192m
DRILLBIT_MAX_PROC_MEM=4194304K
```
In addition, you can specify it as a percent of the total sytem memory prior to the Drillbit starting up:
`DRILLBIT_MAX_PROC_MEM=40%`
For a system with with 48GB free memory, when set to (say) 25% (with settings defined in drill-env.sh), and heap (8GB) and direct (10GB) are defined; the Drillbit fails startup with the following message:
```
2018-01-03 14:27:57 [WARN] 25% of System Memory (47 GB) translates to 12 GB
2018-01-03 14:27:57 [ERROR] Unable to start Drillbit due to memory constraint violations
Total Memory Requested : 19 GB
Check the following settings to possibly modify (or increase the Max Memory Permitted):
DRILLBIT_MAX_PROC_MEM=25%
DRILL_HEAP=8G
DRILL_MAX_DIRECT_MEMORY=10G
DRILLBIT_CODE_CACHE_SIZE=1024m
*NOTE: It is recommended not to specify DRILLBIT_CODE_CACHE_SIZE as this will be auto-computed based on the HeapSize and would not exceed 1GB
```
For all other combinations, the undefined parameters are adjusted to ensure that the total memory allocated is within the value specified by `DRILLBIT_MAX_PROC_MEM`,
For a system with with 48GB free memory, when set to (say) 50% (with settings defined in drill-env.sh), and heap (8GB) and direct (10GB) are defined; the Drillbit startup with the following warning:
```
2018-01-03 14:31:06 [WARN] 50% of System Memory (47 GB) translates to 24 GB
2018-01-03 14:31:06 [WARN] You have an allocation of 4 GB that is currently unused from a total of 24 GB. You can increase your existing memory configuration to use this extra memory
DRILLBIT_MAX_PROC_MEM=50%
DRILL_HEAP=8G
DRILL_MAX_DIRECT_MEMORY=10G
DRILLBIT_CODE_CACHE_SIZE=1024m
*NOTE: It is recommended not to specify DRILLBIT_CODE_CACHE_SIZE as this will be auto-computed based on the HeapSize and would not exceed 1GB
```
In addition, if the available free memory is less than the allocation, an additional warning is provided under the assumption that the OS will reclaim more free memory when required:
```
2018-01-03 14:31:06 [WARN] Total Memory Allocation for Drillbit (19GB) exceeds available free memory (11GB)
2018-01-03 14:31:06 [WARN] Drillbit will start up, but can potentially crash due to oversubscribing of system memory.
```
For more details, refer the attachments in https://issues.apache.org/jira/browse/DRILL-5741
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/kkhatua/drill DRILL-5741
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/drill/pull/1082.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1082
----
commit 2fea048835f729175f77b0a0dea731f741bb70e9
Author: Kunal Khatua <kk...@...>
Date: 2018-01-03T21:22:52Z
DRILL-6068: Support user/distrib-specific config checks during startup
1. Allows for distrib/user specific checks to be done
2. Place-holder files for distribution and user specific checks
3. Moved JVM Version Check to head of script
commit 266529f2338c607bd7845d408cddb721a41ae4ac
Author: Kunal Khatua <kk...@...>
Date: 2018-01-03T22:15:03Z
DRILL-5741: Automatically manage memory allocations during startup
Providing an environment variable - DRILLBIT_MAX_PROC_MEM to ensure that a Drillbit's max memory parameters, cumulatively, don't exceed the value specified.
The variable can be defined in KB, MB, or GB; similar in syntax to how the JVM MaxHeap is specified.
e.g.
```
DRILLBIT_MAX_PROC_MEM=13G
DRILLBIT_MAX_PROC_MEM=8192m
DRILLBIT_MAX_PROC_MEM=4194304K
```
In addition, you can specify it as a percent of the total sytem memory prior to the Drillbit starting up:
`DRILLBIT_MAX_PROC_MEM=40%`
For a system with with 48GB free memory, when set to (say) 25% (with settings defined in drill-env.sh), and heap (8GB) and direct (10GB) are defined; the Drillbit fails startup with the following message:
```
2018-01-03 14:27:57 [WARN] 25% of System Memory (47 GB) translates to 12 GB
2018-01-03 14:27:57 [ERROR] Unable to start Drillbit due to memory constraint violations
Total Memory Requested : 19 GB
Check the following settings to possibly modify (or increase the Max Memory Permitted):
DRILLBIT_MAX_PROC_MEM=25%
DRILL_HEAP=8G
DRILL_MAX_DIRECT_MEMORY=10G
DRILLBIT_CODE_CACHE_SIZE=1024m
*NOTE: It is recommended not to specify DRILLBIT_CODE_CACHE_SIZE as this will be auto-computed based on the HeapSize and would not exceed 1GB
```
For all other combinations, the undefined parameters are adjusted to ensure that the total memory allocated is within the value specified by `DRILLBIT_MAX_PROC_MEM`,
For a system with with 48GB free memory, when set to (say) 50% (with settings defined in drill-env.sh), and heap (8GB) and direct (10GB) are defined; the Drillbit startup with the following warning:
```
2018-01-03 14:31:06 [WARN] 50% of System Memory (47 GB) translates to 24 GB
2018-01-03 14:31:06 [WARN] You have an allocation of 4 GB that is currently unused from a total of 24 GB. You can increase your existing memory configuration to use this extra memory
DRILLBIT_MAX_PROC_MEM=50%
DRILL_HEAP=8G
DRILL_MAX_DIRECT_MEMORY=10G
DRILLBIT_CODE_CACHE_SIZE=1024m
*NOTE: It is recommended not to specify DRILLBIT_CODE_CACHE_SIZE as this will be auto-computed based on the HeapSize and would not exceed 1GB
```
In addition, if the available free memory is less than the allocation, an additional warning is provided under the assumption that the OS will reclaim more free memory when required:
```
2018-01-03 14:31:06 [WARN] Total Memory Allocation for Drillbit (19GB) exceeds available free memory (11GB)
2018-01-03 14:31:06 [WARN] Drillbit will start up, but can potentially crash due to oversubscribing of system memory.
```
For more details, refer the attachments in https://issues.apache.org/jira/browse/DRILL-5741
----
> Automatically manage memory allocations during startup
> ------------------------------------------------------
>
> Key: DRILL-5741
> URL: https://issues.apache.org/jira/browse/DRILL-5741
> Project: Apache Drill
> Issue Type: Improvement
> Components: Server
> Affects Versions: 1.11.0
> Reporter: Kunal Khatua
> Assignee: Kunal Khatua
> Fix For: 1.13.0
>
> Attachments: Auto Mem Allocation Proposal - Computation Logic.pdf, Auto Mem Allocation Proposal - Scenarios.pdf
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> Currently, during startup, a Drillbit can be assigned large values for the following:
> * Xmx (Heap)
> * XX:MaxDirectMemorySize
> * XX:ReservedCodeCacheSize
> * XX:MaxPermSize
> All of this, potentially, can exceed the available memory on a system when a Drillbit is under heavy load. It would be good to have the Drillbit ensure during startup itself that the cumulative value of these parameters does not exceed a pre-defined upper limit for the Drill process.
> This JIRA is a *proposal* to allow for automatic configuration (based on configuration patterns observed in production Drill clusters). It leverages the capability of providing distribution (and user-specific) checks during Drill Startup from DRILL-6068.
> The idea is to remove the need for a user to worry about managing the tuning parameters, by providing the optimal values. In addition, it also allows for the memory allocation to be implicitly managed by simply providing the Drill process with a single dimensional of total process memory (either in absolute values, or as a percentage of the total system memory), while {{distrib-auto.sh}} provides the individual allocations.
> This allocation is then partitioned into allocations for Heap and Direct Memory, with a small portion allocated for the Generated Java CodeCache as well. If any of the individual allocations are also specified (via {{distrib-env.sh}} or {{drill-env.sh}}), the remaining unspecified allocations are adjusted to stay +within the limits+ of the total memory allocation.
> The *details* of the proposal are here:
> https://docs.google.com/spreadsheets/d/1N6VYlQFiPoTV4iD46XbkIrvEQesiGFUU9-GWXYsAPXs/edit#gid=0
> For those unable to access the Google Document, PDFs are attached:
> * [^Auto Mem Allocation Proposal - Computation Logic.pdf] - Provides the equation used for computing the heap, direct and code cache allocations for a given input
> * [^Auto Mem Allocation Proposal - Scenarios.pdf] - Describes the various inputs, and their expected allocations
> The variables that are (_optionally_) defined (in memory, {{distrib-env.sh}} or {{drill-env.sh}} ) are:
> * {{DRILLBIT_MAX_PROC_MEM}} : Total Process Memory
> * {{DRILL_HEAP}} : JVM Max Heap Size
> * {{DRILL_MAX_DIRECT_MEMORY}} : JVM Max Direct Memory Size
> * {{DRILLBIT_CODE_CACHE_SIZE}} : JVM Code Cache Size
> Note: _With JDK8, MaxPermSize is no longer supported, so we do not account for this any more, and will unset the variable if JDK8 or higher is detected._
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)