You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@drill.apache.org by br...@apache.org on 2015/05/07 02:37:04 UTC
drill git commit: NR's multitenant changes-partial
Repository: drill
Updated Branches:
refs/heads/gh-pages 27067e8c5 -> bca47014e
NR's multitenant changes-partial
Project: http://git-wip-us.apache.org/repos/asf/drill/repo
Commit: http://git-wip-us.apache.org/repos/asf/drill/commit/bca47014
Tree: http://git-wip-us.apache.org/repos/asf/drill/tree/bca47014
Diff: http://git-wip-us.apache.org/repos/asf/drill/diff/bca47014
Branch: refs/heads/gh-pages
Commit: bca47014e556ac41809c77aa1f715731318b73f9
Parents: 27067e8
Author: Kristine Hahn <kh...@maprtech.com>
Authored: Wed May 6 17:21:03 2015 -0700
Committer: Kristine Hahn <kh...@maprtech.com>
Committed: Wed May 6 17:21:03 2015 -0700
----------------------------------------------------------------------
_data/docs.json | 44 ++++++++++----------
...-configuring-drill-in-a-dedicated-cluster.md | 30 -------------
.../020-configuring-drill-memory.md | 37 ++++++++++++++++
.../050-configuring-multitenant-resources.md | 24 ++++++++++-
4 files changed, 82 insertions(+), 53 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/drill/blob/bca47014/_data/docs.json
----------------------------------------------------------------------
diff --git a/_data/docs.json b/_data/docs.json
index 2da89e5..4aca3af 100644
--- a/_data/docs.json
+++ b/_data/docs.json
@@ -821,8 +821,8 @@
}
],
"children": [],
- "next_title": "Configuring Drill in a Dedicated Cluster",
- "next_url": "/docs/configuring-drill-in-a-dedicated-cluster/",
+ "next_title": "Configuring Drill Memory",
+ "next_url": "/docs/configuring-drill-memory/",
"parent": "Configure Drill",
"previous_title": "Configure Drill",
"previous_url": "/docs/configure-drill/",
@@ -843,9 +843,9 @@
"parent": "Configure Drill",
"previous_title": "Configure Drill Introduction",
"previous_url": "/docs/configure-drill-introduction/",
- "relative_path": "_docs/configure-drill/020-configuring-drill-in-a-dedicated-cluster.md",
- "title": "Configuring Drill in a Dedicated Cluster",
- "url": "/docs/configuring-drill-in-a-dedicated-cluster/"
+ "relative_path": "_docs/configure-drill/020-configuring-drill-memory.md",
+ "title": "Configuring Drill Memory",
+ "url": "/docs/configuring-drill-memory/"
},
{
"breadcrumbs": [
@@ -922,8 +922,8 @@
"next_title": "Configuring a Multitenant Cluster Introduction",
"next_url": "/docs/configuring-a-multitenant-cluster-introduction/",
"parent": "Configure Drill",
- "previous_title": "Configuring Drill in a Dedicated Cluster",
- "previous_url": "/docs/configuring-drill-in-a-dedicated-cluster/",
+ "previous_title": "Configuring Drill Memory",
+ "previous_url": "/docs/configuring-drill-memory/",
"relative_path": "_docs/configure-drill/040-configuring-a-multitenant-cluster.md",
"title": "Configuring a Multitenant Cluster",
"url": "/docs/configuring-a-multitenant-cluster/"
@@ -1099,8 +1099,8 @@
}
],
"children": [],
- "next_title": "Configuring Drill in a Dedicated Cluster",
- "next_url": "/docs/configuring-drill-in-a-dedicated-cluster/",
+ "next_title": "Configuring Drill Memory",
+ "next_url": "/docs/configuring-drill-memory/",
"parent": "Configure Drill",
"previous_title": "Configure Drill",
"previous_url": "/docs/configure-drill/",
@@ -1150,7 +1150,7 @@
"title": "Configuring Connections on Windows",
"url": "/docs/configuring-connections-on-windows/"
},
- "Configuring Drill in a Dedicated Cluster": {
+ "Configuring Drill Memory": {
"breadcrumbs": [
{
"title": "Configure Drill",
@@ -1163,9 +1163,9 @@
"parent": "Configure Drill",
"previous_title": "Configure Drill Introduction",
"previous_url": "/docs/configure-drill-introduction/",
- "relative_path": "_docs/configure-drill/020-configuring-drill-in-a-dedicated-cluster.md",
- "title": "Configuring Drill in a Dedicated Cluster",
- "url": "/docs/configuring-drill-in-a-dedicated-cluster/"
+ "relative_path": "_docs/configure-drill/020-configuring-drill-memory.md",
+ "title": "Configuring Drill Memory",
+ "url": "/docs/configuring-drill-memory/"
},
"Configuring Multitenant Resources": {
"breadcrumbs": [
@@ -1301,8 +1301,8 @@
"next_title": "Configuring a Multitenant Cluster Introduction",
"next_url": "/docs/configuring-a-multitenant-cluster-introduction/",
"parent": "Configure Drill",
- "previous_title": "Configuring Drill in a Dedicated Cluster",
- "previous_url": "/docs/configuring-drill-in-a-dedicated-cluster/",
+ "previous_title": "Configuring Drill Memory",
+ "previous_url": "/docs/configuring-drill-memory/",
"relative_path": "_docs/configure-drill/040-configuring-a-multitenant-cluster.md",
"title": "Configuring a Multitenant Cluster",
"url": "/docs/configuring-a-multitenant-cluster/"
@@ -9273,8 +9273,8 @@
}
],
"children": [],
- "next_title": "Configuring Drill in a Dedicated Cluster",
- "next_url": "/docs/configuring-drill-in-a-dedicated-cluster/",
+ "next_title": "Configuring Drill Memory",
+ "next_url": "/docs/configuring-drill-memory/",
"parent": "Configure Drill",
"previous_title": "Configure Drill",
"previous_url": "/docs/configure-drill/",
@@ -9295,9 +9295,9 @@
"parent": "Configure Drill",
"previous_title": "Configure Drill Introduction",
"previous_url": "/docs/configure-drill-introduction/",
- "relative_path": "_docs/configure-drill/020-configuring-drill-in-a-dedicated-cluster.md",
- "title": "Configuring Drill in a Dedicated Cluster",
- "url": "/docs/configuring-drill-in-a-dedicated-cluster/"
+ "relative_path": "_docs/configure-drill/020-configuring-drill-memory.md",
+ "title": "Configuring Drill Memory",
+ "url": "/docs/configuring-drill-memory/"
},
{
"breadcrumbs": [
@@ -9374,8 +9374,8 @@
"next_title": "Configuring a Multitenant Cluster Introduction",
"next_url": "/docs/configuring-a-multitenant-cluster-introduction/",
"parent": "Configure Drill",
- "previous_title": "Configuring Drill in a Dedicated Cluster",
- "previous_url": "/docs/configuring-drill-in-a-dedicated-cluster/",
+ "previous_title": "Configuring Drill Memory",
+ "previous_url": "/docs/configuring-drill-memory/",
"relative_path": "_docs/configure-drill/040-configuring-a-multitenant-cluster.md",
"title": "Configuring a Multitenant Cluster",
"url": "/docs/configuring-a-multitenant-cluster/"
http://git-wip-us.apache.org/repos/asf/drill/blob/bca47014/_docs/configure-drill/020-configuring-drill-in-a-dedicated-cluster.md
----------------------------------------------------------------------
diff --git a/_docs/configure-drill/020-configuring-drill-in-a-dedicated-cluster.md b/_docs/configure-drill/020-configuring-drill-in-a-dedicated-cluster.md
deleted file mode 100644
index ec619a9..0000000
--- a/_docs/configure-drill/020-configuring-drill-in-a-dedicated-cluster.md
+++ /dev/null
@@ -1,30 +0,0 @@
----
-title: "Configuring Drill in a Dedicated Cluster"
-parent: "Configure Drill"
----
-
-This section describes how to configure the amount of direct memory allocated to a Drillbit for query processing in a dedicated Drill cluster. When you use Drill in a cluster with other workloads, configure memory as described in section, ["Configuring Drill in a Dedicated Cluster"]({{site.baseurl}}/docs/configuring-drill-in-a-dedicated-cluster).
-
-The default memory for a Drillbit is 8G, but Drill prefers 16G or more
-depending on the workload. The total amount of direct memory that a Drillbit
-allocates to query operations cannot exceed the limit set.
-
-Drill mainly uses Java direct memory and performs well when executing
-operations in memory instead of storing the operations on disk. Drill does not
-write to disk unless absolutely necessary, unlike MapReduce where everything
-is written to disk during each phase of a job.
-
-The JVM’s heap memory does not limit the amount of direct memory available in
-a Drillbit. The on-heap memory for Drill is only about 4-8G, which should
-suffice because Drill avoids having data sit in heap memory.
-
-## Modifying Drillbit Memory
-
-You can modify memory for each Drillbit node in your cluster. To modify the
-memory for a Drillbit, edit the `XX:MaxDirectMemorySize` parameter in the
-Drillbit startup script located in `<drill_installation_directory>/conf/drill-
-env.sh`.
-
-{% include startnote.html %}If this parameter is not set, the limit depends on the amount of available system memory.{% include endnote.html %}
-
-After you edit `<drill_installation_directory>/conf/drill-env.sh`, [restart the Drillbit]({{ site.baseurl }}/docs/starting-drill-in-distributed-mode) on the node.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/bca47014/_docs/configure-drill/020-configuring-drill-memory.md
----------------------------------------------------------------------
diff --git a/_docs/configure-drill/020-configuring-drill-memory.md b/_docs/configure-drill/020-configuring-drill-memory.md
new file mode 100644
index 0000000..150ce1f
--- /dev/null
+++ b/_docs/configure-drill/020-configuring-drill-memory.md
@@ -0,0 +1,37 @@
+---
+title: "Configuring Drill Memory"
+parent: "Configure Drill"
+---
+
+This section describes how to configure the amount of direct memory allocated to a Drillbit for query processing in any Drill cluster, multitenant or not. The default memory for a Drillbit is 8G, but Drill prefers 16G or more depending on the workload. The total amount of direct memory that a Drillbit allocates to query operations cannot exceed the limit set.
+
+Drill uses Java direct memory and performs well when executing
+operations in memory instead of storing the operations on disk. Drill does not
+write to disk unless absolutely necessary, unlike MapReduce where everything
+is written to disk during each phase of a job.
+
+The JVM’s heap memory does not limit the amount of direct memory available in
+a Drillbit. The on-heap memory for Drill is typically set at 4-8G (default is 4), which should
+suffice because Drill avoids having data sit in heap memory.
+
+## Modifying Drillbit Memory
+
+You can modify memory for each Drillbit node in your cluster. To modify the
+memory for a Drillbit, edit the `XX:MaxDirectMemorySize` parameter in the
+Drillbit startup script, `drill-env.sh`, located in `<drill_installation_directory>/conf`.
+
+{% include startnote.html %}If XX:MaxDirectMemorySize is not set, the limit depends on the amount of available system memory.{% include endnote.html %}
+
+After you edit `<drill_installation_directory>/conf/drill-env.sh`, [restart the Drillbit]({{ site.baseurl }}/docs/starting-drill-in-distributed-mode) on the node.
+
+## About the Drillbit startup script
+
+The drill-env.sh file contains the following options:
+
+ DRILL_MAX_DIRECT_MEMORY="8G"
+ DRILL_MAX_HEAP="4G"
+
+ export DRILL_JAVA_OPTS="-Xms1G -Xmx$DRILL_MAX_HEAP -XX:MaxDirectMemorySize=$DRILL_MAX_DIRECT_MEMORY -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=1G -ea"
+
+The DRILL_MAX_DIRECT_MEMORY is the Java direct memory. The DRILL_MAX_HEAP is the maximum theoretical heap limit for the JVM. Xmx specifies the maximum memory allocation pool for a Java Virtual Machine (JVM). Xms specifies the initial memory allocation pool.
+
http://git-wip-us.apache.org/repos/asf/drill/blob/bca47014/_docs/configure-drill/050-configuring-multitenant-resources.md
----------------------------------------------------------------------
diff --git a/_docs/configure-drill/050-configuring-multitenant-resources.md b/_docs/configure-drill/050-configuring-multitenant-resources.md
index 9a944e8..53341fb 100644
--- a/_docs/configure-drill/050-configuring-multitenant-resources.md
+++ b/_docs/configure-drill/050-configuring-multitenant-resources.md
@@ -2,7 +2,29 @@
title: "Configuring Multitenant Resources"
parent: "Configuring a Multitenant Cluster"
---
-Drill operations are memory and CPU-intensive. You need to statically partition the cluster to designate which partition handles which workload. To configure resources for Drill in a MapR cluster, modify one or more of the following files in `/opt/mapr/conf/conf.d` that the installation process creates.
+Drill operations are memory and CPU-intensive. Currently, Drill resources are managed outside MapR Warden service in terms of configuring the resources as well as enforcing the resource usage within the limit. Configure memory for Drill in a multitenant by modifying drill-env.sh. <Add detail on property names etc>
+
+3. To ensure Warden account for resources required for Drill, make sure the following properties are set appropriately in warden.drill-bits.conf. For reference on the meaning of the properties refer to the following MapR doc <add link>
+
+service.heapsize.min
+service.heapsize.max
+service.heapsize.percent
+
+The percent should always be considered as the one to change, it is more intuitive to understand and grows/shrinks according to different node's configuration.
+
+It will be good if someone in Drill QA could try it out and see if it fits Drill's needs.
+
+
+
+Note that these properties should be set in addition to configuring the resources the in drill-env.conf even if you didnt change the defaults in drill-env.sh. Setting up memory limit in drill-env.sh tells Drill how much resources to use during query execution and setting up these warden-drill-bits.conf tells warden not to commit these resources to some other process. In near future, we expect to provide a more deeper integration on these settings
+\<give an example>
+
+4. This configuration is same whether you use Drill is enabled cluster or non-YARN cluster.
+
+
+
+
+You need to statically partition the cluster to designate which partition handles which workload. To configure resources for Drill in a MapR cluster, modify one or more of the following files in `/opt/mapr/conf/conf.d` that the installation process creates.
* `warden.drill-bits.conf`
* `warden.nodemanager.conf`