You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@drill.apache.org by ts...@apache.org on 2015/05/02 08:28:32 UTC
[06/17] drill git commit: fix links
fix links
Project: http://git-wip-us.apache.org/repos/asf/drill/repo
Commit: http://git-wip-us.apache.org/repos/asf/drill/commit/d56fcb40
Tree: http://git-wip-us.apache.org/repos/asf/drill/tree/d56fcb40
Diff: http://git-wip-us.apache.org/repos/asf/drill/diff/d56fcb40
Branch: refs/heads/gh-pages
Commit: d56fcb40fcba45afa9fbaa716c4ba5014aa4d9dd
Parents: bcc1bc0
Author: Kristine Hahn <kh...@maprtech.com>
Authored: Fri May 1 11:24:01 2015 -0700
Committer: Kristine Hahn <kh...@maprtech.com>
Committed: Fri May 1 11:24:01 2015 -0700
----------------------------------------------------------------------
_data/docs.json | 8 +-
...nfiguring-dfferent-workloads-introduction.md | 2 +-
...-configuring-resources-in-a-mixed-cluster.md | 80 ++++++++++++++++++++
3 files changed, 85 insertions(+), 5 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/drill/blob/d56fcb40/_data/docs.json
----------------------------------------------------------------------
diff --git a/_data/docs.json b/_data/docs.json
index 6805349..83fb2b4 100644
--- a/_data/docs.json
+++ b/_data/docs.json
@@ -881,7 +881,7 @@
"parent": "Configuring Different Workloads",
"previous_title": "Configuring Different Workloads Introduction",
"previous_url": "/docs/configuring-different-workloads-introduction/",
- "relative_path": "_docs/manage-drill/015-configuring-drill-in-a-cluster.md",
+ "relative_path": "_docs/manage-drill/015-configuring-resources-in-a-mixed-cluster.md",
"title": "Configuring Resources in a Mixed Cluster",
"url": "/docs/configuring-resources-in-a-mixed-cluster/"
},
@@ -992,7 +992,7 @@
"parent": "Configuring Different Workloads",
"previous_title": "Configuring Different Workloads Introduction",
"previous_url": "/docs/configuring-different-workloads-introduction/",
- "relative_path": "_docs/manage-drill/015-configuring-drill-in-a-cluster.md",
+ "relative_path": "_docs/manage-drill/015-configuring-resources-in-a-mixed-cluster.md",
"title": "Configuring Resources in a Mixed Cluster",
"url": "/docs/configuring-resources-in-a-mixed-cluster/"
},
@@ -3530,7 +3530,7 @@
"parent": "Configuring Different Workloads",
"previous_title": "Configuring Different Workloads Introduction",
"previous_url": "/docs/configuring-different-workloads-introduction/",
- "relative_path": "_docs/manage-drill/015-configuring-drill-in-a-cluster.md",
+ "relative_path": "_docs/manage-drill/015-configuring-resources-in-a-mixed-cluster.md",
"title": "Configuring Resources in a Mixed Cluster",
"url": "/docs/configuring-resources-in-a-mixed-cluster/"
},
@@ -10958,7 +10958,7 @@
"parent": "Configuring Different Workloads",
"previous_title": "Configuring Different Workloads Introduction",
"previous_url": "/docs/configuring-different-workloads-introduction/",
- "relative_path": "_docs/manage-drill/015-configuring-drill-in-a-cluster.md",
+ "relative_path": "_docs/manage-drill/015-configuring-resources-in-a-mixed-cluster.md",
"title": "Configuring Resources in a Mixed Cluster",
"url": "/docs/configuring-resources-in-a-mixed-cluster/"
},
http://git-wip-us.apache.org/repos/asf/drill/blob/d56fcb40/_docs/manage-drill/013-configuring-dfferent-workloads-introduction.md
----------------------------------------------------------------------
diff --git a/_docs/manage-drill/013-configuring-dfferent-workloads-introduction.md b/_docs/manage-drill/013-configuring-dfferent-workloads-introduction.md
index efbbb2e..4bd1664 100644
--- a/_docs/manage-drill/013-configuring-dfferent-workloads-introduction.md
+++ b/_docs/manage-drill/013-configuring-dfferent-workloads-introduction.md
@@ -16,7 +16,7 @@ Drill typically runs along side other workloads, including the following:
You need to plan and configure these resources for use with Drill and other workloads:
* [Memory]({{site.baseurl}}/docs/configuring-resources-in-a-mixed-cluster)
-* [CPU]({{site.baseurl}}/docs/configuring-drill-in-a-mixed-cluster#how-to-manage-drill-cpu-resources)
+* [CPU]({{site.baseurl}}/docs/configuring-resources-in-a-mixed-cluster#how-to-manage-drill-cpu-resources)
* Disk
Configure, memory, queues, and parallelization when users [share a Drillbit]({{site.baseurl}}/docs/configuring-resources-for-a-shared-drillbit).
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/d56fcb40/_docs/manage-drill/015-configuring-resources-in-a-mixed-cluster.md
----------------------------------------------------------------------
diff --git a/_docs/manage-drill/015-configuring-resources-in-a-mixed-cluster.md b/_docs/manage-drill/015-configuring-resources-in-a-mixed-cluster.md
new file mode 100644
index 0000000..474d123
--- /dev/null
+++ b/_docs/manage-drill/015-configuring-resources-in-a-mixed-cluster.md
@@ -0,0 +1,80 @@
+---
+title: "Configuring Resources in a Mixed Cluster"
+parent: "Configuring Different Workloads"
+---
+Drill operations are memory and CPU-intensive. You need to statically partition the cluster to designate which partition handles which workload. To configure resources for Drill in a MapR cluster, modify one or more of the following files in `/opt/mapr/conf/conf.d` that the installation process creates.
+
+* `warden.drill-bits.conf`
+* `warden.nodemanager.conf`
+* `warden.resourcemanager.conf`
+
+Configure Drill memory by modifying `warden.drill-bits.conf` in YARN and non-YARN clusters. Configure other resources by modifying `warden.nodemanager.conf `and `warden.resourcemanager.conf `in a YARN-enabled cluster.
+
+## Configuring Drill Memory in a Mixed Cluster
+
+Add the following lines to the `warden.drill-bits.conf` file to configure memory resources for Drill:
+
+ service.heapsize.min=<some value in MB>
+ service.heapsize.max=<some value in MB>
+ service.heapsize.percent=<a whole number>
+
+The service.heapsize.percent is the percentage of memory for the service bounded by minimum and maximum values.
+
+## Configuring Drill in a YARN-enabled MapR Cluster
+
+To add Drill to a YARN-enabled cluster, change memory resources to suit your application. For example, you have 120G of available memory that you allocate to following workloads in a Yarn-enabled cluster:
+
+File system = 20G
+HBase = 20G
+Yarn = 20G
+OS = 8G
+
+If Yarn does most of the work, give Drill 20G, for example, and give Yarn 60G. If you expect a heavy query load, give Drill 60G and Drill 20G.
+
+{% include startnote.html %}Drill will execute queries within Yarn soon.{% include endnote.html %} [DRILL-142](https://issues.apache.org/jira/browse/DRILL-142)
+
+YARN consists of two main services:
+
+* ResourceManager
+ There is at least one instance in a cluster, more if you configure high availability.
+* NodeManager
+ There is one instance per node.
+
+ResourceManager and NodeManager memory in `warden.resourcemanager.conf` and
+ `warden.nodemanager.conf` are set to the following defaults.
+
+ service.heapsize.min=64
+ service.heapsize.max=325
+ service.heapsize.percent=2
+
+Change these settings for NodeManager and ResourceManager to reconfigure the total memory required for YARN services to run. If you want to place an upper limit on memory set YARN_NODEMANAGER_HEAPSIZE or YARN_RESOURCEMANAGER_HEAPSIZE environment variable in /opt/mapr/hadoop/hadoop-2.5.1/etc/hadoop/yarn-env.sh. The -Xmx option is not set, allowing memory on to grow as needed.
+
+### MapReduce v1 Resources
+
+The following default settings in /opt/mapr/conf/warden.conf control MapReduce v1 memory:
+
+ mr1.memory.percent=50
+ mr1.cpu.percent=50
+ mr1.disk.percent=50
+
+Modify these settings to reconfigure MapReduce v1 resources to suit your application needs, as described in section ["Resource Allocation for Jobs and Applications"](http://doc.mapr.com/display/MapR/Resource+Allocation+for+Jobs+and+Applications) of the MapR documentation. Remaining memory is given to YARN applications.
+
+
+### MapReduce v2 and other Resources
+
+You configure memory for each service by setting three values in `warden.conf`.
+
+ service.command.<servicename>.heapsize.percent
+ service.command.<servicename>.heapsize.max
+ service.command.<servicename>.heapsize.min
+
+Configure memory for other services in the same manner, as described in [MapR documentation](http://doc.mapr.com/display/MapR/warden.%3Cservicename%3E.conf)
+
+For more information about managing memory in a MapR cluster, see the following sections in the MapR documentation:
+
+* [Memory Allocation for Nodes](http://doc.mapr.com/display/MapR40x/Memory+Allocation+for+Nodes)
+* [Cluster Resource Allocation](http://doc.mapr.com/display/MapR40x/Cluster+Resource+Allocation)
+* [Customizing Memory Settings for MapReduce v1](http://doc.mapr.com/display/MapR40x/Customize+Memory+Settings+for+MapReduce+v1)
+
+## How to Manage Drill CPU Resources
+Currently, you do not manage CPU resources within Drill. [Use Linux `cgroups`](http://en.wikipedia.org/wiki/Cgroups) to manage the CPU resources.
\ No newline at end of file