You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@drill.apache.org by br...@apache.org on 2015/05/07 00:46:30 UTC
[06/11] drill git commit: reorg manage-drill
reorg manage-drill
Project: http://git-wip-us.apache.org/repos/asf/drill/repo
Commit: http://git-wip-us.apache.org/repos/asf/drill/commit/769eaac2
Tree: http://git-wip-us.apache.org/repos/asf/drill/tree/769eaac2
Diff: http://git-wip-us.apache.org/repos/asf/drill/diff/769eaac2
Branch: refs/heads/gh-pages
Commit: 769eaac270f46b784dc265b7bfb3247991e195f3
Parents: 26b0e52
Author: Kristine Hahn <kh...@maprtech.com>
Authored: Wed May 6 14:20:41 2015 -0700
Committer: Kristine Hahn <kh...@maprtech.com>
Committed: Wed May 6 14:20:41 2015 -0700
----------------------------------------------------------------------
_docs/030-configuring-user-imperso.textClipping | 0
...-configuring-drill-in-a-dedicated-cluster.md | 30 ----
.../012-configuring-user-impersonation.md | 150 -------------------
.../013-configuring-a-multitenant-cluster.md | 5 -
...guring-a-multitenant-cluster-introduction.md | 22 ---
.../015-configuring-multitenant-resources.md | 80 ----------
.../017-configuring-a-shared-drillbit.md | 65 --------
_docs/manage-drill/020-configuration-options.md | 9 --
...-configuring-drill-in-a-dedicated-cluster.md | 30 ++++
...guring-a-multitenant-cluster-introduction.md | 22 +++
_docs/manage-drill/030-start-stop.md | 42 ------
.../040-configuring-a-multitenant-cluster.md | 5 +
_docs/manage-drill/040-ports-used-by-drill.md | 15 --
.../050-configuring-multitenant-resources.md | 80 ++++++++++
_docs/manage-drill/050-partition-pruning.md | 75 ----------
.../060-configuring-a-shared-drillbit.md | 65 ++++++++
...and-canceling-queries-in-the-Drill-Web-UI.md | 30 ----
.../070-configuring-user-impersonation.md | 150 +++++++++++++++++++
_docs/manage-drill/080-configuration-options.md | 9 ++
_docs/manage-drill/090-start-stop.md | 42 ++++++
_docs/manage-drill/100-ports-used-by-drill.md | 15 ++
_docs/manage-drill/110-partition-pruning.md | 75 ++++++++++
...and-canceling-queries-in-the-Drill-Web-UI.md | 30 ++++
23 files changed, 523 insertions(+), 523 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/drill/blob/769eaac2/_docs/030-configuring-user-imperso.textClipping
----------------------------------------------------------------------
diff --git a/_docs/030-configuring-user-imperso.textClipping b/_docs/030-configuring-user-imperso.textClipping
new file mode 100644
index 0000000..e69de29
http://git-wip-us.apache.org/repos/asf/drill/blob/769eaac2/_docs/manage-drill/011-configuring-drill-in-a-dedicated-cluster.md
----------------------------------------------------------------------
diff --git a/_docs/manage-drill/011-configuring-drill-in-a-dedicated-cluster.md b/_docs/manage-drill/011-configuring-drill-in-a-dedicated-cluster.md
deleted file mode 100644
index 446f75c..0000000
--- a/_docs/manage-drill/011-configuring-drill-in-a-dedicated-cluster.md
+++ /dev/null
@@ -1,30 +0,0 @@
----
-title: "Configuring Drill in a Dedicated Cluster"
-parent: "Manage Drill"
----
-
-This section describes how to configure the amount of direct memory allocated to a Drillbit for query processing in a dedicated Drill cluster. When you use Drill in a cluster with other workloads, configure memory as described in section, ["Configuring Resources in a Mixed Cluster"]({{site.baseurl}}/docs/configuring-resources-in-a-mixed-cluster).
-
-The default memory for a Drillbit is 8G, but Drill prefers 16G or more
-depending on the workload. The total amount of direct memory that a Drillbit
-allocates to query operations cannot exceed the limit set.
-
-Drill mainly uses Java direct memory and performs well when executing
-operations in memory instead of storing the operations on disk. Drill does not
-write to disk unless absolutely necessary, unlike MapReduce where everything
-is written to disk during each phase of a job.
-
-The JVM’s heap memory does not limit the amount of direct memory available in
-a Drillbit. The on-heap memory for Drill is only about 4-8G, which should
-suffice because Drill avoids having data sit in heap memory.
-
-## Modifying Drillbit Memory
-
-You can modify memory for each Drillbit node in your cluster. To modify the
-memory for a Drillbit, edit the `XX:MaxDirectMemorySize` parameter in the
-Drillbit startup script located in `<drill_installation_directory>/conf/drill-
-env.sh`.
-
-{% include startnote.html %}If this parameter is not set, the limit depends on the amount of available system memory.{% include endnote.html %}
-
-After you edit `<drill_installation_directory>/conf/drill-env.sh`, [restart the Drillbit]({{ site.baseurl }}/docs/starting-drill-in-distributed-mode) on the node.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/769eaac2/_docs/manage-drill/012-configuring-user-impersonation.md
----------------------------------------------------------------------
diff --git a/_docs/manage-drill/012-configuring-user-impersonation.md b/_docs/manage-drill/012-configuring-user-impersonation.md
deleted file mode 100644
index 7f22d9d..0000000
--- a/_docs/manage-drill/012-configuring-user-impersonation.md
+++ /dev/null
@@ -1,150 +0,0 @@
----
-title: "Configuring User Impersonation"
-parent: "Manage Drill"
----
-Impersonation allows a service to act on behalf of a client while performing the action requested by the client. By default, user impersonation is disabled in Drill. You can configure user impersonation in the drill-override.conf file.
-
-When you enable impersonation, Drill executes client requests as the user logged in to the client. Drill passes the user credentials to the file system, and the file system checks to see if the user has permission to access the data. When you enable authentication, Drill uses the pluggable authentication module (PAM) to authenticate a user’s identity before the user can access the Drillbit process. See User Authentication.
-
-If impersonation is not configured, Drill executes all of the client requests against the file system as the user that started the Drillbit service on the node. This is typically a privileged user. The file system verifies that the system user has permission to access the data.
-
-
-## Example
-When impersonation is disabled and user Bob issues a query through the SQLLine client, SQLLine passes the query to the connecting Drillbit. The Drillbit executes the query as the system user that started the Drill process on the node. For the purpose of this example, we will assume that the system user has full access to the file system. Drill executes the query and returns the results back to the client.
-
-
-When impersonation is enabled and user Bob issues a query through the SQLLine client, the Drillbit executes the query against the file system as Bob. The file system checks to see if Bob has permission to access the data. If so, Drill returns the query results to the client. If Bob does not have permission, Drill returns an error.
-
-
-## Impersonation Support
-The following table lists the clients, storage plugins, and types of queries that you can use with impersonation in Drill:
-
-<table>
- <tr>
- <th>Type</th>
- <th>Supported</th>
- <th>Not Supported</th>
- </tr>
- <tr>
- <td>Clients</td>
- <td>SQLLine ODBC JDBC</td>
- <td>Drill Web UI REST API</td>
- </tr>
- <tr>
- <td>Storage Plugins</td>
- <td>File System</td>
- <td>Hive HBase</td>
- </tr>
- <tr>
- <td>Queries</td>
- <td>When you enable impersonation, the setting applies to queries on data and metadata. For example, if you issue the SHOW SCHEMAS command, Drill impersonates the user logged into the client to access the requested metadata. If you issue a SELECT query on a workspace, Drill impersonates the user logged in to the client to access the requested data. Drill applies impersonation to queries issued using the following commands: <br>SHOW SCHEMAS <br>SHOW DATABASES<br> SHOW TABLES<br> CTAS<br> SELECT<br> CREATE VIEW<br> DROP VIEW<br> SHOW FILES<br> To successfully run the CTAS and CREATE VIEW commands, a user must have write permissions on the directory where the table or view will exist. Running these commands creates artifacts on the file system.</td>
- <td></td>
- </tr>
-</table>
-
-## Impersonation and Views
-You can use views with impersonation to provide granular access to data and protect sensitive information. When you create a view, Drill stores the view definition in a file and suffixes the file with .drill.view. For example, if you create a view named myview, Drill creates a view file named myview.drill.view and saves it in the current workspace or the workspace specified, such as dfs.views.myview. See [CREATE VIEW]({{site.baseurl}}/docs/create-view-command/) Command.
-
-You can create a view and grant read permissions on the view to give other users access to the data that the view references. When a user queries the view, Drill impersonates the view owner to access the underlying data. A user with read access to a view can create new views from the originating view to further restrict access on data.
-
-### View Permissions
-A user must have write permission on a directory or workspace to create a view, as well as read access on the table(s) and/or view(s) that the view references. When a user creates a view, permission on the view is set to owner by default. Users can query an existing view or create new views from the view if they have read permissions on the view file and the directory or workspace where the view file is stored.
-
-When users query a view, Drill accesses the underlying data as the user that created the view. If a user does not have permission to access a view, the query fails and Drill returns an error. Only the view owner or a superuser can modify view permissions to change them from owner to group or world.
-
-The view owner or a superuser can modify permissions on the view file directly or they can set view permissions at the system or session level prior to creating any views. Any user that alters view permissions must have write access on the directory or workspace in which they are working. See Modifying Permissions on a View File and Modifying SYSTEM|SESSION Level View Permissions.
-
-#### Modifying Permissions on a View File
-Only a view owner or a super user can modify permissions on a view file to change them from owner to group or world readable. Before you grant permission to users to access a view, verify that they have access to the directory or workspace in which the view file is stored.
-
-Use the `chmod` and `chown` commands with the appropriate octal code to change permissions on a view file:
-
-
- hadoop fs –chmod <octal code> <file_name>
- hadoop fs –chown <user>:<group> <file_name>
-Example: `hadoop fs –chmod 750 employees.drill.view`
-
-####Modifying SYSTEM|SESSION Level View Permissions
-Use the `ALTER SESSION|SYSTEM` command with the `new_view_default_permissions` parameter and the appropriate octal code to set view permissions at the system or session level prior to creating a view.
-
- ALTER SESSION SET `new_view_default_permissions` = '<octal_code>';
- ALTER SYSTEM SET `new_view_default_permissions` = '<octal_code>';
-
-Example: ``ALTER SESSION SET `new_view_default_permissions` = '777';``
-
-After you set this parameter, Drill applies the same permissions on each view created during the session or across all sessions if set at the system level.
-
-## Chained Impersonation
-You can configure Drill to allow chained impersonation on views when you enable impersonation in the `drill-override.conf` file. Chained impersonation controls the number of identity transitions that Drill can make when a user queries a view. Each identity transition is equal to one hop.
-
-You can set the maximum number of hops on views to limit the number of times that Drill can impersonate a different user when a user queries a view. The default maximum number of hops is set at 3. When the maximum number of hops is set to 0, Drill does not allow impersonation chaining, and a user can only read data for which they have direct permission to access. You may set chain length to 0 to protect highly sensitive data.
-
-The following example depicts a scenario where the maximum hop number is set to 3, and Drill must impersonate three users to access data when Chad queries a view that Jane created:
-
-
-
-In the previous example, Joe created a view V3 from views that user Frank created. In the following example, Joe created view V3 by joining a view that Frank created with a view that Bob created, thus increasing the number of identity transitions that Drill makes from 3 to 4, which exceeds the maximum hop setting of 3.
-
-In this scenario, when Chad queries Jane’s view Drill returns an error stating that the query cannot complete because the number of hops required to access the data exceeds the maximum hop setting of 3 that is configured.
-
-
-
-If users encounter this error, you can increase the maximum hop setting to accommodate users running queries on views. When configuring the maximum number of hops that Drill can make, consider that joined views increase the number of identity transitions required for Drill to access the underlying data.
-
-#### Configuring Impersonation and Chaining
-Chaining is a system-wide setting that applies to all views. Currently, Drill does not provide an option to allow different chain lengths for different views.
-
-Complete the following steps on each Drillbit node to enable user impersonation, and set the maximum number of chained user hops that Drill allows:
-
-1. Navigate to `<drill_installation_directory>/conf/` and edit `drill-override.conf`.
-2. Under `drill.exe`, add the following:
-
- drill.exec.impersonation: {
- enabled: true,
- max_chained_user_hops: 3
- }
-
-3. Verify that enabled is set to `‘true’`.
-4. Set the maximum number of chained user hops that you want Drill to allow.
-5. (MapR cluster only) Add one of the following lines to the `drill-env.sh` file:
- * If the underlying file system is not secure, add the following line:
- ` export MAPR_IMPERSONATION_ENABLED=true`
- * If the underlying file system has MapR security enabled, add the following line:
- `export MAPR_TICKETFILE_LOCATION=/opt/mapr/conf/mapruserticket`
-6. Restart the Drillbit process on each Drill node.
- * In a MapR cluster, run the following command:
- `maprcli node services -name drill-bits -action restart -nodes <hostname> -f`
- * In a non-MapR environment, run the following command:
- <DRILLINSTALL_HOME>/bin/drillbit.sh restart
-
-
-## Impersonation and Chaining Example
-Frank is a senior HR manager at a company. Frank has access to all of the employee data because he is a member of the hr group. Frank created a table named “employees” in his home directory to store the employee data he uses. Only Frank has access to this table.
-
-drwx------ frank:hr /user/frank/employees
-
-Each record in the employees table consists of the following information:
-emp_id, emp_name, emp_ssn, emp_salary, emp_addr, emp_phone, emp_mgr
-
-Frank needs to share a subset of this information with Joe who is an HR manager reporting to Frank. To share the employee data, Frank creates a view called emp_mgr_view that accesses a subset of the data. The emp_mgr_view filters out sensitive employee information, such as the employee social security numbers, and only shows data for the employees that report directly to Joe or the manager running the query on the view. Frank and Joe both belong to the mgr group. Managers have read permission on Frank’s directory.
-
-rwxr----- frank:mgr /user/frank/emp_mgr_view.drill.view
-
-The emp_mgr_view.drill.view file contains the following view definition:
-(view definition: SELECT emp_id, emp_name, emp_salary, emp_addr, emp_phone FROM \`/user/frank/employee\` WHERE emp_mgr = user())
-
-When Joe issues SELECT * FROM emp_mgr_view, Drill impersonates Frank when accessing the employee data, and the query returns the data that Joe has permission to see based on the view definition. The query results do not include any sensitive data because the view protects that information. If Joe tries to query the employees table directly, Drill returns an error or null values.
-
-Because Joe has read permissions on the emp_mgr_view, he can create new views from it to give other users access to the employee data even though he does not own the employees table and cannot access the employees table directly.
-
-Joe needs to share employee contact data with his direct reports, so he creates a special view called emp_team_view to share the employee contact information with his team. Joe creates the view and writes it to his home directory. Joe and his reports belong to a group named joeteam. The joeteam group has read permissions on Joe’s home directory so they can query the view and create new views from it.
-
-rwxr----- joe:joeteam /user/joe/emp_team_view.drill.view
-
-The emp_team_view.drill.view file contains the following view definition:
-
-(view definition: SELECT emp_id, emp_name, emp_phone FROM `/user/frank/emp_mgr_view.drill`);
-
-When anyone on Joe’s team issues SELECT * FROM emp_team_view, Drill impersonates Joe to access the emp_team_view and then impersonates Frank to access the emp_mgr_view and the employee data. Drill returns the data that Joe’s team has can see based on the view definition. If anyone on Joe’s team tries to query the emp_mgr_view or employees table directly, Drill returns an error or null values.
-
-Because Joe’s team has read permissions on the emp_team_view, they can create new views from it and write the views to any directory for which they have write access. Creating views can continue until Drill reaches the maximum number of impersonation hops.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/769eaac2/_docs/manage-drill/013-configuring-a-multitenant-cluster.md
----------------------------------------------------------------------
diff --git a/_docs/manage-drill/013-configuring-a-multitenant-cluster.md b/_docs/manage-drill/013-configuring-a-multitenant-cluster.md
deleted file mode 100644
index fe72675..0000000
--- a/_docs/manage-drill/013-configuring-a-multitenant-cluster.md
+++ /dev/null
@@ -1,5 +0,0 @@
----
-title: "Configuring a Multitenant Cluster"
-parent: "Manage Drill"
----
-
http://git-wip-us.apache.org/repos/asf/drill/blob/769eaac2/_docs/manage-drill/014-configuring-a-multitenant-cluster-introduction.md
----------------------------------------------------------------------
diff --git a/_docs/manage-drill/014-configuring-a-multitenant-cluster-introduction.md b/_docs/manage-drill/014-configuring-a-multitenant-cluster-introduction.md
deleted file mode 100644
index 978d374..0000000
--- a/_docs/manage-drill/014-configuring-a-multitenant-cluster-introduction.md
+++ /dev/null
@@ -1,22 +0,0 @@
----
-title: "Configuring a Multitenant Cluster Introduction"
-parent: "Configuring a Multitenant Cluster"
----
-
-Drill supports multiple users sharing a Drillbit. You can also run separate Drillbits running on different nodes in the cluster.
-
-Drill typically runs along side other workloads, including the following:
-
-* Mapreduce
-* Yarn
-* HBase
-* Hive and Pig
-* Spark
-
-You need to plan and configure these resources for use with Drill and other workloads:
-
-* [Memory]({{site.baseurl}}/docs/configuring-multitenant-resources)
-* [CPU]({{site.baseurl}}/docs/configuring-multitenant-resources#how-to-manage-drill-cpu-resources)
-* Disk
-
-Configure, memory, queues, and parallelization when users [share a Drillbit]({{site.baseurl}}/docs/configuring-resources-for-a-shared-drillbit).
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/769eaac2/_docs/manage-drill/015-configuring-multitenant-resources.md
----------------------------------------------------------------------
diff --git a/_docs/manage-drill/015-configuring-multitenant-resources.md b/_docs/manage-drill/015-configuring-multitenant-resources.md
deleted file mode 100644
index 9a944e8..0000000
--- a/_docs/manage-drill/015-configuring-multitenant-resources.md
+++ /dev/null
@@ -1,80 +0,0 @@
----
-title: "Configuring Multitenant Resources"
-parent: "Configuring a Multitenant Cluster"
----
-Drill operations are memory and CPU-intensive. You need to statically partition the cluster to designate which partition handles which workload. To configure resources for Drill in a MapR cluster, modify one or more of the following files in `/opt/mapr/conf/conf.d` that the installation process creates.
-
-* `warden.drill-bits.conf`
-* `warden.nodemanager.conf`
-* `warden.resourcemanager.conf`
-
-Configure Drill memory by modifying `warden.drill-bits.conf` in YARN and non-YARN clusters. Configure other resources by modifying `warden.nodemanager.conf `and `warden.resourcemanager.conf `in a YARN-enabled cluster.
-
-## Configuring Drill Memory in a Mixed Cluster
-
-Add the following lines to the `warden.drill-bits.conf` file to configure memory resources for Drill:
-
- service.heapsize.min=<some value in MB>
- service.heapsize.max=<some value in MB>
- service.heapsize.percent=<a whole number>
-
-The service.heapsize.percent is the percentage of memory for the service bounded by minimum and maximum values.
-
-## Configuring Drill in a YARN-enabled MapR Cluster
-
-To add Drill to a YARN-enabled cluster, change memory resources to suit your application. For example, you have 120G of available memory that you allocate to following workloads in a Yarn-enabled cluster:
-
-File system = 20G
-HBase = 20G
-Yarn = 20G
-OS = 8G
-
-If Yarn does most of the work, give Drill 20G, for example, and give Yarn 60G. If you expect a heavy query load, give Drill 60G and Drill 20G.
-
-{% include startnote.html %}Drill will execute queries within Yarn soon.{% include endnote.html %} [DRILL-142](https://issues.apache.org/jira/browse/DRILL-142)
-
-YARN consists of two main services:
-
-* ResourceManager
- There is at least one instance in a cluster, more if you configure high availability.
-* NodeManager
- There is one instance per node.
-
-ResourceManager and NodeManager memory in `warden.resourcemanager.conf` and
- `warden.nodemanager.conf` are set to the following defaults.
-
- service.heapsize.min=64
- service.heapsize.max=325
- service.heapsize.percent=2
-
-Change these settings for NodeManager and ResourceManager to reconfigure the total memory required for YARN services to run. If you want to place an upper limit on memory set YARN_NODEMANAGER_HEAPSIZE or YARN_RESOURCEMANAGER_HEAPSIZE environment variable in /opt/mapr/hadoop/hadoop-2.5.1/etc/hadoop/yarn-env.sh. The -Xmx option is not set, allowing memory on to grow as needed.
-
-### MapReduce v1 Resources
-
-The following default settings in /opt/mapr/conf/warden.conf control MapReduce v1 memory:
-
- mr1.memory.percent=50
- mr1.cpu.percent=50
- mr1.disk.percent=50
-
-Modify these settings to reconfigure MapReduce v1 resources to suit your application needs, as described in section ["Resource Allocation for Jobs and Applications"](http://doc.mapr.com/display/MapR/Resource+Allocation+for+Jobs+and+Applications) of the MapR documentation. Remaining memory is given to YARN applications.
-
-
-### MapReduce v2 and other Resources
-
-You configure memory for each service by setting three values in `warden.conf`.
-
- service.command.<servicename>.heapsize.percent
- service.command.<servicename>.heapsize.max
- service.command.<servicename>.heapsize.min
-
-Configure memory for other services in the same manner, as described in [MapR documentation](http://doc.mapr.com/display/MapR/warden.%3Cservicename%3E.conf)
-
-For more information about managing memory in a MapR cluster, see the following sections in the MapR documentation:
-
-* [Memory Allocation for Nodes](http://doc.mapr.com/display/MapR40x/Memory+Allocation+for+Nodes)
-* [Cluster Resource Allocation](http://doc.mapr.com/display/MapR40x/Cluster+Resource+Allocation)
-* [Customizing Memory Settings for MapReduce v1](http://doc.mapr.com/display/MapR40x/Customize+Memory+Settings+for+MapReduce+v1)
-
-## How to Manage Drill CPU Resources
-Currently, you do not manage CPU resources within Drill. [Use Linux `cgroups`](http://en.wikipedia.org/wiki/Cgroups) to manage the CPU resources.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/769eaac2/_docs/manage-drill/017-configuring-a-shared-drillbit.md
----------------------------------------------------------------------
diff --git a/_docs/manage-drill/017-configuring-a-shared-drillbit.md b/_docs/manage-drill/017-configuring-a-shared-drillbit.md
deleted file mode 100644
index 3f83736..0000000
--- a/_docs/manage-drill/017-configuring-a-shared-drillbit.md
+++ /dev/null
@@ -1,65 +0,0 @@
----
-title: "Configuring Resources for a Shared Drillbit"
-parent: "Configuring a Multitenant Cluster"
----
-To manage a cluster in which multiple users share a Drillbit, you configure Drill queuing and parallelization in addition to memory, as described in the previous section.
-
-##Configuring Drill Query Queuing
-
-Set [options in sys.options]({{site.baseurl}}/docs/configuration-options-introduction/) to enable and manage query queuing, which is turned off by default. There are two types of queues: large and small. You configure a maximum number of queries that each queue allows by configuring the following options in the `sys.options` table:
-
-* exec.queue.large
-* exec.queue.small
-
-### Example Configuration
-
-For example, you configure the queue reserved for large queries to hold a 5-query maximum. You configure the queue reserved for small queries to hold 20 queries. Users start to run queries, and Drill receives the following query requests in this order:
-
-* Query A (blue): 1 billion records, Drill estimates 10 million rows will be processed
-* Query B (red): 2 billion records, Drill estimates 20 million rows will be processed
-* Query C: 1 billion records
-* Query D: 100 records
-
-The exec.queue.threshold default is 30 million, which is the estimated rows to be processed by the query. Queries A and B are queued in the large queue. The estimated rows to be processed reaches the 30 million threshold, filling the queue to capacity. The query C request arrives and goes on the wait list, and then query D arrives. Query D is queued immediately in the small queue because of its small size, as shown in the following diagram:
-
-
-
-The Drill queuing configuration in this example tends to give many users running small queries a rapid response. Users running a large query might experience some delay until an earlier-received large query returns, freeing space in the large queue to process queries that are waiting.
-
-## Controlling Parallelization
-
-By default, Drill parallelizes operations when number of records manipulated within a fragment reaches 100,000. When parallelization of operations is high, the cluster operates as fast as possible, which is fine for a single user. In a contentious multi-tenant situation, however, you need to reduce parallelization to levels based on user needs.
-
-### Parallelization Configuration Procedure
-
-To configure parallelization, configure the following options in the `sys.options` table:
-
-* `planner.width.max.per.node`
- The maximum degree of distribution of a query across cores and cluster nodes.
-* `planner.width.max.per.query`
- Same as max per node but applies to the query as executed by the entire cluster.
-
-Configure the `planner.width.max.per.node` to achieve fine grained, absolute control over parallelization.
-
-<!-- ??For example, setting the `planner.width.max.per.query` to 60 will not accelerate Drill operations because overlapping does not occur when executing 60 queries at the same time.??
-
-### Example of Configuring Parallelization
-
-For example, the default settings parallelize 70 percent of operations up to 1,000 cores. If you have 30 cores per node in a 10-node cluster, or 300 cores, parallelization occurs on approximately 210 cores. Consequently, a single user can get 70 percent usage from a cluster and no more due to the constraints configured by the `planner.width.max.per.query`.
-
-A parallelizer in the Foreman transforms the physical plan into multiple phases. A complicated query can have multiple, major fragments. A default parallelization of 70 percent of operations allows some overlap of query phases. In the example, 210 ??for each core or major fragment to a maximum of 410??.
-
-??Drill uses pipelines, blocking/nonblocking, memory is not fungible. CPU resources are fungible. There is contention for CPUs.?? -->
-
-## Data Isolation
-
-Tenants can share data on a cluster using Drill views and impersonation. ??Link to impersonation doc.??
-
-
-
-
-
-
-
-
-
http://git-wip-us.apache.org/repos/asf/drill/blob/769eaac2/_docs/manage-drill/020-configuration-options.md
----------------------------------------------------------------------
diff --git a/_docs/manage-drill/020-configuration-options.md b/_docs/manage-drill/020-configuration-options.md
deleted file mode 100644
index 41275eb..0000000
--- a/_docs/manage-drill/020-configuration-options.md
+++ /dev/null
@@ -1,9 +0,0 @@
----
-title: "Configuration Options"
-parent: "Manage Drill"
----
-
-
-
-
-
http://git-wip-us.apache.org/repos/asf/drill/blob/769eaac2/_docs/manage-drill/020-configuring-drill-in-a-dedicated-cluster.md
----------------------------------------------------------------------
diff --git a/_docs/manage-drill/020-configuring-drill-in-a-dedicated-cluster.md b/_docs/manage-drill/020-configuring-drill-in-a-dedicated-cluster.md
new file mode 100644
index 0000000..446f75c
--- /dev/null
+++ b/_docs/manage-drill/020-configuring-drill-in-a-dedicated-cluster.md
@@ -0,0 +1,30 @@
+---
+title: "Configuring Drill in a Dedicated Cluster"
+parent: "Manage Drill"
+---
+
+This section describes how to configure the amount of direct memory allocated to a Drillbit for query processing in a dedicated Drill cluster. When you use Drill in a cluster with other workloads, configure memory as described in section, ["Configuring Resources in a Mixed Cluster"]({{site.baseurl}}/docs/configuring-resources-in-a-mixed-cluster).
+
+The default memory for a Drillbit is 8G, but Drill prefers 16G or more
+depending on the workload. The total amount of direct memory that a Drillbit
+allocates to query operations cannot exceed the limit set.
+
+Drill mainly uses Java direct memory and performs well when executing
+operations in memory instead of storing the operations on disk. Drill does not
+write to disk unless absolutely necessary, unlike MapReduce where everything
+is written to disk during each phase of a job.
+
+The JVM’s heap memory does not limit the amount of direct memory available in
+a Drillbit. The on-heap memory for Drill is only about 4-8G, which should
+suffice because Drill avoids having data sit in heap memory.
+
+## Modifying Drillbit Memory
+
+You can modify memory for each Drillbit node in your cluster. To modify the
+memory for a Drillbit, edit the `XX:MaxDirectMemorySize` parameter in the
+Drillbit startup script located in `<drill_installation_directory>/conf/drill-
+env.sh`.
+
+{% include startnote.html %}If this parameter is not set, the limit depends on the amount of available system memory.{% include endnote.html %}
+
+After you edit `<drill_installation_directory>/conf/drill-env.sh`, [restart the Drillbit]({{ site.baseurl }}/docs/starting-drill-in-distributed-mode) on the node.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/769eaac2/_docs/manage-drill/030-configuring-a-multitenant-cluster-introduction.md
----------------------------------------------------------------------
diff --git a/_docs/manage-drill/030-configuring-a-multitenant-cluster-introduction.md b/_docs/manage-drill/030-configuring-a-multitenant-cluster-introduction.md
new file mode 100644
index 0000000..978d374
--- /dev/null
+++ b/_docs/manage-drill/030-configuring-a-multitenant-cluster-introduction.md
@@ -0,0 +1,22 @@
+---
+title: "Configuring a Multitenant Cluster Introduction"
+parent: "Configuring a Multitenant Cluster"
+---
+
+Drill supports multiple users sharing a Drillbit. You can also run separate Drillbits running on different nodes in the cluster.
+
+Drill typically runs along side other workloads, including the following:
+
+* Mapreduce
+* Yarn
+* HBase
+* Hive and Pig
+* Spark
+
+You need to plan and configure these resources for use with Drill and other workloads:
+
+* [Memory]({{site.baseurl}}/docs/configuring-multitenant-resources)
+* [CPU]({{site.baseurl}}/docs/configuring-multitenant-resources#how-to-manage-drill-cpu-resources)
+* Disk
+
+Configure, memory, queues, and parallelization when users [share a Drillbit]({{site.baseurl}}/docs/configuring-resources-for-a-shared-drillbit).
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/769eaac2/_docs/manage-drill/030-start-stop.md
----------------------------------------------------------------------
diff --git a/_docs/manage-drill/030-start-stop.md b/_docs/manage-drill/030-start-stop.md
deleted file mode 100644
index 591b6ab..0000000
--- a/_docs/manage-drill/030-start-stop.md
+++ /dev/null
@@ -1,42 +0,0 @@
----
-title: "Starting/Stopping Drill"
-parent: "Manage Drill"
----
-How you start Drill depends on the installation method you followed. If you installed Drill in embedded mode, invoking SQLLine automatically starts a Drillbit locally. If you installed Drill in distributed mode, and the Drillbit on a node did not start, start the Drillbit before attempting to run queries. How to start Drill is covered in detail the section, ["Install Drill"]({{site.baseurl}}/docs/install-drill/).
-
-## Examples of Starting Drill
-Issue the **sqlline** command from the Drill installation directory. The simplest example of how to start SQLLine is to identify the protocol, JDBC, and zookeeper node or nodes in the **sqlline** command. This example starts SQLLine on a node in an embedded, single-node cluster:
-
- sqlline -u jdbc:drill:zk=local
-
-This example also starts SQLLine using the `dfs` storage plugin. Specifying the storage plugin when you start up eliminates the need to specify the storage plugin in the query:
-
-
- bin/sqlline –u jdbc:drill:schema=dfs;zk=centos26
-
-This command starts SQLLine in distributed, (multi-node) mode in a cluster configured to run zookeeper on three nodes:
-
- bin/sqlline –u jdbc:drill:zk=cento23,zk=centos24,zk=centos26:5181
-
-## Exiting SQLLine
-
-To exit SQLLine, issue the following command:
-
- !quit
-
-## Stopping Drill
-
-In some cases, such as stopping while a query is in progress, the `!quit` command does not stop Drill running in embedded mode. In distributed mode, you stop the Drillbit service instead of killing the Drillbit process.
-
-To stop the Drill process on Mac OS X and Linux, use the kill command. On Windows, use the **TaskKill** command.
-
-For example, on Mac OS X and Linux, follow these steps:
-
- 1. Issue a CTRL Z to stop the query, then start Drill again. If the startup message indicates success, skip the rest of the steps. If not, proceed to step 2.
- 2. Search for the Drill process IDs.
-
- $ ps auwx | grep drill
- 3. Kill each process using the process numbers in the grep output. For example:
-
- $ sudo kill -9 2674
-
http://git-wip-us.apache.org/repos/asf/drill/blob/769eaac2/_docs/manage-drill/040-configuring-a-multitenant-cluster.md
----------------------------------------------------------------------
diff --git a/_docs/manage-drill/040-configuring-a-multitenant-cluster.md b/_docs/manage-drill/040-configuring-a-multitenant-cluster.md
new file mode 100644
index 0000000..fe72675
--- /dev/null
+++ b/_docs/manage-drill/040-configuring-a-multitenant-cluster.md
@@ -0,0 +1,5 @@
+---
+title: "Configuring a Multitenant Cluster"
+parent: "Manage Drill"
+---
+
http://git-wip-us.apache.org/repos/asf/drill/blob/769eaac2/_docs/manage-drill/040-ports-used-by-drill.md
----------------------------------------------------------------------
diff --git a/_docs/manage-drill/040-ports-used-by-drill.md b/_docs/manage-drill/040-ports-used-by-drill.md
deleted file mode 100644
index 42ecd20..0000000
--- a/_docs/manage-drill/040-ports-used-by-drill.md
+++ /dev/null
@@ -1,15 +0,0 @@
----
-title: "Ports Used by Drill"
-parent: "Manage Drill"
----
-The following table provides a list of the ports that Drill uses, the port
-type, and a description of how Drill uses the port:
-
-| Port | Type | Description |
-|-------|------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| 8047 | TCP | Needed for the Drill Web UI. |
-| 31010 | TCP | User port address. Used between nodes in a Drill cluster. Needed for an external client, such as Tableau, to connect into thecluster nodes. Also needed for the Drill Web UI. |
-| 31011 | TCP | Control port address. Used between nodes in a Drill cluster. Needed for multi-node installation of Apache Drill. |
-| 31012 | TCP | Data port address. Used between nodes in a Drill cluster. Needed for multi-node installation of Apache Drill. |
-| 46655 | UDP | Used for JGroups and Infinispan. Needed for multi-node installation of Apache Drill. |
-
http://git-wip-us.apache.org/repos/asf/drill/blob/769eaac2/_docs/manage-drill/050-configuring-multitenant-resources.md
----------------------------------------------------------------------
diff --git a/_docs/manage-drill/050-configuring-multitenant-resources.md b/_docs/manage-drill/050-configuring-multitenant-resources.md
new file mode 100644
index 0000000..9a944e8
--- /dev/null
+++ b/_docs/manage-drill/050-configuring-multitenant-resources.md
@@ -0,0 +1,80 @@
+---
+title: "Configuring Multitenant Resources"
+parent: "Configuring a Multitenant Cluster"
+---
+Drill operations are memory and CPU-intensive. You need to statically partition the cluster to designate which partition handles which workload. To configure resources for Drill in a MapR cluster, modify one or more of the following files in `/opt/mapr/conf/conf.d` that the installation process creates.
+
+* `warden.drill-bits.conf`
+* `warden.nodemanager.conf`
+* `warden.resourcemanager.conf`
+
+Configure Drill memory by modifying `warden.drill-bits.conf` in YARN and non-YARN clusters. Configure other resources by modifying `warden.nodemanager.conf `and `warden.resourcemanager.conf `in a YARN-enabled cluster.
+
+## Configuring Drill Memory in a Mixed Cluster
+
+Add the following lines to the `warden.drill-bits.conf` file to configure memory resources for Drill:
+
+ service.heapsize.min=<some value in MB>
+ service.heapsize.max=<some value in MB>
+ service.heapsize.percent=<a whole number>
+
+The service.heapsize.percent is the percentage of memory for the service bounded by minimum and maximum values.
+
+## Configuring Drill in a YARN-enabled MapR Cluster
+
+To add Drill to a YARN-enabled cluster, change memory resources to suit your application. For example, you have 120G of available memory that you allocate to following workloads in a Yarn-enabled cluster:
+
+File system = 20G
+HBase = 20G
+Yarn = 20G
+OS = 8G
+
+If Yarn does most of the work, give Drill 20G, for example, and give Yarn 60G. If you expect a heavy query load, give Drill 60G and Drill 20G.
+
+{% include startnote.html %}Drill will execute queries within Yarn soon.{% include endnote.html %} [DRILL-142](https://issues.apache.org/jira/browse/DRILL-142)
+
+YARN consists of two main services:
+
+* ResourceManager
+ There is at least one instance in a cluster, more if you configure high availability.
+* NodeManager
+ There is one instance per node.
+
+ResourceManager and NodeManager memory in `warden.resourcemanager.conf` and
+ `warden.nodemanager.conf` are set to the following defaults.
+
+ service.heapsize.min=64
+ service.heapsize.max=325
+ service.heapsize.percent=2
+
+Change these settings for NodeManager and ResourceManager to reconfigure the total memory required for YARN services to run. If you want to place an upper limit on memory set YARN_NODEMANAGER_HEAPSIZE or YARN_RESOURCEMANAGER_HEAPSIZE environment variable in /opt/mapr/hadoop/hadoop-2.5.1/etc/hadoop/yarn-env.sh. The -Xmx option is not set, allowing memory on to grow as needed.
+
+### MapReduce v1 Resources
+
+The following default settings in /opt/mapr/conf/warden.conf control MapReduce v1 memory:
+
+ mr1.memory.percent=50
+ mr1.cpu.percent=50
+ mr1.disk.percent=50
+
+Modify these settings to reconfigure MapReduce v1 resources to suit your application needs, as described in section ["Resource Allocation for Jobs and Applications"](http://doc.mapr.com/display/MapR/Resource+Allocation+for+Jobs+and+Applications) of the MapR documentation. Remaining memory is given to YARN applications.
+
+
+### MapReduce v2 and other Resources
+
+You configure memory for each service by setting three values in `warden.conf`.
+
+ service.command.<servicename>.heapsize.percent
+ service.command.<servicename>.heapsize.max
+ service.command.<servicename>.heapsize.min
+
+Configure memory for other services in the same manner, as described in [MapR documentation](http://doc.mapr.com/display/MapR/warden.%3Cservicename%3E.conf)
+
+For more information about managing memory in a MapR cluster, see the following sections in the MapR documentation:
+
+* [Memory Allocation for Nodes](http://doc.mapr.com/display/MapR40x/Memory+Allocation+for+Nodes)
+* [Cluster Resource Allocation](http://doc.mapr.com/display/MapR40x/Cluster+Resource+Allocation)
+* [Customizing Memory Settings for MapReduce v1](http://doc.mapr.com/display/MapR40x/Customize+Memory+Settings+for+MapReduce+v1)
+
+## How to Manage Drill CPU Resources
+Currently, you do not manage CPU resources within Drill. [Use Linux `cgroups`](http://en.wikipedia.org/wiki/Cgroups) to manage the CPU resources.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/769eaac2/_docs/manage-drill/050-partition-pruning.md
----------------------------------------------------------------------
diff --git a/_docs/manage-drill/050-partition-pruning.md b/_docs/manage-drill/050-partition-pruning.md
deleted file mode 100644
index 75f2edd..0000000
--- a/_docs/manage-drill/050-partition-pruning.md
+++ /dev/null
@@ -1,75 +0,0 @@
----
-title: "Partition Pruning"
-parent: "Manage Drill"
----
-Partition pruning is a performance optimization that limits the number of
-files and partitions that Drill reads when querying file systems and Hive
-tables. Drill only reads a subset of the files that reside in a file system or
-a subset of the partitions in a Hive table when a query matches certain filter
-criteria.
-
-For Drill to apply partition pruning to Hive tables, you must have created the
-tables in Hive using the `PARTITION BY` clause:
-
-`CREATE TABLE <table_name> (<column_name>) PARTITION BY (<column_name>);`
-
-When you create Hive tables using the `PARTITION BY` clause, each partition of
-data is automatically split out into different directories as data is written
-to disk. For more information about Hive partitioning, refer to the [Apache
-Hive wiki](https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL/#LanguageManualDDL-PartitionedTables).
-
-Typically, table data in a file system is organized by directories and
-subdirectories. Queries on table data may contain `WHERE` clause filters on
-specific directories.
-
-Drill’s query planner evaluates the filters as part of a Filter operator. If
-no partition filters are present, the underlying Scan operator reads all files
-in all directories and then sends the data to operators downstream, such as
-Filter.
-
-When partition filters are present, the query planner determines if it can
-push the filters down to the Scan such that the Scan only reads the
-directories that match the partition filters, thus reducing disk I/O.
-
-## Partition Pruning Example
-
-The /`Users/max/data/logs` directory in a file system contains subdirectories
-that span a few years.
-
-The following image shows the hierarchical structure of the `…/logs` directory
-and (sub) directories:
-
-
-
-The following query requests log file data for 2013 from the `…/logs`
-directory in the file system:
-
- SELECT * FROM dfs.`/Users/max/data/logs` WHERE cust_id < 10 and dir0 = 2013 limit 2;
-
-If you run the `EXPLAIN PLAN` command for the query, you can see that the`
-…/logs` directory is filtered by the scan operator.
-
- EXPLAIN PLAN FOR SELECT * FROM dfs.`/Users/max/data/logs` WHERE cust_id < 10 and dir0 = 2013 limit 2;
-
-The following image shows a portion of the physical plan when partition
-pruning is applied:
-
-
-
-## Filter Examples
-
-The following queries include examples of the types of filters eligible for
-partition pruning optimization:
-
-**Example 1: Partition filters ANDed together**
-
- SELECT * FROM dfs.`/Users/max/data/logs` WHERE dir0 = '2014' AND dir1 = '1'
-
-**Example 2: Partition filter ANDed with regular column filter**
-
- SELECT * FROM dfs.`/Users/max/data/logs` WHERE cust_id < 10 AND dir0 = 2013 limit 2;
-
-**Example 3: Combination of AND, OR involving partition filters**
-
- SELECT * FROM dfs.`/Users/max/data/logs` WHERE (dir0 = '2013' AND dir1 = '1') OR (dir0 = '2014' AND dir1 = '2')
-
http://git-wip-us.apache.org/repos/asf/drill/blob/769eaac2/_docs/manage-drill/060-configuring-a-shared-drillbit.md
----------------------------------------------------------------------
diff --git a/_docs/manage-drill/060-configuring-a-shared-drillbit.md b/_docs/manage-drill/060-configuring-a-shared-drillbit.md
new file mode 100644
index 0000000..3f83736
--- /dev/null
+++ b/_docs/manage-drill/060-configuring-a-shared-drillbit.md
@@ -0,0 +1,65 @@
+---
+title: "Configuring Resources for a Shared Drillbit"
+parent: "Configuring a Multitenant Cluster"
+---
+To manage a cluster in which multiple users share a Drillbit, you configure Drill queuing and parallelization in addition to memory, as described in the previous section.
+
+##Configuring Drill Query Queuing
+
+Set [options in sys.options]({{site.baseurl}}/docs/configuration-options-introduction/) to enable and manage query queuing, which is turned off by default. There are two types of queues: large and small. You configure a maximum number of queries that each queue allows by configuring the following options in the `sys.options` table:
+
+* exec.queue.large
+* exec.queue.small
+
+### Example Configuration
+
+For example, you configure the queue reserved for large queries to hold a 5-query maximum. You configure the queue reserved for small queries to hold 20 queries. Users start to run queries, and Drill receives the following query requests in this order:
+
+* Query A (blue): 1 billion records, Drill estimates 10 million rows will be processed
+* Query B (red): 2 billion records, Drill estimates 20 million rows will be processed
+* Query C: 1 billion records
+* Query D: 100 records
+
+The exec.queue.threshold default is 30 million, which is the estimated rows to be processed by the query. Queries A and B are queued in the large queue. The estimated rows to be processed reaches the 30 million threshold, filling the queue to capacity. The query C request arrives and goes on the wait list, and then query D arrives. Query D is queued immediately in the small queue because of its small size, as shown in the following diagram:
+
+
+
+The Drill queuing configuration in this example tends to give many users running small queries a rapid response. Users running a large query might experience some delay until an earlier-received large query returns, freeing space in the large queue to process queries that are waiting.
+
+## Controlling Parallelization
+
+By default, Drill parallelizes operations when number of records manipulated within a fragment reaches 100,000. When parallelization of operations is high, the cluster operates as fast as possible, which is fine for a single user. In a contentious multi-tenant situation, however, you need to reduce parallelization to levels based on user needs.
+
+### Parallelization Configuration Procedure
+
+To configure parallelization, configure the following options in the `sys.options` table:
+
+* `planner.width.max.per.node`
+ The maximum degree of distribution of a query across cores and cluster nodes.
+* `planner.width.max.per.query`
+ Same as max per node but applies to the query as executed by the entire cluster.
+
+Configure the `planner.width.max.per.node` to achieve fine grained, absolute control over parallelization.
+
+<!-- ??For example, setting the `planner.width.max.per.query` to 60 will not accelerate Drill operations because overlapping does not occur when executing 60 queries at the same time.??
+
+### Example of Configuring Parallelization
+
+For example, the default settings parallelize 70 percent of operations up to 1,000 cores. If you have 30 cores per node in a 10-node cluster, or 300 cores, parallelization occurs on approximately 210 cores. Consequently, a single user can get 70 percent usage from a cluster and no more due to the constraints configured by the `planner.width.max.per.query`.
+
+A parallelizer in the Foreman transforms the physical plan into multiple phases. A complicated query can have multiple, major fragments. A default parallelization of 70 percent of operations allows some overlap of query phases. In the example, 210 ??for each core or major fragment to a maximum of 410??.
+
+??Drill uses pipelines, blocking/nonblocking, memory is not fungible. CPU resources are fungible. There is contention for CPUs.?? -->
+
+## Data Isolation
+
+Tenants can share data on a cluster using Drill views and impersonation. ??Link to impersonation doc.??
+
+
+
+
+
+
+
+
+
http://git-wip-us.apache.org/repos/asf/drill/blob/769eaac2/_docs/manage-drill/060-monitoring-and-canceling-queries-in-the-Drill-Web-UI.md
----------------------------------------------------------------------
diff --git a/_docs/manage-drill/060-monitoring-and-canceling-queries-in-the-Drill-Web-UI.md b/_docs/manage-drill/060-monitoring-and-canceling-queries-in-the-Drill-Web-UI.md
deleted file mode 100644
index 0033838..0000000
--- a/_docs/manage-drill/060-monitoring-and-canceling-queries-in-the-Drill-Web-UI.md
+++ /dev/null
@@ -1,30 +0,0 @@
----
-title: "Monitoring and Canceling Queries in the Drill Web UI"
-parent: "Manage Drill"
----
-You can monitor and cancel queries from the Drill Web UI. To access the Drill
-Web UI, the Drillbit process must be running on the Drill node that you use to
-access the Drill Web UI.
-
-To monitor or cancel a query from the Drill Web UI, complete the following
-steps:
-
- 1. Navigate to the Drill Web UI at `<drill_node_ip_address>:8047.`
-When you access the Drill Web UI, you see some general information about Drill
-running in your cluster, such as the nodes running the Drillbit process, the
-various ports Drill is using, and the amount of direct memory assigned to
-Drill.
-
-
- 2. Select **Profiles** in the toolbar. A list of running and completed queries appears. Drill assigns a query ID to each query and lists the Foreman node. The Foreman is the Drillbit node that receives the query from the client or application. The Foreman drives the entire query.
-
-
- 3. Click the **Query ID** for the query that you want to monitor or cancel. The Query and Planning window appears.
-
-
- 4. Select **Edit Query**.
- 5. Click **Cancel query **to cancel the** query. The following message appears:
- 
-
- 6. Optionally, you can re-run the query to see a query summary in this window.
-
http://git-wip-us.apache.org/repos/asf/drill/blob/769eaac2/_docs/manage-drill/070-configuring-user-impersonation.md
----------------------------------------------------------------------
diff --git a/_docs/manage-drill/070-configuring-user-impersonation.md b/_docs/manage-drill/070-configuring-user-impersonation.md
new file mode 100644
index 0000000..7f22d9d
--- /dev/null
+++ b/_docs/manage-drill/070-configuring-user-impersonation.md
@@ -0,0 +1,150 @@
+---
+title: "Configuring User Impersonation"
+parent: "Manage Drill"
+---
+Impersonation allows a service to act on behalf of a client while performing the action requested by the client. By default, user impersonation is disabled in Drill. You can configure user impersonation in the drill-override.conf file.
+
+When you enable impersonation, Drill executes client requests as the user logged in to the client. Drill passes the user credentials to the file system, and the file system checks to see if the user has permission to access the data. When you enable authentication, Drill uses the pluggable authentication module (PAM) to authenticate a user’s identity before the user can access the Drillbit process. See User Authentication.
+
+If impersonation is not configured, Drill executes all of the client requests against the file system as the user that started the Drillbit service on the node. This is typically a privileged user. The file system verifies that the system user has permission to access the data.
+
+
+## Example
+When impersonation is disabled and user Bob issues a query through the SQLLine client, SQLLine passes the query to the connecting Drillbit. The Drillbit executes the query as the system user that started the Drill process on the node. For the purpose of this example, we will assume that the system user has full access to the file system. Drill executes the query and returns the results back to the client.
+
+
+When impersonation is enabled and user Bob issues a query through the SQLLine client, the Drillbit executes the query against the file system as Bob. The file system checks to see if Bob has permission to access the data. If so, Drill returns the query results to the client. If Bob does not have permission, Drill returns an error.
+
+
+## Impersonation Support
+The following table lists the clients, storage plugins, and types of queries that you can use with impersonation in Drill:
+
+<table>
+ <tr>
+ <th>Type</th>
+ <th>Supported</th>
+ <th>Not Supported</th>
+ </tr>
+ <tr>
+ <td>Clients</td>
+ <td>SQLLine ODBC JDBC</td>
+ <td>Drill Web UI REST API</td>
+ </tr>
+ <tr>
+ <td>Storage Plugins</td>
+ <td>File System</td>
+ <td>Hive HBase</td>
+ </tr>
+ <tr>
+ <td>Queries</td>
+ <td>When you enable impersonation, the setting applies to queries on data and metadata. For example, if you issue the SHOW SCHEMAS command, Drill impersonates the user logged into the client to access the requested metadata. If you issue a SELECT query on a workspace, Drill impersonates the user logged in to the client to access the requested data. Drill applies impersonation to queries issued using the following commands: <br>SHOW SCHEMAS <br>SHOW DATABASES<br> SHOW TABLES<br> CTAS<br> SELECT<br> CREATE VIEW<br> DROP VIEW<br> SHOW FILES<br> To successfully run the CTAS and CREATE VIEW commands, a user must have write permissions on the directory where the table or view will exist. Running these commands creates artifacts on the file system.</td>
+ <td></td>
+ </tr>
+</table>
+
+## Impersonation and Views
+You can use views with impersonation to provide granular access to data and protect sensitive information. When you create a view, Drill stores the view definition in a file and suffixes the file with .drill.view. For example, if you create a view named myview, Drill creates a view file named myview.drill.view and saves it in the current workspace or the workspace specified, such as dfs.views.myview. See [CREATE VIEW]({{site.baseurl}}/docs/create-view-command/) Command.
+
+You can create a view and grant read permissions on the view to give other users access to the data that the view references. When a user queries the view, Drill impersonates the view owner to access the underlying data. A user with read access to a view can create new views from the originating view to further restrict access on data.
+
+### View Permissions
+A user must have write permission on a directory or workspace to create a view, as well as read access on the table(s) and/or view(s) that the view references. When a user creates a view, permission on the view is set to owner by default. Users can query an existing view or create new views from the view if they have read permissions on the view file and the directory or workspace where the view file is stored.
+
+When users query a view, Drill accesses the underlying data as the user that created the view. If a user does not have permission to access a view, the query fails and Drill returns an error. Only the view owner or a superuser can modify view permissions to change them from owner to group or world.
+
+The view owner or a superuser can modify permissions on the view file directly or they can set view permissions at the system or session level prior to creating any views. Any user that alters view permissions must have write access on the directory or workspace in which they are working. See Modifying Permissions on a View File and Modifying SYSTEM|SESSION Level View Permissions.
+
+#### Modifying Permissions on a View File
+Only a view owner or a super user can modify permissions on a view file to change them from owner to group or world readable. Before you grant permission to users to access a view, verify that they have access to the directory or workspace in which the view file is stored.
+
+Use the `chmod` and `chown` commands with the appropriate octal code to change permissions on a view file:
+
+
+ hadoop fs –chmod <octal code> <file_name>
+ hadoop fs –chown <user>:<group> <file_name>
+Example: `hadoop fs –chmod 750 employees.drill.view`
+
+####Modifying SYSTEM|SESSION Level View Permissions
+Use the `ALTER SESSION|SYSTEM` command with the `new_view_default_permissions` parameter and the appropriate octal code to set view permissions at the system or session level prior to creating a view.
+
+ ALTER SESSION SET `new_view_default_permissions` = '<octal_code>';
+ ALTER SYSTEM SET `new_view_default_permissions` = '<octal_code>';
+
+Example: ``ALTER SESSION SET `new_view_default_permissions` = '777';``
+
+After you set this parameter, Drill applies the same permissions on each view created during the session or across all sessions if set at the system level.
+
+## Chained Impersonation
+You can configure Drill to allow chained impersonation on views when you enable impersonation in the `drill-override.conf` file. Chained impersonation controls the number of identity transitions that Drill can make when a user queries a view. Each identity transition is equal to one hop.
+
+You can set the maximum number of hops on views to limit the number of times that Drill can impersonate a different user when a user queries a view. The default maximum number of hops is set at 3. When the maximum number of hops is set to 0, Drill does not allow impersonation chaining, and a user can only read data for which they have direct permission to access. You may set chain length to 0 to protect highly sensitive data.
+
+The following example depicts a scenario where the maximum hop number is set to 3, and Drill must impersonate three users to access data when Chad queries a view that Jane created:
+
+
+
+In the previous example, Joe created a view V3 from views that user Frank created. In the following example, Joe created view V3 by joining a view that Frank created with a view that Bob created, thus increasing the number of identity transitions that Drill makes from 3 to 4, which exceeds the maximum hop setting of 3.
+
+In this scenario, when Chad queries Jane’s view Drill returns an error stating that the query cannot complete because the number of hops required to access the data exceeds the maximum hop setting of 3 that is configured.
+
+
+
+If users encounter this error, you can increase the maximum hop setting to accommodate users running queries on views. When configuring the maximum number of hops that Drill can make, consider that joined views increase the number of identity transitions required for Drill to access the underlying data.
+
+#### Configuring Impersonation and Chaining
+Chaining is a system-wide setting that applies to all views. Currently, Drill does not provide an option to allow different chain lengths for different views.
+
+Complete the following steps on each Drillbit node to enable user impersonation, and set the maximum number of chained user hops that Drill allows:
+
+1. Navigate to `<drill_installation_directory>/conf/` and edit `drill-override.conf`.
+2. Under `drill.exe`, add the following:
+
+ drill.exec.impersonation: {
+ enabled: true,
+ max_chained_user_hops: 3
+ }
+
+3. Verify that enabled is set to `‘true’`.
+4. Set the maximum number of chained user hops that you want Drill to allow.
+5. (MapR cluster only) Add one of the following lines to the `drill-env.sh` file:
+ * If the underlying file system is not secure, add the following line:
+ ` export MAPR_IMPERSONATION_ENABLED=true`
+ * If the underlying file system has MapR security enabled, add the following line:
+ `export MAPR_TICKETFILE_LOCATION=/opt/mapr/conf/mapruserticket`
+6. Restart the Drillbit process on each Drill node.
+ * In a MapR cluster, run the following command:
+ `maprcli node services -name drill-bits -action restart -nodes <hostname> -f`
+ * In a non-MapR environment, run the following command:
+ <DRILLINSTALL_HOME>/bin/drillbit.sh restart
+
+
+## Impersonation and Chaining Example
+Frank is a senior HR manager at a company. Frank has access to all of the employee data because he is a member of the hr group. Frank created a table named “employees” in his home directory to store the employee data he uses. Only Frank has access to this table.
+
+drwx------ frank:hr /user/frank/employees
+
+Each record in the employees table consists of the following information:
+emp_id, emp_name, emp_ssn, emp_salary, emp_addr, emp_phone, emp_mgr
+
+Frank needs to share a subset of this information with Joe who is an HR manager reporting to Frank. To share the employee data, Frank creates a view called emp_mgr_view that accesses a subset of the data. The emp_mgr_view filters out sensitive employee information, such as the employee social security numbers, and only shows data for the employees that report directly to Joe or the manager running the query on the view. Frank and Joe both belong to the mgr group. Managers have read permission on Frank’s directory.
+
+rwxr----- frank:mgr /user/frank/emp_mgr_view.drill.view
+
+The emp_mgr_view.drill.view file contains the following view definition:
+(view definition: SELECT emp_id, emp_name, emp_salary, emp_addr, emp_phone FROM \`/user/frank/employee\` WHERE emp_mgr = user())
+
+When Joe issues SELECT * FROM emp_mgr_view, Drill impersonates Frank when accessing the employee data, and the query returns the data that Joe has permission to see based on the view definition. The query results do not include any sensitive data because the view protects that information. If Joe tries to query the employees table directly, Drill returns an error or null values.
+
+Because Joe has read permissions on the emp_mgr_view, he can create new views from it to give other users access to the employee data even though he does not own the employees table and cannot access the employees table directly.
+
+Joe needs to share employee contact data with his direct reports, so he creates a special view called emp_team_view to share the employee contact information with his team. Joe creates the view and writes it to his home directory. Joe and his reports belong to a group named joeteam. The joeteam group has read permissions on Joe’s home directory so they can query the view and create new views from it.
+
+rwxr----- joe:joeteam /user/joe/emp_team_view.drill.view
+
+The emp_team_view.drill.view file contains the following view definition:
+
+(view definition: SELECT emp_id, emp_name, emp_phone FROM `/user/frank/emp_mgr_view.drill`);
+
+When anyone on Joe’s team issues SELECT * FROM emp_team_view, Drill impersonates Joe to access the emp_team_view and then impersonates Frank to access the emp_mgr_view and the employee data. Drill returns the data that Joe’s team has can see based on the view definition. If anyone on Joe’s team tries to query the emp_mgr_view or employees table directly, Drill returns an error or null values.
+
+Because Joe’s team has read permissions on the emp_team_view, they can create new views from it and write the views to any directory for which they have write access. Creating views can continue until Drill reaches the maximum number of impersonation hops.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/drill/blob/769eaac2/_docs/manage-drill/080-configuration-options.md
----------------------------------------------------------------------
diff --git a/_docs/manage-drill/080-configuration-options.md b/_docs/manage-drill/080-configuration-options.md
new file mode 100644
index 0000000..41275eb
--- /dev/null
+++ b/_docs/manage-drill/080-configuration-options.md
@@ -0,0 +1,9 @@
+---
+title: "Configuration Options"
+parent: "Manage Drill"
+---
+
+
+
+
+
http://git-wip-us.apache.org/repos/asf/drill/blob/769eaac2/_docs/manage-drill/090-start-stop.md
----------------------------------------------------------------------
diff --git a/_docs/manage-drill/090-start-stop.md b/_docs/manage-drill/090-start-stop.md
new file mode 100644
index 0000000..591b6ab
--- /dev/null
+++ b/_docs/manage-drill/090-start-stop.md
@@ -0,0 +1,42 @@
+---
+title: "Starting/Stopping Drill"
+parent: "Manage Drill"
+---
+How you start Drill depends on the installation method you followed. If you installed Drill in embedded mode, invoking SQLLine automatically starts a Drillbit locally. If you installed Drill in distributed mode, and the Drillbit on a node did not start, start the Drillbit before attempting to run queries. How to start Drill is covered in detail the section, ["Install Drill"]({{site.baseurl}}/docs/install-drill/).
+
+## Examples of Starting Drill
+Issue the **sqlline** command from the Drill installation directory. The simplest example of how to start SQLLine is to identify the protocol, JDBC, and zookeeper node or nodes in the **sqlline** command. This example starts SQLLine on a node in an embedded, single-node cluster:
+
+ sqlline -u jdbc:drill:zk=local
+
+This example also starts SQLLine using the `dfs` storage plugin. Specifying the storage plugin when you start up eliminates the need to specify the storage plugin in the query:
+
+
+ bin/sqlline –u jdbc:drill:schema=dfs;zk=centos26
+
+This command starts SQLLine in distributed, (multi-node) mode in a cluster configured to run zookeeper on three nodes:
+
+ bin/sqlline –u jdbc:drill:zk=cento23,zk=centos24,zk=centos26:5181
+
+## Exiting SQLLine
+
+To exit SQLLine, issue the following command:
+
+ !quit
+
+## Stopping Drill
+
+In some cases, such as stopping while a query is in progress, the `!quit` command does not stop Drill running in embedded mode. In distributed mode, you stop the Drillbit service instead of killing the Drillbit process.
+
+To stop the Drill process on Mac OS X and Linux, use the kill command. On Windows, use the **TaskKill** command.
+
+For example, on Mac OS X and Linux, follow these steps:
+
+ 1. Issue a CTRL Z to stop the query, then start Drill again. If the startup message indicates success, skip the rest of the steps. If not, proceed to step 2.
+ 2. Search for the Drill process IDs.
+
+ $ ps auwx | grep drill
+ 3. Kill each process using the process numbers in the grep output. For example:
+
+ $ sudo kill -9 2674
+
http://git-wip-us.apache.org/repos/asf/drill/blob/769eaac2/_docs/manage-drill/100-ports-used-by-drill.md
----------------------------------------------------------------------
diff --git a/_docs/manage-drill/100-ports-used-by-drill.md b/_docs/manage-drill/100-ports-used-by-drill.md
new file mode 100644
index 0000000..42ecd20
--- /dev/null
+++ b/_docs/manage-drill/100-ports-used-by-drill.md
@@ -0,0 +1,15 @@
+---
+title: "Ports Used by Drill"
+parent: "Manage Drill"
+---
+The following table provides a list of the ports that Drill uses, the port
+type, and a description of how Drill uses the port:
+
+| Port | Type | Description |
+|-------|------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| 8047 | TCP | Needed for the Drill Web UI. |
+| 31010 | TCP | User port address. Used between nodes in a Drill cluster. Needed for an external client, such as Tableau, to connect into thecluster nodes. Also needed for the Drill Web UI. |
+| 31011 | TCP | Control port address. Used between nodes in a Drill cluster. Needed for multi-node installation of Apache Drill. |
+| 31012 | TCP | Data port address. Used between nodes in a Drill cluster. Needed for multi-node installation of Apache Drill. |
+| 46655 | UDP | Used for JGroups and Infinispan. Needed for multi-node installation of Apache Drill. |
+
http://git-wip-us.apache.org/repos/asf/drill/blob/769eaac2/_docs/manage-drill/110-partition-pruning.md
----------------------------------------------------------------------
diff --git a/_docs/manage-drill/110-partition-pruning.md b/_docs/manage-drill/110-partition-pruning.md
new file mode 100644
index 0000000..75f2edd
--- /dev/null
+++ b/_docs/manage-drill/110-partition-pruning.md
@@ -0,0 +1,75 @@
+---
+title: "Partition Pruning"
+parent: "Manage Drill"
+---
+Partition pruning is a performance optimization that limits the number of
+files and partitions that Drill reads when querying file systems and Hive
+tables. Drill only reads a subset of the files that reside in a file system or
+a subset of the partitions in a Hive table when a query matches certain filter
+criteria.
+
+For Drill to apply partition pruning to Hive tables, you must have created the
+tables in Hive using the `PARTITION BY` clause:
+
+`CREATE TABLE <table_name> (<column_name>) PARTITION BY (<column_name>);`
+
+When you create Hive tables using the `PARTITION BY` clause, each partition of
+data is automatically split out into different directories as data is written
+to disk. For more information about Hive partitioning, refer to the [Apache
+Hive wiki](https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL/#LanguageManualDDL-PartitionedTables).
+
+Typically, table data in a file system is organized by directories and
+subdirectories. Queries on table data may contain `WHERE` clause filters on
+specific directories.
+
+Drill’s query planner evaluates the filters as part of a Filter operator. If
+no partition filters are present, the underlying Scan operator reads all files
+in all directories and then sends the data to operators downstream, such as
+Filter.
+
+When partition filters are present, the query planner determines if it can
+push the filters down to the Scan such that the Scan only reads the
+directories that match the partition filters, thus reducing disk I/O.
+
+## Partition Pruning Example
+
+The /`Users/max/data/logs` directory in a file system contains subdirectories
+that span a few years.
+
+The following image shows the hierarchical structure of the `…/logs` directory
+and (sub) directories:
+
+
+
+The following query requests log file data for 2013 from the `…/logs`
+directory in the file system:
+
+ SELECT * FROM dfs.`/Users/max/data/logs` WHERE cust_id < 10 and dir0 = 2013 limit 2;
+
+If you run the `EXPLAIN PLAN` command for the query, you can see that the`
+…/logs` directory is filtered by the scan operator.
+
+ EXPLAIN PLAN FOR SELECT * FROM dfs.`/Users/max/data/logs` WHERE cust_id < 10 and dir0 = 2013 limit 2;
+
+The following image shows a portion of the physical plan when partition
+pruning is applied:
+
+
+
+## Filter Examples
+
+The following queries include examples of the types of filters eligible for
+partition pruning optimization:
+
+**Example 1: Partition filters ANDed together**
+
+ SELECT * FROM dfs.`/Users/max/data/logs` WHERE dir0 = '2014' AND dir1 = '1'
+
+**Example 2: Partition filter ANDed with regular column filter**
+
+ SELECT * FROM dfs.`/Users/max/data/logs` WHERE cust_id < 10 AND dir0 = 2013 limit 2;
+
+**Example 3: Combination of AND, OR involving partition filters**
+
+ SELECT * FROM dfs.`/Users/max/data/logs` WHERE (dir0 = '2013' AND dir1 = '1') OR (dir0 = '2014' AND dir1 = '2')
+
http://git-wip-us.apache.org/repos/asf/drill/blob/769eaac2/_docs/manage-drill/120-monitoring-and-canceling-queries-in-the-Drill-Web-UI.md
----------------------------------------------------------------------
diff --git a/_docs/manage-drill/120-monitoring-and-canceling-queries-in-the-Drill-Web-UI.md b/_docs/manage-drill/120-monitoring-and-canceling-queries-in-the-Drill-Web-UI.md
new file mode 100644
index 0000000..0033838
--- /dev/null
+++ b/_docs/manage-drill/120-monitoring-and-canceling-queries-in-the-Drill-Web-UI.md
@@ -0,0 +1,30 @@
+---
+title: "Monitoring and Canceling Queries in the Drill Web UI"
+parent: "Manage Drill"
+---
+You can monitor and cancel queries from the Drill Web UI. To access the Drill
+Web UI, the Drillbit process must be running on the Drill node that you use to
+access the Drill Web UI.
+
+To monitor or cancel a query from the Drill Web UI, complete the following
+steps:
+
+ 1. Navigate to the Drill Web UI at `<drill_node_ip_address>:8047.`
+When you access the Drill Web UI, you see some general information about Drill
+running in your cluster, such as the nodes running the Drillbit process, the
+various ports Drill is using, and the amount of direct memory assigned to
+Drill.
+
+
+ 2. Select **Profiles** in the toolbar. A list of running and completed queries appears. Drill assigns a query ID to each query and lists the Foreman node. The Foreman is the Drillbit node that receives the query from the client or application. The Foreman drives the entire query.
+
+
+ 3. Click the **Query ID** for the query that you want to monitor or cancel. The Query and Planning window appears.
+
+
+ 4. Select **Edit Query**.
+ 5. Click **Cancel query **to cancel the** query. The following message appears:
+ 
+
+ 6. Optionally, you can re-run the query to see a query summary in this window.
+