You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@drill.apache.org by br...@apache.org on 2015/06/24 02:16:21 UTC
[1/6] drill git commit: DRILL-3321
Repository: drill
Updated Branches:
refs/heads/gh-pages 3d1c00554 -> 957c5d868
DRILL-3321
editorial
broken links
1.1 features
Project: http://git-wip-us.apache.org/repos/asf/drill/repo
Commit: http://git-wip-us.apache.org/repos/asf/drill/commit/cc329855
Tree: http://git-wip-us.apache.org/repos/asf/drill/tree/cc329855
Diff: http://git-wip-us.apache.org/repos/asf/drill/diff/cc329855
Branch: refs/heads/gh-pages
Commit: cc329855f9e955f2041d2b50b3a7264ede884ba6
Parents: 3d1c005
Author: Kristine Hahn <kh...@maprtech.com>
Authored: Mon Jun 22 18:13:20 2015 -0700
Committer: Kristine Hahn <kh...@maprtech.com>
Committed: Mon Jun 22 18:17:16 2015 -0700
----------------------------------------------------------------------
.../010-configure-drill-introduction.md | 24 +++++++++++++++---
.../040-persistent-configuration-storage.md | 21 +++++++++-------
_docs/connect-a-data-source/050-workspaces.md | 21 +++++++++++++---
.../015-using-jdbc-driver.md | 2 +-
.../065-query-directory-functions.md | 2 +-
.../data-types/020-date-time-and-timestamp.md | 4 +--
.../sql-commands/035-partition-by-clause.md | 26 +++++++++-----------
7 files changed, 67 insertions(+), 33 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/drill/blob/cc329855/_docs/configure-drill/010-configure-drill-introduction.md
----------------------------------------------------------------------
diff --git a/_docs/configure-drill/010-configure-drill-introduction.md b/_docs/configure-drill/010-configure-drill-introduction.md
index 62f17f3..42b1f50 100644
--- a/_docs/configure-drill/010-configure-drill-introduction.md
+++ b/_docs/configure-drill/010-configure-drill-introduction.md
@@ -2,9 +2,27 @@
title: "Configure Drill Introduction"
parent: "Configure Drill"
---
-When using Drill, you need to make sufficient memory available Drill when running Drill alone or along side other workloads on the cluster. The next section, ["Configuring Drill Memory"]({{site.baseurl}}/docs/configuring-drill-memory) describes how to configure memory for a Drill cluster. Configuring other resources for [multitenancy clusters]({{site.baseurl}}/docs/configuring-multitenant-resources) or for [sharing a Drillbit]({{site.baseurl}}/docs/configuring-a-shared-drillbit) on a cluster is covered later.
+
+This section briefly describes the following key Drill configuration tasks and provides links to configuration procedures:
+
+* Memory Configuration
+* Multitenancy Configuration
+* Performance and Functionality Configuration
+* Query Profile Data Storage Configuration
+
+## Memory Configuration
+
+When using Drill, you need to make sufficient memory available Drill when running Drill alone or along side other workloads on the cluster. The next section, ["Configuring Drill Memory"]({{site.baseurl}}/docs/configuring-drill-memory) describes how to configure memory for a Drill cluster.
+
+## Multitenancy Configuration
+
+You can configure resources for [multitenancy clusters]({{site.baseurl}}/docs/configuring-multitenant-resources) or for [sharing a Drillbit]({{site.baseurl}}/docs/configuring-a-shared-drillbit) on a cluster.
+
+## Performance and Functionality Configuration
You can also modify options for performance or functionality. For example, changing the default storage format is a typical functional change. The default storage format for CTAS
-statements is Parquet. Using a configuration option, you can modify Drill to store the output data in CSV or JSON format.
+statements is Parquet. Using a configuration option, you can modify Drill to store the output data in CSV or JSON format. The section, ["Configuration Options Introduction"]({{site.baseurl}}/docs/configuration-options-introduction) summarizes the many options you can configure.
+
+## Query Profile Data Storage Configuration
-The section, ["Configuration Options Introduction"]({{site.baseurl}}/docs/configuration-options-introduction) summarizes the many options you can configure.
+To enjoy a problem-free Drill Web UI experience, you need to [configure the ZooKeeper PStore]({{site.baseurl}}/docs/persistent-configuration-storage/#configuring-the-zookeeper-pstore).
http://git-wip-us.apache.org/repos/asf/drill/blob/cc329855/_docs/configure-drill/configuration-options/040-persistent-configuration-storage.md
----------------------------------------------------------------------
diff --git a/_docs/configure-drill/configuration-options/040-persistent-configuration-storage.md b/_docs/configure-drill/configuration-options/040-persistent-configuration-storage.md
index 053f25b..f23a9d9 100644
--- a/_docs/configure-drill/configuration-options/040-persistent-configuration-storage.md
+++ b/_docs/configure-drill/configuration-options/040-persistent-configuration-storage.md
@@ -19,20 +19,23 @@ modes:
{% include startnote.html %}Switching between storage modes does not migrate configuration data.{% include endnote.html %}
-## ZooKeeper for Persistent Configuration Storage
+## Configuring ZooKeeper PStore
-To make Drill installation and configuration simple, Drill uses ZooKeeper to
+Drill uses ZooKeeper to
store persistent configuration data. The ZooKeeper PStore provider stores all
of the persistent configuration data in ZooKeeper except for query profile
-data.
+data. The ZooKeeper PStore provider offloads query profile data to the
+${DRILL_LOG_DIR:-/var/log/drill} directory on Drill nodes.
-The ZooKeeper PStore provider offloads query profile data to the
-${DRILL_LOG_DIR:-/var/log/drill} directory on Drill nodes. If you want the
-query profile data stored in a specific location, you can configure where
-ZooKeeper offloads the data.
+You need to configure the ZooKeeper PStore to use the Drill Web UI when running multiple Drillbits.
-To modify where the ZooKeeper PStore provider offloads query profile data,
-configure the `sys.store.provider.zk.blobroot` property in the `drill.exec`
+## Why Configure the ZooKeeper PStore
+
+When you run multiple DrillBits, configure a specific location for ZooKeeper to offload the query profile data instead of accepting the default temporary location. All Drillbits in the cluster cannot access the temporary location. Consequently, when you do not configure a location on the distributed file system, queries sent to do some Drillbits do not appear in the Completed section of the Drill Web UI. Also, some Running links that you click to get information about the running queries are broken links.
+
+## Configuring the ZooKeeper PStore
+
+To configure the ZooKeeper PStore, set the `sys.store.provider.zk.blobroot` property in the `drill.exec`
block in `<drill_installation_directory>/conf/drill-override.conf` on each
Drill node and then restart the Drillbit service.
http://git-wip-us.apache.org/repos/asf/drill/blob/cc329855/_docs/connect-a-data-source/050-workspaces.md
----------------------------------------------------------------------
diff --git a/_docs/connect-a-data-source/050-workspaces.md b/_docs/connect-a-data-source/050-workspaces.md
index 361bfec..b1156c9 100644
--- a/_docs/connect-a-data-source/050-workspaces.md
+++ b/_docs/connect-a-data-source/050-workspaces.md
@@ -3,9 +3,24 @@ title: "Workspaces"
parent: "Storage Plugin Configuration"
---
When you register an instance of a file system data source, you can configure
-one or more workspaces for the instance. The workspace defines the default directory location of files in a local or distributed file system. The `default`
-workspace points to the root of the file system. Drill searches the workspace to locate data when
-you run a query.
+one or more workspaces for the instance. The workspace defines the directory location of files in a local or distributed file system. Drill searches the workspace to locate data when
+you run a query. The `default`
+workspace points to the root of the file system.
+
+Configuring `workspaces` in the storage plugin definition to include the file location simplifies the query, which is important when querying the same data source repeatedly. After you configure a long path name in the workspaces location property, instead of
+using the full path to the data source, you use dot notation in the FROM
+clause.
+
+``<workspaces>.`<location>```
+
+To query the data source while you are _not_ connected to
+that storage plugin, include the plugin name. This syntax assumes you did not issue a USE statement to connect to a storage plugin that defines the
+location of the data:
+
+``<plugin>.<workspaces>.`<location>```
+
+
+## No Workspaces for Hive and HBase
You cannot create workspaces for
`hive` and `hbase` storage plugins, though Hive databases show up as workspaces in
http://git-wip-us.apache.org/repos/asf/drill/blob/cc329855/_docs/odbc-jdbc-interfaces/015-using-jdbc-driver.md
----------------------------------------------------------------------
diff --git a/_docs/odbc-jdbc-interfaces/015-using-jdbc-driver.md b/_docs/odbc-jdbc-interfaces/015-using-jdbc-driver.md
index 9f471eb..4bf05e7 100755
--- a/_docs/odbc-jdbc-interfaces/015-using-jdbc-driver.md
+++ b/_docs/odbc-jdbc-interfaces/015-using-jdbc-driver.md
@@ -2,7 +2,7 @@
title: "Using the JDBC Driver"
parent: "ODBC/JDBC Interfaces"
---
-This section explains how to install and use the JDBC driver for Apache Drill. For specific examples of client tool connections to Drill via JDBC, see [Using JDBC with SQuirreL]({{ site.baseurl }}/docs/.../) and [Configuring Spotfire Server]({{ site.baseurl }}/docs/.../).
+This section explains how to install and use the JDBC driver for Apache Drill. For specific examples of client tool connections to Drill via JDBC, see [Using JDBC with SQuirreL]({{ site.baseurl }}/docs/using-jdbc-with-squirrel-on-windows) and [Configuring Spotfire Server]({{ site.baseurl }}/docs/configuring-tibco-spotfire-server-with-drill/).
### Prerequisites
http://git-wip-us.apache.org/repos/asf/drill/blob/cc329855/_docs/sql-reference/065-query-directory-functions.md
----------------------------------------------------------------------
diff --git a/_docs/sql-reference/065-query-directory-functions.md b/_docs/sql-reference/065-query-directory-functions.md
index 6e6802b..feb9a05 100644
--- a/_docs/sql-reference/065-query-directory-functions.md
+++ b/_docs/sql-reference/065-query-directory-functions.md
@@ -22,7 +22,7 @@ The following syntax shows how to construct a SELECT statement that using the MA
SELECT * FROM <plugin>.<workspace>.`<filename>`
WHERE dir<n> = MAXDIR('<plugin>.<workspace>', '<filename>');
-Enclose both arguments to the query directory function in single-quotation marks, not backticks. The first argument to the function is the plugin and workspace names in dot notation, and the second argument is the directory name. The dir<n> variable, `dir0`, `dir1`, and so on, refers to
+Enclose both arguments to the query directory function in single-quotation marks, not back ticks. The first argument to the function is the plugin and workspace names in dot notation, and the second argument is the directory name. The dir<n> variable, `dir0`, `dir1`, and so on, refers to
subdirectories in your workspace path, as explained in section, ["Querying Directories"]({{site.baseurl}}/docs/querying-directories).
## Query Directory Function Example
http://git-wip-us.apache.org/repos/asf/drill/blob/cc329855/_docs/sql-reference/data-types/020-date-time-and-timestamp.md
----------------------------------------------------------------------
diff --git a/_docs/sql-reference/data-types/020-date-time-and-timestamp.md b/_docs/sql-reference/data-types/020-date-time-and-timestamp.md
index 8683aa0..60d997f 100644
--- a/_docs/sql-reference/data-types/020-date-time-and-timestamp.md
+++ b/_docs/sql-reference/data-types/020-date-time-and-timestamp.md
@@ -63,9 +63,9 @@ When you want to use interval data in input, use INTERVAL as a keyword that intr
To cast interval data to interval types you can query from a data source such as JSON, see the example in the section, ["Casting Intervals"]({{site.baseurl}}/docs/data-type-conversion/#casting-intervals).
-### Literal Interval Exampls
+### Literal Interval Examples
-In the following example, the INTERVAL keyword followed by 200 adds 200 years to the timestamp. The parentheticated 3 in `YEAR(3)` specifies the precision of the year interval, 3 digits in this case to support the hundreds interval.
+In the following example, the INTERVAL keyword followed by 200 adds 200 years to the timestamp. The 3 in parentheses in `YEAR(3)` specifies the precision of the year interval, 3 digits in this case to support the hundreds interval.
SELECT CURRENT_TIMESTAMP + INTERVAL '200' YEAR(3) FROM sys.version;
+--------------------------+
http://git-wip-us.apache.org/repos/asf/drill/blob/cc329855/_docs/sql-reference/sql-commands/035-partition-by-clause.md
----------------------------------------------------------------------
diff --git a/_docs/sql-reference/sql-commands/035-partition-by-clause.md b/_docs/sql-reference/sql-commands/035-partition-by-clause.md
index b34e5e7..8208663 100644
--- a/_docs/sql-reference/sql-commands/035-partition-by-clause.md
+++ b/_docs/sql-reference/sql-commands/035-partition-by-clause.md
@@ -2,36 +2,34 @@
title: "PARTITION BY Clause"
parent: "SQL Commands"
---
-You can take advantage of automatic partitioning in Drill 1.1 using the PARTITION BY CLAUSE in the CTAS command:
+You can take advantage of automatic partitioning in Drill 1.1 by using the PARTITION BY clause in the CTAS command:
+
+## Syntax
CREATE TABLE table_name [ (column_name, . . .) ]
[ PARTITION_BY (column_name, . . .) ]
AS SELECT_statement;
-The CTAS statement that uses the PARTITION BY clause must store the data in Parquet format. The CTAS statement needs to meet one of the following requirements:
-
-* The column list in the PARTITION by clause are included in the column list following the table_name
-* The SELECT statement has to use a * column if the base table in the SELECT statement is schema-less, and when the partition column is resolved to * column in a schema-less query, this * column cannot be a result of a join operation.
+The CTAS statement that uses the PARTITION BY clause must store the data in Parquet format and meet one of the following requirements:
+* The columns in the column list in the PARTITION BY clause are included in the column list following the table_name
+* The SELECT statement has to use a * column (SELECT *) if the base table in the SELECT statement is schema-less, and when the partition column is resolved to a * column in a schema-less query, this * column cannot be a result of a join operation.
-To create and verify the contents of a table that contains this row:
+The output of using the PARTITION BY clause creates separate files. Each file contains one partition value, and Drill can create multiple files for the same partition value.
- 1. Set the workspace to a writable workspace.
- 2. Set the `store.format` option to Parquet
- 3. Run a CTAS statement with the PARTITION BY clause.
- 4. Go to the directory where the table is stored and check the contents of the file.
- 5. Run a query against the new table.
+Partition pruning uses the parquet column stats to determine which which columns can be used to prune.
Examples:
+ USE cp;
CREATE TABLE mytable1 PARTITION BY (r_regionkey) AS
- SELECT r_regionkey, r_name FROM cp.`tpch/region.parquet`
+ SELECT r_regionkey, r_name FROM cp.`tpch/region.parquet`;
CREATE TABLE mytable2 PARTITION BY (r_regionkey) AS
- SELECT * FROM cp.`tpch/region.parquet`
+ SELECT * FROM cp.`tpch/region.parquet`;
CREATE TABLE mytable3 PARTITION BY (r_regionkey) AS
SELECT r.r_regionkey, r.r_name, n.n_nationkey, n.n_name
FROM cp.`tpch/nation.parquet` n, cp.`tpch/region.parquet` r
- WHERE n.n_regionkey = r.r_regionkey
+ WHERE n.n_regionkey = r.r_regionkey;
[4/6] drill git commit: add zk connect example plus edits
Posted by br...@apache.org.
add zk connect example plus edits
formatting
Project: http://git-wip-us.apache.org/repos/asf/drill/repo
Commit: http://git-wip-us.apache.org/repos/asf/drill/commit/7de2ea77
Tree: http://git-wip-us.apache.org/repos/asf/drill/tree/7de2ea77
Diff: http://git-wip-us.apache.org/repos/asf/drill/diff/7de2ea77
Branch: refs/heads/gh-pages
Commit: 7de2ea77b985d3c892e28dd00377b55585d92454
Parents: e033059
Author: Kristine Hahn <kh...@maprtech.com>
Authored: Tue Jun 23 13:47:07 2015 -0700
Committer: Kristine Hahn <kh...@maprtech.com>
Committed: Tue Jun 23 14:27:31 2015 -0700
----------------------------------------------------------------------
.../020-configuring-odbc-on-linux.md | 6 ++--
.../040-testing-the-odbc-connection.md | 34 +++++++++++---------
2 files changed, 22 insertions(+), 18 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/drill/blob/7de2ea77/_docs/odbc-jdbc-interfaces/configuring-odbc/020-configuring-odbc-on-linux.md
----------------------------------------------------------------------
diff --git a/_docs/odbc-jdbc-interfaces/configuring-odbc/020-configuring-odbc-on-linux.md b/_docs/odbc-jdbc-interfaces/configuring-odbc/020-configuring-odbc-on-linux.md
index ad83541..e89103a 100644
--- a/_docs/odbc-jdbc-interfaces/configuring-odbc/020-configuring-odbc-on-linux.md
+++ b/_docs/odbc-jdbc-interfaces/configuring-odbc/020-configuring-odbc-on-linux.md
@@ -45,7 +45,7 @@ Only include the path to the shared libraries corresponding to the driver matchi
## Step 2: Define the ODBC Data Sources in .odbc.ini
-Define the ODBC data sources in the `~/.odbc.ini` configuration file for your environment. You set the following properties for using Drill in embedded mode:
+Define the ODBC data sources in the `~/.odbc.ini` configuration file for your environment. To use Drill in embedded mode, set the following properties:
ConnectionType=Direct
HOST=localhost
@@ -53,7 +53,7 @@ Define the ODBC data sources in the `~/.odbc.ini` configuration file for your en
ZKQuorum=
ZKClusterID=
-You set the following properties for using Drill in distributed mode:
+To use Drill in distributed mode, set the following properties, described in detail in section ["Direct and ZooKeeper Quorum Properties"]({{site.baseurl}}/docs/configuring-odbc-on-linux/#direct-and-zookeeper-quorum-properties):
ConnectionType=ZooKeeper
HOST=
@@ -108,7 +108,7 @@ The following Linux sample shows a possible configuration for using Drill in dis
To password protect the DSN, uncomment the AuthenticationType, select Basic Authentication for the AuthenticationType, and configure UID and PWD properties.
### Direct and ZooKeeper Quorum Properties
-To use Drill in distributed mode, set ConnectionType to Zookeeper, get the ZKQuorum and ZKClusterID values from the `drill-override.conf` file, and define the ZKQuorum and ZKClusterID properties. Format ZKQuorum as a comma separated list of ZooKeeper nodes in the following format:
+To use Drill in distributed mode, set ConnectionType to Zookeeper, get the ZKQuorum and ZKClusterID values from the `drill-override.conf` file, and define the ZKQuorum and ZKClusterID properties. The `drill-override.conf` is in the `/drill/drill-<version>/conf` directory. Format ZKQuorum as a comma separated list of ZooKeeper nodes in the following format:
`<host name/ip address> : <port number>, <host name/ip address> : <port number>, . . .`
For example:
http://git-wip-us.apache.org/repos/asf/drill/blob/7de2ea77/_docs/odbc-jdbc-interfaces/configuring-odbc/040-testing-the-odbc-connection.md
----------------------------------------------------------------------
diff --git a/_docs/odbc-jdbc-interfaces/configuring-odbc/040-testing-the-odbc-connection.md b/_docs/odbc-jdbc-interfaces/configuring-odbc/040-testing-the-odbc-connection.md
index 46ee4e2..88bf8e3 100644
--- a/_docs/odbc-jdbc-interfaces/configuring-odbc/040-testing-the-odbc-connection.md
+++ b/_docs/odbc-jdbc-interfaces/configuring-odbc/040-testing-the-odbc-connection.md
@@ -19,25 +19,27 @@ There are 32-bit and 64-bit installations of the iODBC driver manager available.
Visit [http://www.iodbc.org](http://www.iodbc.org/) for further details on
using the iODBC driver manager.
-**Example**
+### Example of a Test on Linux
To test the ODBC connection on a Linux cluster, follow these steps:
-1. [Start Drill]({{site.baseurl}}/docs/starting-drill-on-linux-and-mac-os-x/). For example, to start Drill in local mode on a linux cluster:
+1. [Start Drill]({{site.baseurl}}/docs/starting-drill-on-linux-and-mac-os-x/). For example, to start Drill in local mode on a linux cluster:
+
[root@centos23 drill-1.0.0]# bin/drill-localhost
apache drill 1.0.0
"the only truly happy people are children, the creative minority and drill users"
-2. Run `iodbctest` or `iodbctestw`.
- The prompt for an ODBC connection string appears.
+2. In the `samples` directory of the driver manager installation, run `iodbctest` or `iodbctestw`.
+
[root@centos23 libiodbc-3.52.7]# samples/iodbctest
- iODBC Demonstration program
+ iODBC Demonstration program
This program shows an interactive SQL processor
Driver Manager: 03.52.0709.0909
+ The prompt for an ODBC connection string appears.
Enter ODBC connect string (? shows list): ?
-3. Type ? to see the DSN name.
+3. Type ? to see the DSN name.
Output is:
DSN | Driver
@@ -47,26 +49,28 @@ To test the ODBC connection on a Linux cluster, follow these steps:
4. If you are connecting directly to a Drillbit, type an ODBC connection string using the following format:
- `DSN=<DSN name>;ConnectionType=Direct;Host=<Host Name>;Port=<Port Number>`
+ DSN=<DSN name>;ConnectionType=Direct;Host=<Host Name>;Port=<Port Number>
OR
If you are connecting to a ZooKeeper cluster, type an ODBC connection string using the following format:
- `DSN=<DSN Name>;ConnectionType=ZooKeeper;ZKQuorum=<Server1:Port1>
-,<Server2:Port2>;ZKClusterID=<Cluster Name>`
+ DSN=<DSN Name>;ConnectionType=ZooKeeper;ZKQuorum=<Server1:Port1>,<Server2:Port2>;ZKClusterID=<Cluster Name>`
- The following example shows a connection string:
+ The output of a successful test is:
+ `Driver: 1.0.0.1001 (MapR Drill ODBC Driver)
+ SQL> `
+ After the `SQL>` prompt appears, type `quit;`, and go to the Drill shell to run commands. Do not attempt to run SQL commands from this prompt.
- `DSN=Sample MapR Drill DSN 64;ConnectionType=Direct;Host=localhost;Port=31010`
+### Example Connection Strings
- The output of a successful test is:
+The following example shows a connection string for a direct connection:
- `Driver: 1.0.0.1001 (MapR Drill ODBC Driver)
+ DSN=Sample MapR Drill DSN 64;ConnectionType=Direct;Host=localhost;Port=31010
- SQL> `
+The following example shows a connection string for a ZooKeeper cluster connection:
- After he `SQL>` prompt appears, go to the Drill shell to run commands. Do not attempt to run SQL commands from this prompt.
+ DSN=Sample MapR Drill DSN 64;ConnectionType=ZooKeeper;ZKQuorum=centos23.lab:5181;ZKClusterID=docs41cluster-drillbits
## Testing the ODBC Connection on Mac OS X
[6/6] drill git commit: drill lowercase
Posted by br...@apache.org.
drill lowercase
Project: http://git-wip-us.apache.org/repos/asf/drill/repo
Commit: http://git-wip-us.apache.org/repos/asf/drill/commit/957c5d86
Tree: http://git-wip-us.apache.org/repos/asf/drill/tree/957c5d86
Diff: http://git-wip-us.apache.org/repos/asf/drill/diff/957c5d86
Branch: refs/heads/gh-pages
Commit: 957c5d86846cab136b39ebde672dc2f762cca155
Parents: 06cc252
Author: Kristine Hahn <kh...@maprtech.com>
Authored: Tue Jun 23 17:02:04 2015 -0700
Committer: Kristine Hahn <kh...@maprtech.com>
Committed: Tue Jun 23 17:02:04 2015 -0700
----------------------------------------------------------------------
_docs/connect-a-data-source/100-mapr-db-format.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/drill/blob/957c5d86/_docs/connect-a-data-source/100-mapr-db-format.md
----------------------------------------------------------------------
diff --git a/_docs/connect-a-data-source/100-mapr-db-format.md b/_docs/connect-a-data-source/100-mapr-db-format.md
index 74f8387..57d66c2 100755
--- a/_docs/connect-a-data-source/100-mapr-db-format.md
+++ b/_docs/connect-a-data-source/100-mapr-db-format.md
@@ -2,7 +2,7 @@
title: "MapR-DB Format"
parent: "Connect a Data Source"
---
-The MapR-DB format is not included in the Apache drill release. Drill includes a `maprdb` format for MapR-DB that is defined within the
+The MapR-DB format is not included in the Apache Drill release. Drill includes a `maprdb` format for MapR-DB that is defined within the
default `dfs` storage plugin instance when you install Drill from the `mapr-drill` package on a MapR node. The `maprdb` format improves the
estimated number of rows that Drill uses to plan a query. It also enables you
to query tables like you would query files in a file system because MapR-DB
[3/6] drill git commit: fix log dir info
Posted by br...@apache.org.
fix log dir info
fix link
typo
Project: http://git-wip-us.apache.org/repos/asf/drill/repo
Commit: http://git-wip-us.apache.org/repos/asf/drill/commit/e0330591
Tree: http://git-wip-us.apache.org/repos/asf/drill/tree/e0330591
Diff: http://git-wip-us.apache.org/repos/asf/drill/diff/e0330591
Branch: refs/heads/gh-pages
Commit: e0330591a113eda730f3c9a7082e17f73e42c0f3
Parents: 837e4ad
Author: Kristine Hahn <kh...@maprtech.com>
Authored: Tue Jun 23 11:17:13 2015 -0700
Committer: Kristine Hahn <kh...@maprtech.com>
Committed: Tue Jun 23 11:19:07 2015 -0700
----------------------------------------------------------------------
_docs/configure-drill/010-configure-drill-introduction.md | 2 +-
.../040-persistent-configuration-storage.md | 3 +--
.../connect-a-data-source/035-plugin-configuration-basics.md | 7 ++++++-
3 files changed, 8 insertions(+), 4 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/drill/blob/e0330591/_docs/configure-drill/010-configure-drill-introduction.md
----------------------------------------------------------------------
diff --git a/_docs/configure-drill/010-configure-drill-introduction.md b/_docs/configure-drill/010-configure-drill-introduction.md
index 42b1f50..3a50315 100644
--- a/_docs/configure-drill/010-configure-drill-introduction.md
+++ b/_docs/configure-drill/010-configure-drill-introduction.md
@@ -25,4 +25,4 @@ statements is Parquet. Using a configuration option, you can modify Drill to sto
## Query Profile Data Storage Configuration
-To enjoy a problem-free Drill Web UI experience, you need to [configure the ZooKeeper PStore]({{site.baseurl}}/docs/persistent-configuration-storage/#configuring-the-zookeeper-pstore).
+To enjoy a problem-free Drill Web UI experience, you need to [configure the ZooKeeper PStore]({{site.baseurl}}/docs/persistent-configuration-storage/#configuring-zookeeper-pstore).
http://git-wip-us.apache.org/repos/asf/drill/blob/e0330591/_docs/configure-drill/configuration-options/040-persistent-configuration-storage.md
----------------------------------------------------------------------
diff --git a/_docs/configure-drill/configuration-options/040-persistent-configuration-storage.md b/_docs/configure-drill/configuration-options/040-persistent-configuration-storage.md
index 95f6bdc..2c7d5c7 100644
--- a/_docs/configure-drill/configuration-options/040-persistent-configuration-storage.md
+++ b/_docs/configure-drill/configuration-options/040-persistent-configuration-storage.md
@@ -24,8 +24,7 @@ modes:
Drill uses ZooKeeper to
store persistent configuration data. The ZooKeeper PStore provider stores all
of the persistent configuration data in ZooKeeper except for query profile
-data. The ZooKeeper PStore provider offloads query profile data to the
-${DRILL_LOG_DIR:-/var/log/drill} directory on Drill nodes.
+data. The ZooKeeper PStore provider offloads query profile data to the Drill log directory on Drill nodes.
You need to configure the ZooKeeper PStore to use the Drill Web UI when running multiple Drillbits.
http://git-wip-us.apache.org/repos/asf/drill/blob/e0330591/_docs/connect-a-data-source/035-plugin-configuration-basics.md
----------------------------------------------------------------------
diff --git a/_docs/connect-a-data-source/035-plugin-configuration-basics.md b/_docs/connect-a-data-source/035-plugin-configuration-basics.md
index 7844c28..23f7919 100644
--- a/_docs/connect-a-data-source/035-plugin-configuration-basics.md
+++ b/_docs/connect-a-data-source/035-plugin-configuration-basics.md
@@ -193,6 +193,11 @@ For example, this command creates a plugin named myplugin for reading files of a
If you need to add a storage plugin to Drill and do not want to use a web browser, you can create a [bootstrap-storage-plugins.json](https://github.com/apache/drill/blob/master/contrib/storage-hbase/src/main/resources/bootstrap-storage-plugins.json) file and include it on the classpath when starting Drill. The storage plugin loads when Drill starts up.
-Bootstrapping a storage plugin works only when the first drillbit in the cluster first starts up. After cluster startup, you have to use the REST API or Drill Web UI to add a storage plugin.
+Bootstrapping a storage plugin works only when the first drillbit in the cluster first starts up. The configuration is
+stored in zookeeper, preventing Drill from picking up the boostrap-storage-plugins.json again.
+
+After cluster startup, you have to use the REST API or Drill Web UI to add a storage plugin. Alternatively, you
+can modify the entry in zookeeper by uploading the json file for
+that plugin to the /drill directory of the zookeeper installation, or just delete the /drill directory if you do not have configuration properties to preserve.
If you configure an HBase storage plugin using bootstrap-storage-plugins.json file and HBase is not installed, you might experience a delay when executing the queries. Configure the [HBase client timeout](http://hbase.apache.org/book.html#config.files) and retry settings in the config block of HBase plugin instance configuration.
[2/6] drill git commit: differentiate titles
Posted by br...@apache.org.
differentiate titles
Project: http://git-wip-us.apache.org/repos/asf/drill/repo
Commit: http://git-wip-us.apache.org/repos/asf/drill/commit/837e4adb
Tree: http://git-wip-us.apache.org/repos/asf/drill/tree/837e4adb
Diff: http://git-wip-us.apache.org/repos/asf/drill/diff/837e4adb
Branch: refs/heads/gh-pages
Commit: 837e4adbc48ccff0a14562e268ff5cbfeb24686c
Parents: cc32985
Author: Kristine Hahn <kh...@maprtech.com>
Authored: Mon Jun 22 18:25:03 2015 -0700
Committer: Kristine Hahn <kh...@maprtech.com>
Committed: Mon Jun 22 18:25:03 2015 -0700
----------------------------------------------------------------------
.../040-persistent-configuration-storage.md | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/drill/blob/837e4adb/_docs/configure-drill/configuration-options/040-persistent-configuration-storage.md
----------------------------------------------------------------------
diff --git a/_docs/configure-drill/configuration-options/040-persistent-configuration-storage.md b/_docs/configure-drill/configuration-options/040-persistent-configuration-storage.md
index f23a9d9..95f6bdc 100644
--- a/_docs/configure-drill/configuration-options/040-persistent-configuration-storage.md
+++ b/_docs/configure-drill/configuration-options/040-persistent-configuration-storage.md
@@ -29,11 +29,11 @@ ${DRILL_LOG_DIR:-/var/log/drill} directory on Drill nodes.
You need to configure the ZooKeeper PStore to use the Drill Web UI when running multiple Drillbits.
-## Why Configure the ZooKeeper PStore
+### Why Configure the ZooKeeper PStore
When you run multiple DrillBits, configure a specific location for ZooKeeper to offload the query profile data instead of accepting the default temporary location. All Drillbits in the cluster cannot access the temporary location. Consequently, when you do not configure a location on the distributed file system, queries sent to do some Drillbits do not appear in the Completed section of the Drill Web UI. Also, some Running links that you click to get information about the running queries are broken links.
-## Configuring the ZooKeeper PStore
+### How to Configure the ZooKeeper PStore
To configure the ZooKeeper PStore, set the `sys.store.provider.zk.blobroot` property in the `drill.exec`
block in `<drill_installation_directory>/conf/drill-override.conf` on each
[5/6] drill git commit: add zk connect example plus edits
Posted by br...@apache.org.
add zk connect example plus edits
formatting
cannot hv 2 pg same name
typo
formatting
conflicting titles
Project: http://git-wip-us.apache.org/repos/asf/drill/repo
Commit: http://git-wip-us.apache.org/repos/asf/drill/commit/06cc2524
Tree: http://git-wip-us.apache.org/repos/asf/drill/tree/06cc2524
Diff: http://git-wip-us.apache.org/repos/asf/drill/diff/06cc2524
Branch: refs/heads/gh-pages
Commit: 06cc2524dce2b5c8444b9f286998644d1e2b0c73
Parents: 7de2ea7
Author: Kristine Hahn <kh...@maprtech.com>
Authored: Tue Jun 23 13:47:07 2015 -0700
Committer: Kristine Hahn <kh...@maprtech.com>
Committed: Tue Jun 23 16:52:25 2015 -0700
----------------------------------------------------------------------
_data/docs.json | 54 +++-----------
_docs/archived-pages/030-partition-pruning.md | 75 --------------------
.../connect-a-data-source/100-mapr-db-format.md | 2 +-
.../performance-tuning/020-partition-pruning.md | 4 +-
4 files changed, 13 insertions(+), 122 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/drill/blob/06cc2524/_data/docs.json
----------------------------------------------------------------------
diff --git a/_data/docs.json b/_data/docs.json
index ddd3a4e..6f6aca1 100644
--- a/_data/docs.json
+++ b/_data/docs.json
@@ -511,31 +511,14 @@
}
],
"children": [],
- "next_title": "Partition Pruning",
- "next_url": "/docs/partition-pruning/",
+ "next_title": "Progress Reports",
+ "next_url": "/docs/progress-reports/",
"parent": "Archived Pages",
"previous_title": "How to Run the Drill Demo",
"previous_url": "/docs/how-to-run-the-drill-demo/",
"relative_path": "_docs/archived-pages/020-what-is-apache-drill.md",
"title": "What is Apache Drill",
"url": "/docs/what-is-apache-drill/"
- },
- {
- "breadcrumbs": [
- {
- "title": "Archived Pages",
- "url": "/docs/archived-pages/"
- }
- ],
- "children": [],
- "next_title": "Progress Reports",
- "next_url": "/docs/progress-reports/",
- "parent": "Archived Pages",
- "previous_title": "What is Apache Drill",
- "previous_url": "/docs/what-is-apache-drill/",
- "relative_path": "_docs/archived-pages/030-partition-pruning.md",
- "title": "Partition Pruning",
- "url": "/docs/partition-pruning/"
}
],
"next_title": "How to Run the Drill Demo",
@@ -6168,8 +6151,8 @@
"next_title": "2014 Q1 Drill Report",
"next_url": "/docs/2014-q1-drill-report/",
"parent": "",
- "previous_title": "Partition Pruning",
- "previous_url": "/docs/partition-pruning/",
+ "previous_title": "What is Apache Drill",
+ "previous_url": "/docs/what-is-apache-drill/",
"relative_path": "_docs/160-progress-reports.md",
"title": "Progress Reports",
"url": "/docs/progress-reports/"
@@ -10933,8 +10916,8 @@
}
],
"children": [],
- "next_title": "Partition Pruning",
- "next_url": "/docs/partition-pruning/",
+ "next_title": "Progress Reports",
+ "next_url": "/docs/progress-reports/",
"parent": "Archived Pages",
"previous_title": "How to Run the Drill Demo",
"previous_url": "/docs/how-to-run-the-drill-demo/",
@@ -15316,31 +15299,14 @@
}
],
"children": [],
- "next_title": "Partition Pruning",
- "next_url": "/docs/partition-pruning/",
+ "next_title": "Progress Reports",
+ "next_url": "/docs/progress-reports/",
"parent": "Archived Pages",
"previous_title": "How to Run the Drill Demo",
"previous_url": "/docs/how-to-run-the-drill-demo/",
"relative_path": "_docs/archived-pages/020-what-is-apache-drill.md",
"title": "What is Apache Drill",
"url": "/docs/what-is-apache-drill/"
- },
- {
- "breadcrumbs": [
- {
- "title": "Archived Pages",
- "url": "/docs/archived-pages/"
- }
- ],
- "children": [],
- "next_title": "Progress Reports",
- "next_url": "/docs/progress-reports/",
- "parent": "Archived Pages",
- "previous_title": "What is Apache Drill",
- "previous_url": "/docs/what-is-apache-drill/",
- "relative_path": "_docs/archived-pages/030-partition-pruning.md",
- "title": "Partition Pruning",
- "url": "/docs/partition-pruning/"
}
],
"next_title": "How to Run the Drill Demo",
@@ -15376,8 +15342,8 @@
"next_title": "2014 Q1 Drill Report",
"next_url": "/docs/2014-q1-drill-report/",
"parent": "",
- "previous_title": "Partition Pruning",
- "previous_url": "/docs/partition-pruning/",
+ "previous_title": "What is Apache Drill",
+ "previous_url": "/docs/what-is-apache-drill/",
"relative_path": "_docs/160-progress-reports.md",
"title": "Progress Reports",
"url": "/docs/progress-reports/"
http://git-wip-us.apache.org/repos/asf/drill/blob/06cc2524/_docs/archived-pages/030-partition-pruning.md
----------------------------------------------------------------------
diff --git a/_docs/archived-pages/030-partition-pruning.md b/_docs/archived-pages/030-partition-pruning.md
deleted file mode 100644
index 3dc79ff..0000000
--- a/_docs/archived-pages/030-partition-pruning.md
+++ /dev/null
@@ -1,75 +0,0 @@
----
-title: "Partition Pruning"
-parent: "Archived Pages"
----
-Partition pruning is a performance optimization that limits the number of
-files and partitions that Drill reads when querying file systems and Hive
-tables. Drill only reads a subset of the files that reside in a file system or
-a subset of the partitions in a Hive table when a query matches certain filter
-criteria.
-
-For Drill to apply partition pruning to Hive tables, you must have created the
-tables in Hive using the `PARTITION BY` clause:
-
-`CREATE TABLE <table_name> (<column_name>) PARTITION BY (<column_name>);`
-
-When you create Hive tables using the `PARTITION BY` clause, each partition of
-data is automatically split out into different directories as data is written
-to disk. For more information about Hive partitioning, refer to the [Apache
-Hive wiki](https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL/#LanguageManualDDL-PartitionedTables).
-
-Typically, table data in a file system is organized by directories and
-subdirectories. Queries on table data may contain `WHERE` clause filters on
-specific directories.
-
-Drill’s query planner evaluates the filters as part of a Filter operator. If
-no partition filters are present, the underlying Scan operator reads all files
-in all directories and then sends the data to operators downstream, such as
-Filter.
-
-When partition filters are present, the query planner determines if it can
-push the filters down to the Scan such that the Scan only reads the
-directories that match the partition filters, thus reducing disk I/O.
-
-## Partition Pruning Example
-
-The /`Users/max/data/logs` directory in a file system contains subdirectories
-that span a few years.
-
-The following image shows the hierarchical structure of the `…/logs` directory
-and (sub) directories:
-
-
-
-The following query requests log file data for 2013 from the `…/logs`
-directory in the file system:
-
- SELECT * FROM dfs.`/Users/max/data/logs` WHERE cust_id < 10 and dir0 = 2013 limit 2;
-
-If you run the `EXPLAIN PLAN` command for the query, you can see that the`
-…/logs` directory is filtered by the scan operator.
-
- EXPLAIN PLAN FOR SELECT * FROM dfs.`/Users/max/data/logs` WHERE cust_id < 10 and dir0 = 2013 limit 2;
-
-The following image shows a portion of the physical plan when partition
-pruning is applied:
-
-
-
-## Filter Examples
-
-The following queries include examples of the types of filters eligible for
-partition pruning optimization:
-
-**Example 1: Partition filters ANDed together**
-
- SELECT * FROM dfs.`/Users/max/data/logs` WHERE dir0 = '2014' AND dir1 = '1'
-
-**Example 2: Partition filter ANDed with regular column filter**
-
- SELECT * FROM dfs.`/Users/max/data/logs` WHERE cust_id < 10 AND dir0 = 2013 limit 2;
-
-**Example 3: Combination of AND, OR involving partition filters**
-
- SELECT * FROM dfs.`/Users/max/data/logs` WHERE (dir0 = '2013' AND dir1 = '1') OR (dir0 = '2014' AND dir1 = '2')
-
http://git-wip-us.apache.org/repos/asf/drill/blob/06cc2524/_docs/connect-a-data-source/100-mapr-db-format.md
----------------------------------------------------------------------
diff --git a/_docs/connect-a-data-source/100-mapr-db-format.md b/_docs/connect-a-data-source/100-mapr-db-format.md
index 25098dd..74f8387 100755
--- a/_docs/connect-a-data-source/100-mapr-db-format.md
+++ b/_docs/connect-a-data-source/100-mapr-db-format.md
@@ -2,7 +2,7 @@
title: "MapR-DB Format"
parent: "Connect a Data Source"
---
-The MapR-DB format is not included in apache drill release. Drill includes a `maprdb` format for MapR-DB that is defined within the
+The MapR-DB format is not included in the Apache drill release. Drill includes a `maprdb` format for MapR-DB that is defined within the
default `dfs` storage plugin instance when you install Drill from the `mapr-drill` package on a MapR node. The `maprdb` format improves the
estimated number of rows that Drill uses to plan a query. It also enables you
to query tables like you would query files in a file system because MapR-DB
http://git-wip-us.apache.org/repos/asf/drill/blob/06cc2524/_docs/performance-tuning/020-partition-pruning.md
----------------------------------------------------------------------
diff --git a/_docs/performance-tuning/020-partition-pruning.md b/_docs/performance-tuning/020-partition-pruning.md
index 49be254..7c3f272 100755
--- a/_docs/performance-tuning/020-partition-pruning.md
+++ b/_docs/performance-tuning/020-partition-pruning.md
@@ -7,12 +7,12 @@ Partition pruning is a performance optimization that limits the number of files
The query planner in Drill performs partition pruning by evaluating the filters. If no partition filters are present, the underlying Scan operator reads all files in all directories and then sends the data to operators, such as Filter, downstream. When partition filters are present, the query planner pushes the filters down to the Scan if possible. The Scan reads only the directories that match the partition filters, thus reducing disk I/O.
-## How to Use Partition Pruning
+## How Partition Data
You can partition data manually or automatically to take advantage of partition pruning in Drill. In Drill 1.0 and earlier, you need to organize your data in such a way to take advantage of partition pruning. In Drill 1.1.0 and later, if the data source is Parquet, you can partition data automatically using CTAS--no data organization tasks required.
## Automatic Partitioning
-Automatic partitioning in Drill 1.1.0 and later occurs when you write Parquet date using the [[PARTITION BY]({{site.baseurl}}/docs/partition-by-clause/) clause in the CTAS statemebnt.
+Automatic partitioning in Drill 1.1.0 and later occurs when you write Parquet date using the [PARTITION BY]({{site.baseurl}}/docs/partition-by-clause/) clause in the CTAS statemebnt.
Automatic partitioning creates separate files, but not separate directories, for different partitions. Each file contains exactly one partition value, but there could be multiple files for the same partition value.