You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@drill.apache.org by br...@apache.org on 2018/12/28 06:25:37 UTC

[drill] branch gh-pages updated: Doc updates and edits for Drill 1.15

This is an automated email from the ASF dual-hosted git repository.

bridgetb pushed a commit to branch gh-pages
in repository https://gitbox.apache.org/repos/asf/drill.git


The following commit(s) were added to refs/heads/gh-pages by this push:
     new ca8ee66  Doc updates and edits for Drill 1.15
ca8ee66 is described below

commit ca8ee666973e694fc11fc55edd9e94d508fb62de
Author: Bridget Bevens <bb...@maprtech.com>
AuthorDate: Thu Dec 27 22:24:55 2018 -0800

    Doc updates and edits for Drill 1.15
---
 .../120-configuring-the-drill-shell.md             | 18 ++++--
 .../configuration-options/020-start-up-options.md  | 20 +++++--
 ...093-configuring-custom-acls-to-secure-znodes.md | 25 ++++++--
 .../030-date-time-functions-and-arithmetic.md      | 66 +++++++++++-----------
 4 files changed, 81 insertions(+), 48 deletions(-)

diff --git a/_docs/configure-drill/120-configuring-the-drill-shell.md b/_docs/configure-drill/120-configuring-the-drill-shell.md
index 1f5783a..41eb5e8 100644
--- a/_docs/configure-drill/120-configuring-the-drill-shell.md
+++ b/_docs/configure-drill/120-configuring-the-drill-shell.md
@@ -1,11 +1,11 @@
 ---
 title: "Configuring the Drill Shell"
-date: 2018-12-27
+date: 2018-12-28
 parent: "Configure Drill"
 ---  
 Drill uses SQLLine as the Drill shell. SQLLine is a pure-Java console-based utility for connecting to relational databases and running SQL commands. 
 
-Starting in Drill 1.15, Drill uses SQLLine 1.6, which you can customize through the Drill [configuration file, drill-sqlline-override.conf]({{site.baseurl}}/docs/configuring-the-drill-shell/#customizing-sqlline-in-the-drill-sqlline-override-conf-file). Before installing and running Drill with SQLLine 1.6, delete the old SQLLine history file The history file is located in the following location:  
+Starting in Drill 1.15, Drill uses SQLLine 1.6, which you can customize through the Drill [configuration file, drill-sqlline-override.conf]({{site.baseurl}}/docs/configuring-the-drill-shell/#customizing-sqlline-in-the-drill-sqlline-override-conf-file). Before installing and running Drill with SQLLine 1.6, delete the old SQLLine history file located in:  
 
 
 - $HOME/.sqlline/history (UNIX, Linux, Mac OS)
@@ -19,9 +19,12 @@ Formatting tables takes time, which you may notice when running a huge query usi
 
 ## Drill Shell Commands
 
-The following table lists the Drill shell commands that you can run from the command line:
+The following table lists some of the Drill shell commands that you can run from the command line. To see a complete list of shell commands issue the `!help` command.   
 
-| Command       | Description                                                                                                                           |
+**Note:** Some new shell commands, such as `!reset`, and key-strokes were introduced when Drill upgraded to SQLLine 1.6 (Drill 1.15 and later); however, not all of the commands are supported.   
+
+
+| **Command**       | **Description **                                                                                                                          |
 |---------------|---------------------------------------------------------------------------------------------------------------------------------------|
 | !brief        | Set verbose mode off.                                                                                                                 |
 | !close        | Close the current connection to the database.                                                                                         |
@@ -34,7 +37,10 @@ The following table lists the Drill shell commands that you can run from the com
 | !properties   | Connect to the database specified in the properties file(s).                                                                          |
 | !quit         | Exits the Drill shell.                                                                                                                |
 | !reconnect    | Reconnect to the database.                                                                                                            |
-| !record       | Record all output to the specified file.                                                                                              |
+| !record       | Record all output to the specified file.
+|
+| !reset        | Reset a sqlline variable
+|                                                                                             |
 | !run          | Run a script from the specified file.                                                                                                 |
 | !save         | Save the current variables and aliases.                                                                                               |
 | !script       | Start saving a script to a file.                                                                                                      |
@@ -70,7 +76,7 @@ The following table lists some of the set command variables that you can use wit
 
 **Note:** Issue the !set command without a variable to see the full list of command variables.
 
-| Variable Name   | Valid Variable Values  | Description                                                                                                                                                            |
+| **Variable Name**   | **Valid Variable Values**  | **Description**                                                                                                                                                            |
 |-----------------|------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
 | autoCommit      | true/false             | Enable/disable automatic transaction commit. Should remain enabled (true). Drill performs read-only operations primarily, and autocommit writes. Drill JDBC throws an exception if autoCommit is disabled.                                                                                             |
 | autoSave        | true/false             | Automatically save preferences.                                                                                                                                        |
diff --git a/_docs/configure-drill/configuration-options/020-start-up-options.md b/_docs/configure-drill/configuration-options/020-start-up-options.md
index 78164a1..bfb2e38 100644
--- a/_docs/configure-drill/configuration-options/020-start-up-options.md
+++ b/_docs/configure-drill/configuration-options/020-start-up-options.md
@@ -1,6 +1,6 @@
 ---
 title: "Start-Up Options"
-date: 2018-07-06 21:22:11 UTC
+date: 2018-12-28
 parent: "Configuration Options"
 ---
 The start-up options for Drill reside in a [HOCON](https://github.com/typesafehub/config/blob/master/HOCON.md) configuration file format, which is a hybrid between a properties file and a JSON file. Drill start-up options consist of a group of files with a nested relationship. At the bottom of the file hierarchy are the default files that Drill provides, starting with `drill-default.conf`. 
@@ -42,10 +42,20 @@ Run the following query to see a list of the available start-up options:
 
 ## Configuring Start-Up Options
 
-You can configure start-up options for each drillbit in `<drill_home>/conf/drill-override.conf` .
+You can configure start-up options for each Drillbit in `<drill_home>/conf/drill-override.conf`.
 
 The summary of start-up options, also known as boot options, lists default values. The following descriptions provide more detail on key options that are frequently reconfigured:
 
+* **store.parquet.complex.batch.num_records**  
+  Introduced in Drill 1.15. Sets batch sizing (number of rows per batch) for the Parquet reader. The default value is 4,000 rows. If rows are large and queries return out-of-memory errors, reduce the value. You can set this option at the system level in `<drill_home>/conf/drill-override.conf` or at the session level using the SET command, as shown:  
+
+		SET `store.parquet.complex.batch.num_records`=2000;
+		+-------+---------------------------------------------------+
+		|  ok   |                      summary                      |
+		+-------+---------------------------------------------------+
+		| true  | store.parquet.complex.batch.num_records updated.  |
+		+-------+---------------------------------------------------+
+
 * **drill.exec.http.ssl_enabled**  
   Introduced in Drill 1.2. Enables or disables [HTTPS support]({{site.baseurl}}/docs/configuring-web-console-and-rest-api-security/#https-support). Settings are TRUE and FALSE, respectively. The default is FALSE.  
   
@@ -56,10 +66,10 @@ The summary of start-up options, also known as boot options, lists default value
   Defines the amount of memory available, in terms of record batches, to hold data on the downstream side of an operation. Drill pushes data downstream as quickly as possible to make data immediately available. This requires Drill to use memory to hold the data pending operations. When data on a downstream operation is required, that data is immediately available so Drill does not have to go over the network to process it. Providing more memory to this option increases the speed at which [...]
   
 * **drill.exec.spill.fs**  
-Introduced in Drill 1.11. The default file system on the local machine into which the Sort, Hash Aggregate, and Hash Join operators spill data. This is the recommended option to use for spilling. You can configure this option so that data spills into a distributed file system, such as hdfs. For example, "hdfs:///". The default setting is "file:///". See [Sort-Based and Hash-Based Memory Constrained Operators]({{site.baseurl}}/docs/sort-based-and-hash-based-memory-constrained-operators/)  [...]
+  Introduced in Drill 1.11. The default file system on the local machine into which the Sort, Hash Aggregate, and Hash Join operators spill data. This is the recommended option to use for spilling. You can configure this option so that data spills into a distributed file system, such as hdfs. For example, "hdfs:///". The default setting is "file:///". See [Sort-Based and Hash-Based Memory Constrained Operators]({{site.baseurl}}/docs/sort-based-and-hash-based-memory-constrained-operators/ [...]
   
 * **drill.exec.spill.directories**  
-Introduced in Drill 1.11. The list of directories into which the Sort, Hash Aggregate, and Hash Join operators spill data. The list must be an array with directories separated by a comma, for example ["/fs1/drill/spill" , "/fs2/drill/spill" , "/fs3/drill/spill"]. This is the recommended option for spilling to multiple directories. The default setting is ["/tmp/drill/spill"]. See [Sort-Based and Hash-Based Memory Constrained Operators]({{site.baseurl}}/docs/sort-based-and-hash-based-memor [...]
+  Introduced in Drill 1.11. The list of directories into which the Sort, Hash Aggregate, and Hash Join operators spill data. The list must be an array with directories separated by a comma, for example ["/fs1/drill/spill" , "/fs2/drill/spill" , "/fs3/drill/spill"]. This is the recommended option for spilling to multiple directories. The default setting is ["/tmp/drill/spill"]. See [Sort-Based and Hash-Based Memory Constrained Operators]({{site.baseurl}}/docs/sort-based-and-hash-based-mem [...]
 
 * **drill.exec.zk.connect**  
   Provides Drill with the ZooKeeper quorum to use to connect to data sources. Change this setting to point to the ZooKeeper quorum that you want Drill to use. You must configure this option on each Drillbit node.  
@@ -71,7 +81,7 @@ Introduced in Drill 1.11. The list of directories into which the Sort, Hash Aggr
   Introduced in Drill 1.11. Sets the maximum number of most recent profiles to retain in memory when the `drill.exec.profiles.store.inmemory` option is enabled. Default is 1000.  
   
 * **drill.exec.storage.action\_on\_plugins\_override\_file**  
-Introduced in Drill 1.14. Determines what happens to the [`storage-plugins-override.conf`]({{site.baseurl}}/docs/configuring-storage-plugins/#configuring-storage-plugins-with-the-storage-plugins-override.conf-file) file after Drill successfully updates storage plugin configurations. If you do not want Drill to apply the configurations after restarting, set the option to `"rename"` or `"remove"`.  This option accepts the following values:  
+  Introduced in Drill 1.14. Determines what happens to the [`storage-plugins-override.conf`]({{site.baseurl}}/docs/configuring-storage-plugins/#configuring-storage-plugins-with-the-storage-plugins-override.conf-file) file after Drill successfully updates storage plugin configurations. If you do not want Drill to apply the configurations after restarting, set the option to `"rename"` or `"remove"`.  This option accepts the following values:  
 
        - `"none"` - (Default) The file remains in the directory after Drill uses the file.  
        - `"rename"` - The `storage-plugins-override.conf` file name is changed to `storage-plugins-override-[current_timestamp].conf` after Drill uses the file.  
diff --git a/_docs/configure-drill/securing-drill/093-configuring-custom-acls-to-secure-znodes.md b/_docs/configure-drill/securing-drill/093-configuring-custom-acls-to-secure-znodes.md
index 965d69d..b43c2a9 100644
--- a/_docs/configure-drill/securing-drill/093-configuring-custom-acls-to-secure-znodes.md
+++ b/_docs/configure-drill/securing-drill/093-configuring-custom-acls-to-secure-znodes.md
@@ -1,15 +1,18 @@
 ---
 title: "Configuring Custom ACLs to Secure znodes"
-date: 2018-12-08
+date: 2018-12-28
 parent: "Securing Drill"
 ---  
 
-Drill uses ZooKeeper for dynamic service discovery in a cluster; ZooKeeper uses the Curator framework and Service Discovery recipe to discover services. In addition to discovering services, Drill uses ZooKeeper to store certain cluster-level configuration and query profile information in znodes. A znode is an internal data tree in ZooKeeper that stores coordination- and execution-related information. Each time a Drillbit starts up and establish a new session with Zookeeper (using the Cur [...]
+Drill uses the Curator Framework to interact with ZooKeeper to discover services in a cluster. In addition to discovering services, Drill uses ZooKeeper to store certain cluster-level configuration and query profile information in znodes. A znode is an internal data tree in ZooKeeper that stores coordination- and execution-related information. If information in the znodes is not properly secured, cluster privacy and/or security is compromised.   
 
-ZooKeeper uses ACLs (access control lists) to control access to znodes and secure the information they store. ACLs specify sets of ids and permissions that are associated with the ids. The ZooKeeper ACLs are set such that only the Drillbit process user can access (create, delete, read, write, administer) all the ZooKeeper nodes in a Drill cluster, except for the service discovery znodes. When a Drillbit shuts-down, the ZooKeeper session ends and the znode is removed.  
+Drill allows users to create a custom ACL (access control list) on the znodes to secure data. ACLs specify sets of ids and permissions that are associated with the ids. ZooKeeper uses ACLs to control access to znodes and secure the information they store.   
 
-Prior to Drill 1.15, ZooKeeper ACLs in secure and unsecure clusters were set to [world:all], meaning that all users had create, delete, read, write, and administrator access to the zknodes. Starting in Drill 1.15, ACLs in unsecure clusters are set to [world:all]. ACLs in secure clusters are set to [authid: all], which provides only the authenticated user that created the znode with full access. Discovery znodes (znodes with the list of Drillbits) have an additional ACL set to [world:read [...]
+Prior to Drill 1.15, ZooKeeper ACLs in secure and unsecure clusters were set to [world:all], meaning that all users had create, delete, read, write, and administrator access to the zknodes. Starting in Drill 1.15, ACLs in unsecure clusters are set to [world:all]. ACLs in secure clusters are set to [authid: all], which provides only the authenticated user that created the znode with full access. Discovery znodes (znodes with the list of Drillbits) have an additional ACL set to [world:read [...]
 
+- View the [drill-override-example.conf](https://github.com/apache/drill/blob/master/distribution/src/resources/drill-override-example.conf) file to see example ACL configurations.
+
+  
 ##Securing znodes
 Complete the following steps to create a custom ACL and secure znodes:  
 
@@ -22,7 +25,19 @@ Complete the following steps to create a custom ACL and secure znodes:
 4. In `$DRILL_HOME/conf/drill-override.conf`, set `zk.acl_provider` to the `ZKACLProviderTemplate` type.  
 5. Restart Drill.
   
-When you restart Drill, the ACL, as mentioned in your custom class, is applied to the znode created when Drill starts.
+When you restart Drill, the ACL, as mentioned in your custom class, is applied to the znode created when Drill starts.  
+
+***
+**NOTE**  
+Existing ACLs for persistent znodes will not be affected if a Drillbit is restarted with a different ACL setting. ACLs are applied only at znode creation time, and Drill does not recreate any znode that is already present. If you want to change an ACL for existing znodes, connect to the ZooKeeper server using zkCli and then use option a or b, as described:  
+
+- a) Shutdown Drillbits, delete the persistent znodes, change the ACL settings and then restart the Drillbit   
+
+- b) Manually change the ACLs on the existing znodes to reflect the new ACL settings, using the setAcl command in the zkCli.
+
+For either option to work, an authenticated connection between the zkCli and ZooKeeper Server must be established. 
+
+***
 
 For additional information, refer to:  
 
diff --git a/_docs/sql-reference/sql-functions/030-date-time-functions-and-arithmetic.md b/_docs/sql-reference/sql-functions/030-date-time-functions-and-arithmetic.md
index 177b493..708d08e 100644
--- a/_docs/sql-reference/sql-functions/030-date-time-functions-and-arithmetic.md
+++ b/_docs/sql-reference/sql-functions/030-date-time-functions-and-arithmetic.md
@@ -1,6 +1,6 @@
 ---
 title: "Date/Time Functions and Arithmetic"
-date: 2018-12-27
+date: 2018-12-28
 parent: "SQL Functions"
 ---
 
@@ -570,63 +570,65 @@ SELECT UNIX_TIMESTAMP('2015-05-29 08:18:53.0', 'yyyy-MM-dd HH:mm:ss.SSS') FROM (
 Adds an interval of time, in the given time units, to the date expression.   
 
 ###TIMESTAMPADD Syntax  
-TIMESTAMPADD(*time\_unit,interval,date\_expression*)  
+TIMESTAMPADD(*time\_unit,interval, keyword date\_expression*)  
 
 ###TIMESTAMPADD Usage Notes  
-- A date expressions is in the format YYYY-MM-DD.
-- Supports the following time units: Day, Week, Month, Quarter, Year
+- *Keyword* is the type of *date\_expression*: date, time, or timestamp
+- Supports date, time, and timestamp values in the following formats:
+	- Date format: YYYY-MM-DD
+	- Time format: HH:MI:SS
+	- Timestamp format: YYYY-MM-DD HH:MI:SS
+- Supports the following time units: Nanosecond, Microsecond, Second, Minute, Hour, Day, Month, Year, Week, Quarter 
 - Drill uses the unit of time to infer the return type.
 - You can include the SQL_TSI_ prefix with the any of the supported time units, as shown: 
   
-		SELECT TIMESTAMPADD(SQL_TSI_YEAR,3,'1982-05-06');  
+		SELECT TIMESTAMPADD(SQL_TSI_MINUTE,5,TIME '05:05:05');
+		+-----------+
+		|  EXPR$0   |
+		+-----------+
+		| 05:10:05  |
+		+-----------+  
 
 
 ###TIMESTAMPADD Examples   
 
 Add three years to the given date:  
 
-	SELECT TIMESTAMPADD(YEAR,3,'1982-05-06');
+	SELECT TIMESTAMPADD(YEAR,3,DATE '1982-05-06');
 	+------------------------+
 	|         EXPR$0         |
 	+------------------------+
 	| 1985-05-06 00:00:00.0  |
 	+------------------------+   
- 
-Add one quarter of a year (3 months) to the given date:  
- 
-	SELECT TIMESTAMPADD(QUARTER,1,'1982-05-06');
-	+------------------------+
-	|         EXPR$0         |
-	+------------------------+
-	| 1982-08-06 00:00:00.0  |
-	+------------------------+    
 
-Add three months to the given date:  
+Add one quarter of a year (3 months) to the given date:   
 
-	SELECT TIMESTAMPADD(MONTH,3,'1982-05-06');
+	SELECT TIMESTAMPADD(QUARTER,1,DATE'1982-05-06');
 	+------------------------+
 	|         EXPR$0         |
 	+------------------------+
 	| 1982-08-06 00:00:00.0  |
-	+------------------------+  
+	+------------------------+    
+ 
+Add 225 seconds to the given time: 
 
-Add three weeks to the given date:  
+	SELECT TIMESTAMPADD(SECOND,225,TIME '02:02:02');
+	+-----------+
+	|  EXPR$0   |
+	+-----------+
+	| 02:05:47  |
+	+-----------+  
 
-	SELECT TIMESTAMPADD(WEEK,3,'1982-05-06');
-	+------------------------+
-	|         EXPR$0         |
-	+------------------------+
-	| 1982-05-27 00:00:00.0  |
-	+------------------------+  
+Add 5555500000 microseconds to the given timestamp value:
+
+SELECT TIMESTAMPADD(MICROSECOND,5555500000, TIMESTAMP '2003-02-01 12:05:35');
++--------------------------+
+|          EXPR$0          |
++--------------------------+
+| 2003-02-01 12:26:35.532  |
++--------------------------+
 
-Add three days to the given date:  
 
-	SELECT TIMESTAMPADD(DAY,3,'1982-05-06');
-	+------------------------+
-	|         EXPR$0         |
-	+------------------------+
-	| 1982-05-09 00:00:00.0  |
-	+------------------------+
 
 ##TIMESTAMPDIFF  
 Calculates an interval of time, in the given time units, by subtracting *datetime\_expression1* from *datetime\_expression2* (*datetime\_expression2* − *datetime\_expression1*).