You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@drill.apache.org by br...@apache.org on 2017/07/31 20:59:57 UTC
drill git commit: edits to docs for 1.11
Repository: drill
Updated Branches:
refs/heads/gh-pages 9392e3174 -> 7cab70c9f
edits to docs for 1.11
Project: http://git-wip-us.apache.org/repos/asf/drill/repo
Commit: http://git-wip-us.apache.org/repos/asf/drill/commit/7cab70c9
Tree: http://git-wip-us.apache.org/repos/asf/drill/tree/7cab70c9
Diff: http://git-wip-us.apache.org/repos/asf/drill/diff/7cab70c9
Branch: refs/heads/gh-pages
Commit: 7cab70c9fa618fabe1a217d0bb00ec84f5a442f7
Parents: 9392e31
Author: Bridget Bevens <bb...@maprtech.com>
Authored: Mon Jul 31 13:58:53 2017 -0700
Committer: Bridget Bevens <bb...@maprtech.com>
Committed: Mon Jul 31 13:58:53 2017 -0700
----------------------------------------------------------------------
.../010-securing-drill-introduction.md | 4 +-
.../020-secure-communication-paths.md | 15 +++----
.../070-configuring-user-authentication.md | 4 +-
...090-configuring-kerberos-auththentication.md | 43 ++++++++++++++++----
blog/_posts/2017-07-31-drill-1.11-released.md | 30 ++++++++------
5 files changed, 64 insertions(+), 32 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/drill/blob/7cab70c9/_docs/configure-drill/securing-drill/010-securing-drill-introduction.md
----------------------------------------------------------------------
diff --git a/_docs/configure-drill/securing-drill/010-securing-drill-introduction.md b/_docs/configure-drill/securing-drill/010-securing-drill-introduction.md
index 2701088..3d6846d 100644
--- a/_docs/configure-drill/securing-drill/010-securing-drill-introduction.md
+++ b/_docs/configure-drill/securing-drill/010-securing-drill-introduction.md
@@ -1,6 +1,6 @@
---
title: "Securing Drill Introduction"
-date: 2017-03-16 01:47:58 UTC
+date: 2017-07-31 20:58:54 UTC
parent: "Securing Drill"
---
@@ -20,7 +20,7 @@ Before connecting to a data source, you can configure Drill security features an
- [Configuring User Impersonation]({{site.baseurl}}/docs/configuring-user-impersonation/)
- [Configuring Inbound Impersonation]({{site.baseurl}}/docs/configuring-inbound-impersonation/)
- [Configuring User Impersonation with Hive Authorization]({{site.baseurl}}/docs/configuring-user-impersonation-with-hive-authorization/)
-- **Encryption** - Drill does not support encryption as of Drill 1.10.
+- **Encryption** - Drill supports client-to-drillbit encryption in Drill 1.11.
http://git-wip-us.apache.org/repos/asf/drill/blob/7cab70c9/_docs/configure-drill/securing-drill/020-secure-communication-paths.md
----------------------------------------------------------------------
diff --git a/_docs/configure-drill/securing-drill/020-secure-communication-paths.md b/_docs/configure-drill/securing-drill/020-secure-communication-paths.md
index dfe7f4b..4811705 100644
--- a/_docs/configure-drill/securing-drill/020-secure-communication-paths.md
+++ b/_docs/configure-drill/securing-drill/020-secure-communication-paths.md
@@ -1,6 +1,6 @@
---
title: "Secure Communication Paths"
-date: 2017-03-17 22:19:49 UTC
+date: 2017-07-31 20:58:55 UTC
parent: "Securing Drill"
---
As illustrated in the following figure, Drill 1.10 features five secure communication paths. Security features for each communication path are described their respective sections.
@@ -25,15 +25,16 @@ The Web Console and REST API clients are web clients. Web clients can:
---
**Note**
-Impersonation and authorization are available through the web clients only when authentication is enabled. Otherwise, the user identity is unknown.
+Impersonation, authorization, and encryption are available through the web clients only when authentication and encryption are enabled. Otherwise, the user identity is unknown and encryption is not used.
---
-| Security Capability | Description | Reference |
-|---------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------|
-| Authentication | Users authenticate to a drillbit using a username and password form authenticator. By default, authentication is disabled. | [Configuring Web Console and REST API Security]({{site.baseurl}}/docs/configuring-web-console-and-rest-api-security) |
-| Impersonation | Drill acts on behalf of the user on the session. This is usually the connection user (or the user that authenticates). This user can impersonate another user, which is allowed if the connection user is authorized to impersonate the target user based on the inbound impersonation policies (USER role). By default, impersonation is disabled. | [Configuring User Impersonation]({{site.baseurl}}/docs/configuring-user-impersonation/#impersonation-and-views) and [Configuring Inbound Impersonation]({{site.baseurl}}/docs/configuring-inbound-impersonation) |
-| Authorization | Queries execute on behalf of the web user. Users and administrators have different navigation bars. Various tabs are shown based on privileges. For example, only administrators can see the Storage tab and create/read/update/delete storage plugin configuration. | [Configuring Web Console and REST API Security]({{site.baseurl}}/docs/configuring-web-console-and-rest-api-security) |
+| Security Capability | Description | Reference |
+|---------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| Authentication | Users authenticate to a drillbit using a username and password form authenticator. By default, authentication is disabled. | [Configuring Web Console and REST API Security]({{site.baseurl}}/docs/configuring-web-console-and-rest-api-security) |
+| Encryption | Drill 1.11 supports encryption between a Drill client and Drillbit using the Kerberos mechanism over a Java SASL framework. Encrypting the client-to-drillbit communication pathway ensures data integrity and prevents data tampering as well as snooping. On the server side, enable encryption in the drill-override.conf file with the security.user.encryption.sasl.enabled parameter. On the client side, use the sasl_encrypt parameter in the connection string. | [Configuring Kerberos Authentication]({{site.baseurl}}/docs/configuring-kerberos-authentication/) |
+| Impersonation | Drill acts on behalf of the user on the session. This is usually the connection user (or the user that authenticates). This user can impersonate another user, which is allowed if the connection user is authorized to impersonate the target user based on the inbound impersonation policies (USER role). By default, impersonation is disabled. | [Configuring User Impersonation]({{site.baseurl}}/docs/configuring-user-impersonation/#impersonation-and-views) and [Configuring Inbound Impersonation]({{site.baseurl}}/docs/configuring-inbound-impersonation) |
+| Authorization | Queries execute on behalf of the web user. Users and administrators have different navigation bars. Various tabs are shown based on privileges. For example, only administrators can see the Storage tab and create/read/update/delete storage plugin configuration. | [Configuring Web Console and REST API Security]({{site.baseurl}}/docs/configuring-web-console-and-rest-api-security) | |
## Java and C++ Client to Drillbit
http://git-wip-us.apache.org/repos/asf/drill/blob/7cab70c9/_docs/configure-drill/securing-drill/070-configuring-user-authentication.md
----------------------------------------------------------------------
diff --git a/_docs/configure-drill/securing-drill/070-configuring-user-authentication.md b/_docs/configure-drill/securing-drill/070-configuring-user-authentication.md
index 27aa9c1..fb7a6ea 100644
--- a/_docs/configure-drill/securing-drill/070-configuring-user-authentication.md
+++ b/_docs/configure-drill/securing-drill/070-configuring-user-authentication.md
@@ -1,11 +1,11 @@
---
title: "Configuring User Authentication"
-date: 2017-05-17 01:38:49 UTC
+date: 2017-07-31 20:58:56 UTC
parent: "Securing Drill"
---
Authentication is the process of establishing confidence of authenticity. A Drill client user is authenticated when a drillbit process running in a Drill cluster confirms the identity it is presented with. Drill 1.10 supports several authentication mechanisms through which users can prove their identity before accessing cluster data:
-* **Kerberos** - New in Drill 1.10. See [Configuring Kerberos Authentication]({{site.baseurl}}/docs/configuring-kerberos-authentication/).
+* **Kerberos** - Featuring Drill client to Drillbit encryption in Drill 1.11. See [Configuring Kerberos Authentication]({{site.baseurl}}/docs/configuring-kerberos-authentication/).
* **Plain** [also known as basic authentication (auth), which is username and password-based authentication, through the Linux Pluggable Authentication Module (PAM)] - See [Configuring Plain Authentication]({{site.baseurl}}/docs/configuring-plain-authentication/).
* **Custom authenticators** - See [Creating Custom Authenticators]({{site.baseurl}}/docs/creating-custom-authenticators).
http://git-wip-us.apache.org/repos/asf/drill/blob/7cab70c9/_docs/configure-drill/securing-drill/090-configuring-kerberos-auththentication.md
----------------------------------------------------------------------
diff --git a/_docs/configure-drill/securing-drill/090-configuring-kerberos-auththentication.md b/_docs/configure-drill/securing-drill/090-configuring-kerberos-auththentication.md
index 172146a..3261444 100644
--- a/_docs/configure-drill/securing-drill/090-configuring-kerberos-auththentication.md
+++ b/_docs/configure-drill/securing-drill/090-configuring-kerberos-auththentication.md
@@ -1,9 +1,9 @@
---
title: "Configuring Kerberos Authentication"
-date: 2017-05-17 01:38:52 UTC
+date: 2017-07-31 20:58:57 UTC
parent: "Securing Drill"
---
-In release 1.10 Drill supports Kerberos v5 network security authentication. To use Kerberos with Drill and establish connectivity, use the JDBC driver packaged with Drill 1.10.
+In release 1.11 Drill supports Kerberos v5 network security authentication and client-to-drillbit encryption. To use Kerberos with Drill and establish connectivity, use the JDBC driver packaged with Drill 1.11.
Kerberos allows trusted hosts to prove their identity over a network to an information system. A Kerberos *realm* is unique authentication domain. A centralized *key distribution center (KDC)* coordinates authentication between a clients and servers. Clients and servers obtain and use tickets from the KDC using a special *keytab* file to communicate with the KDC and prove their identity to gain access to a drillbit. Administrators must create *principal* (user or server) identities and passwords to ensure the secure exchange of mutual authentication information passed to and from the drillbit.
@@ -14,7 +14,7 @@ See the [MIT Kerberos](http://web.mit.edu/kerberos/ "MIT Kerberos") documentatio
## Prerequisites
-The required Kerberos (JDBC) plugin is part of the 1.10 Drill package. To use it, you must have a working Kerberos infrastructure, which Drill does not provide. You must be working in a Linux-based or Windows Active Directory (AD) Kerberos environment with secure clusters and have a Drill server configured for Kerberos. See [Enabling Authentication]({{site.baseurl}}/docs/configuring-kerberos-authentication/#enabling-authentication).
+The required Kerberos (JDBC) plugin is part of the 1.11 Drill package. To use it, you must have a working Kerberos infrastructure, which Drill does not provide. You must be working in a Linux-based or Windows Active Directory (AD) Kerberos environment with secure clusters and have a Drill server configured for Kerberos. See [Enabling Authentication]({{site.baseurl}}/docs/configuring-kerberos-authentication/#enabling-authentication).
## Client Authentication Process
@@ -38,9 +38,17 @@ This section provides a high-level overview of the Kerberos client authenticatio
For Kerberos server authentication information, see the [MIT Kerberos](http://web.mit.edu/kerberos/ "MIT Kerberos") administration documentation.
-## Enabling Authentication
+## Enabling Authentication and Encryption
During startup, a drillbit service must authenticate. At runtime, Drill uses the keytab file. Trust is based on the keytab file; its secrets are shared with the KDC. The drillbit service also uses this keytab credential to validate service tickets from clients. Based on this information, the drillbit determines whether the client’s identity can be verified to use its service.
+To enable encryption,set the following parameters in the `drill-override.conf` file (as shown in the second example below):
+
+
+- `security.user.encryption.sasl.enabled` to true. This parameter determines if the drillbit is enabled for encryption. Only Drill 1.11 drillbits support encryption.
+
+- `security.user.encryption.sasl.max_wrapped_size`. This parameter specifies the maximum size of encoded buffer in bytes (maxbuffer parameter in sasl) that the client and server will receive. Using this the SASL framework exposes maximum buffer size that the wrap function will accept, so that Drill client/server can chop the Outbound RPC message with the size. The maximum recommended value is 16777215. The default is 65536.
+
+
@@ -97,11 +105,13 @@ During startup, a drillbit service must authenticate. At runtime, Drill uses the
auth.principal:“drill/<clustername>@<REALM>.COM”,
auth.keytab:“/etc/drill/conf/drill.keytab”
}
- security.user.auth: {
- enabled: true,
- packages += "org.apache.drill.exec.rpc.user.security",
- impl: "pam",
- pam_profiles: ["sudo", "login"]
+ security.user: {
+ auth.enabled: true,
+ auth.packages += "org.apache.drill.exec.rpc.user.security",
+ auth.impl: "pam",
+ auth.pam_profiles: ["sudo", "login"],
+ encryption.sasl.enabled: true,
+ encryption.sasl.max_wrapped_size: 65536,
}
}
@@ -141,11 +151,26 @@ The following table lists configuration options for connection URLs. See the Con
| auth | Authentication mechanism. The value is deduced if not specified. Kerberos if principal is provided. Plain if a user and password is provided. A Drill client can also explicitly specify a particular authentication mechanism to use using this parameter. For example, <auth=kerberos> for Kerberos along with service_name, service_host or principal and <auth=plain> for the Plain authentication with username and password. | Optional | The preference order is Kerberos and Plain. |
| principal | Drillbit service principal. The format of the principal is primary/instance@realm. For Kerberos, the Drill service principal is derived if the value is not provided using this configuration. service_name (primary) and service_host (instance) are used to generate a valid principal. Since the ticket or keytab contains the realm information, the realm is optional. | Optional | |
| keytab | For Kerberos, if the client chooses to authenticate using a keytab rather than a ticket, set the keytab parameter to the location of the keytab file. The client principal must be provided through the user parameter. A Kerberos ticket is used as the default credential (It is assumed to be present on client-side. The Drill client does not generate the required credentials.) | Optional | |
+| sasl_encrypt | When set to true, ensures that a client connects to a server with encryption capabilities. For example, Drill 1.11 drillbits, which support client-to-drillbit encryption. | Optional | false |
| service_name | Primary name of the drillbit service principal. | Optional | drill |
| service_host | Instance name of the drillbit service principal. | Optional | Since this value is usually the hostname of the node where a drillbit is running, the default value is the drillbit hostname is provided either through ZooKeeper or through a direct connection string. |
| realm | Kerberos realm name for the drillbit service principal. The ticket or keytab contains the realm information. | Optional | |
+### Client Encryption
+A client can specify that it requires a server with encryption capabilities only by setting the `sasl_encrypt` connection parameter to **true**. If the cluster to which client is connecting has encryption disabled, the client will fail to connect to that server.
+
+ drill.exec {
+ security: {
+ user.auth.enabled: true,
+ auth.mechanisms: ["KERBEROS"],
+ auth.principal: "drill/serverhostname@REALM.COM",
+ auth.keytab: "/etc/drill/conf/drill.keytab",
+ user.encryption.sasl.enabled: true
+ }
+ }
+
+
### Connection URL Examples
The following five examples show the JDBC connection URL that the embedded JDBC client uses for Kerberos authentication. The first section, Example of a Simple Connection URL, includes a simple connection string and the second section, Examples of Connection URLs Used with Previously Generated TGTs, includes examples to use with previously generated TGTs.
http://git-wip-us.apache.org/repos/asf/drill/blob/7cab70c9/blog/_posts/2017-07-31-drill-1.11-released.md
----------------------------------------------------------------------
diff --git a/blog/_posts/2017-07-31-drill-1.11-released.md b/blog/_posts/2017-07-31-drill-1.11-released.md
index c379ff4..3628bd1 100644
--- a/blog/_posts/2017-07-31-drill-1.11-released.md
+++ b/blog/_posts/2017-07-31-drill-1.11-released.md
@@ -12,16 +12,16 @@ The release provides the following bug fixes and improvements:
## Cryptography-Related Functions (DRILL-5634)
Drill provides the following cryptographic-related functions:
-
-- aes_encrypt()
-- aes_decrypt()
-- md5()
-- sha()
-- sha1()
-- sha2()
-
+
+- aes_encrypt()
+- aes_decrypt()
+- md5()
+- sha()
+- sha1()
+- sha2()
+
## Spill to Disk for Hash Aggregate Operator (DRILL-5457)
-The Hash aggregate operator can spill data to disk in cases where the operation exceeds the set memory limit.
+The Hash aggregate operator can spill data to disk in cases where the operation exceeds the set memory limit. Note that you may need to increase the default value of the `planner.memory.max_query_memory_per_node` option due to insufficient memory.
## Format Plugin Support for PCAP Files (DRILL-5432)
A “pcap” format plugin enables Drill to read PCAP files. You must add the “pcap” format to the dfs storage plugin configuration, as shown:
@@ -40,7 +40,7 @@ The `store.parquet.writer.use_single_fs_block` option enables Drill to write a P
The `drill.exec.profiles.store.inmemory` option enables Drill to store query profiles in memory instead of writing the query profiles to disk. The `drill.exec.profiles.store.capacity` option sets the maximum number of most recent profiles to retain in memory.
## Configurable CTAS Directory and File Permissions Option (DRILL-5391)
-You can use the `exec.persistent_table.umask` configuration option, at the system or session level, to modify permissions on directories and files that result from running the CTAS command. By default, the option is set to 002, which sets the default directory permissions to 775 and default file permissions to -664.
+You can use the `exec.persistent_table.umask` configuration option, at the system or session level, to modify permissions on directories and files that result from running the CTAS command. By default, the option is set to 002, which sets the default directory permissions to 775 and default file permissions to 664.
## Support for Network Encryption (DRILL-4335)
Drill can use SASL to support network encryption between the Drill client and drillbits, and also between drillbits.
@@ -48,8 +48,14 @@ Drill can use SASL to support network encryption between the Drill client and dr
## Metadata file Stores Relative Paths (DRILL-3867)
Drill now stores the relative path in the metadata file (versus the absolute path), which enables you to move partitioned Parquet directories from one location in DFS to another without having to rebuild the Parquet metadata files; the metadata remains valid in the new location.
-## Support for ANSI_QUOTES (DRILL-3510)
-In addition to back ticks, the SQL parser in Drill can use double quotes as identifier quotes. Use the `planner.parser.quoting_identifiers` configuration option, at the system or session level, to set the type of identifier quotes that the SQL parser in Drill uses.
+## Support for Additional Quoting Identifiers (DRILL-3510)
+In addition to back ticks, the SQL parser in Drill can use double quotes and square brackets as identifier quotes. Use the `planner.parser.quoting_identifiers` configuration option, at the system or session level, to set the type of identifier quotes that the SQL parser in Drill uses, as shown:
+
+ ALTER SESSION SET planner.parser.quoting_identifiers = '"';
+ ALTER SESSION SET planner.parser.quoting_identifiers = '[';
+ ALTER SESSION SET planner.parser.quoting_identifiers = '`';
+
+The default setting is back ticks. The quoting identifier used in queries must match the setting. If you use another type of quoting identifier, Drill returns an error.
You can find a complete list of JIRAs resolved in the 1.11.0 release [here](https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12313820&version=12339943).