You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kudu.apache.org by da...@apache.org on 2017/05/23 01:19:05 UTC
kudu git commit: docs: revise security doc based on KUDU-1875 and add
release notes for 1.4
Repository: kudu
Updated Branches:
refs/heads/master 952222b52 -> d354693c1
docs: revise security doc based on KUDU-1875 and
add release notes for 1.4
Change-Id: I2e386203f5ed3ef66e2ec136e67738b8c7eb8b1a
Reviewed-on: http://gerrit.cloudera.org:8080/6922
Reviewed-by: Dan Burkert <da...@apache.org>
Tested-by: Dan Burkert <da...@apache.org>
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/d354693c
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/d354693c
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/d354693c
Branch: refs/heads/master
Commit: d354693c17b802fa34ef9362a8bf9e748b887a6b
Parents: 952222b
Author: hahao <ha...@cloudera.com>
Authored: Thu May 18 12:20:02 2017 -0700
Committer: Dan Burkert <da...@apache.org>
Committed: Tue May 23 01:18:47 2017 +0000
----------------------------------------------------------------------
docs/prior_release_notes.adoc | 174 +++++++++++++++++++++++++++++++++++
docs/release_notes.adoc | 179 +++++--------------------------------
docs/security.adoc | 23 +++--
3 files changed, 212 insertions(+), 164 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/kudu/blob/d354693c/docs/prior_release_notes.adoc
----------------------------------------------------------------------
diff --git a/docs/prior_release_notes.adoc b/docs/prior_release_notes.adoc
index ba1971e..f1ea542 100644
--- a/docs/prior_release_notes.adoc
+++ b/docs/prior_release_notes.adoc
@@ -37,6 +37,180 @@ reproduced on this page. Please consult the
link:http://kudu.apache.org/releases/[documentation of the appropriate release]
for a list of known issues and limitations.
+[[rn_1.3.0]]
+== Release notes specific to 1.3.0
+
+[[rn_1.3.0_new_features]]
+== New features
+
+* Kudu 1.3 adds support for strong authentication based on Kerberos. This optional feature
+ allows users to authenticate themselves using Kerberos tickets, and also provides
+ mutual authentication of servers using Kerberos credentials stored in keytabs. This
+ feature is optional, but recommended for deployments requiring security.
+
+* Kudu 1.3 adds support for encryption of data on the network using Transport Layer Security
+ (TLS). Kudu will now use TLS to encrypt all network traffic between clients and servers as
+ well as any internal traffic among servers, with the exception of traffic determined to
+ be within a localhost network connection. Encryption is enabled by default whenever it can
+ be determined that both the client and server support the feature.
+
+* Kudu 1.3 adds coarse-grained service-level authorization of access to the cluster.
+ The operator may set up lists of permitted users who may act as administrators and
+ as clients of the cluster. Combined with the strong authentication feature described
+ above, this can enable a secure environment for some use cases. Note that fine-grained
+ access control (e.g. table-level or column-level) is not yet supported.
+
+* Kudu 1.3 adds a background task to tablet servers which removes historical versions of
+ data which have fallen behind the configured data retention time. This reduces disk space
+ usage in all workloads, but particularly in those with a higher volume of updates or
+ upserts.
+
+* Kudu now incorporates Google Breakpad, a library which writes crash reports in
+ the case of a server crash. These reports can be found within the configured log directory,
+ and can be useful during bug diagnosis.
+
+
+== Optimizations and improvements
+
+* Kudu servers will now change the file permissions of data directories and contained
+ data files based on a new configuration flag `--umask`. As a result, after upgrading,
+ permissions on disk may be more restrictive than in previous versions. The new default
+ configuration improves data security.
+
+* Kudu's web UI will now redact strings which may include sensitive user data. For example,
+ the monitoring page which shows in-progress scans no longer includes the scanner predicate
+ values. The tracing and RPC diagnostics endpoints no longer include contents of RPCs which
+ may include table data.
+
+* By default, Kudu now reserves 1% of each configured data volume as free space. If a volume
+ is seen to have less than 1% of disk space free, Kudu will stop writing to that volume
+ to avoid completely filling up the disk.
+
+* The default encoding for numeric columns (int, float, and double) has been changed
+ to `BIT_SHUFFLE`. The default encoding for binary and string columns has been
+ changed to `DICT_ENCODING`. Dictionary encoding automatically falls back to the old
+ default (`PLAIN`) when cardinality is too high to be effectively encoded.
++
+These new defaults match the default behavior of other storage mechanisms such as
+ Apache Parquet and are likely to perform better out of the box.
+
+* Kudu now uses `LZ4` compression when writing its Write Ahead Log (WAL). This improves
+ write performance and stability for many use cases.
+
+* Kudu now uses `LZ4` compression when writing delta files. This can improve both
+ read and write performance as well as save substantial disk usage, especially
+ for workloads involving a high number of updates or upserts containing compressible
+ data.
+
+* The Kudu API now supports the ability to express `IS NULL` and `IS NOT NULL` predicates
+ on scanners. The Spark DataSource integration will take advantage of these new
+ predicates when possible.
+
+* Both {cpp} and Java clients have been optimized to prune partitions more effectively
+ when performing scans using the `IN (...)` predicate.
+
+* The exception messages produced by the Java client are now truncated to a maximum length
+ of 32KB.
+
+
+[[rn_1.3.0_fixed_issues]]
+== Fixed Issues
+
+* link:https://issues.apache.org/jira/browse/KUDU-1893[KUDU-1893]
+ Fixed a critical bug in which wrong results would be returned when evaluating
+ predicates applied to columns added using the `ALTER TABLE` operation.
+
+* link:https://issues.apache.org/jira/browse/KUDU-1905[KUDU-1905]
+ Fixed a crash after inserting a row sharing a primary key with a recently-deleted
+ row in tables where the primary key is comprised of all of the columns.
+
+* link:https://issues.apache.org/jira/browse/KUDU-1899[KUDU-1899]
+ Fixed a crash after inserting a row with an empty string as the single-column
+ primary key.
+
+* link:https://issues.apache.org/jira/browse/KUDU-1904[KUDU-1904]
+ Fixed a potential crash when performing random reads against a column using RLE
+ encoding and containing long runs of NULL values.
+
+* link:https://issues.apache.org/jira/browse/KUDU-1853[KUDU-1853]
+ Fixed an issue where disk space could be leaked on servers which experienced an error
+ during the process of copying tablet data from another server.
+
+* link:https://issues.apache.org/jira/browse/KUDU-1856[KUDU-1856]
+ Fixed an issue in which disk space could be leaked by Kudu servers storing data on
+ partitions using the XFS file system. Any leaked disk space will be automatically
+ recovered upon upgrade.
+
+* link:https://issues.apache.org/jira/browse/KUDU-1888[KUDU-1888],
+ link:https://issues.apache.org/jira/browse/KUDU-1906[KUDU-1906]
+ Fixed multiple issues in the Java client where operation callbacks would never be
+ triggered, causing the client to hang.
+
+
+[[rn_1.3.0_wire_compatibility]]
+== Wire Protocol compatibility
+
+Kudu 1.3.0 is wire-compatible with previous versions of Kudu:
+
+* Kudu 1.3 clients may connect to servers running Kudu 1.0. If the client uses features
+ that are not available on the target server, an error will be returned.
+* Kudu 1.0 clients may connect to servers running Kudu 1.3 with the exception of the
+ below-mentioned restrictions regarding secure clusters.
+* Rolling upgrade between Kudu 1.2 and Kudu 1.3 servers is believed to be possible
+ though has not been sufficiently tested. Users are encouraged to shut down all nodes
+ in the cluster, upgrade the software, and then restart the daemons on the new version.
+
+The authentication features newly introduced in Kudu 1.3 place the following limitations
+on wire compatibility with older versions:
+
+* If a Kudu 1.3 cluster is configured with authentication or encryption set to "required",
+ older clients will be unable to connect.
+* If a Kudu 1.3 cluster is configured with authentication and encryption set to "optional"
+ or "disabled", older clients will still be able to connect.
+
+
+[[rn_1.3.0_incompatible_changes]]
+== Incompatible Changes in Kudu 1.3.0
+
+* Due to storage format changes in Kudu 1.3, downgrade from Kudu 1.3 to earlier versions
+ is not supported. After upgrading to Kudu 1.3, attempting to restart with an earlier
+ version will result in an error.
+
+* In order to support running MapReduce and Spark jobs on secure clusters, these
+ frameworks now connect to the cluster at job submission time to retrieve authentication
+ credentials which can later be used by the tasks to be spawned. This means that
+ the process submitting jobs to Kudu clusters must have direct access to that cluster.
+
+* The embedded web servers in Kudu processes now specify the `X-Frame-Options: DENY` HTTP
+ header which prevents embedding Kudu web pages in HTML `iframe` elements.
+
+[[rn_1.3.0_client_compatibility]]
+=== Client Library Compatibility
+
+* The Kudu 1.3 Java client library is API- and ABI-compatible with Kudu 1.2. Applications
+ written against Kudu 1.2 will compile and run against the Kudu 1.3 client library and
+ vice-versa, unless one of the following newly added APIs is used:
+** `[Async]KuduClient.exportAuthenticationCredentials(...)` (unstable API)
+** `[Async]KuduClient.importAuthenticationCredentials(...)` (unstable API)
+** `[Async]KuduClient.getMasterAddressesAsString()`
+** `KuduPredicate.newIsNotNullPredicate()`
+** `KuduPredicate.newIsNullPredicate()`
+
+* The Kudu 1.3 {cpp} client is API- and ABI-forward-compatible with Kudu 1.2.
+ Applications written and compiled against the Kudu 1.2 client library will run without
+ modification against the Kudu 1.3 client library. Applications written and compiled
+ against the Kudu 1.3 client library will run without modification against the Kudu 1.2
+ client library unless they use one of the following new APIs:
+** `kudu::DisableOpenSSLInitialization()`
+** `KuduClientBuilder::import_authentication_credentials(...)`
+** `KuduClient::ExportAuthenticationCredentials(...)`
+** `KuduClient::NewIsNotNullPredicate(...)`
+** `KuduClient::NewIsNullPredicate(...)`
+
+* The Kudu 1.3 Python client is API-compatible with Kudu 1.2. Applications
+ written against Kudu 1.2 will continue to run against the Kudu 1.3 client
+ and vice-versa.
+
[[rn_1.2.0]]
== Release notes specific to 1.2.0
http://git-wip-us.apache.org/repos/asf/kudu/blob/d354693c/docs/release_notes.adoc
----------------------------------------------------------------------
diff --git a/docs/release_notes.adoc b/docs/release_notes.adoc
index bba1c4f..ee19f36 100644
--- a/docs/release_notes.adoc
+++ b/docs/release_notes.adoc
@@ -16,7 +16,7 @@
// under the License.
[[release_notes]]
-= Apache Kudu 1.3.0 Release Notes
+= Apache Kudu 1.4.0 Release Notes
:author: Kudu Team
:imagesdir: ./images
@@ -28,181 +28,44 @@
:sectlinks:
:experimental:
-[[rn_1.3.0]]
+[[rn_1.4.0]]
-[[rn_1.3.0_new_features]]
+[[rn_1.4.0_new_features]]
== New features
-* Kudu 1.3 adds support for strong authentication based on Kerberos. This optional feature
- allows users to authenticate themselves using Kerberos tickets, and also provides
- mutual authentication of servers using Kerberos credentials stored in keytabs. This
- feature is optional, but recommended for deployments requiring security.
-
-* Kudu 1.3 adds support for encryption of data on the network using Transport Layer Security
- (TLS). Kudu will now use TLS to encrypt all network traffic between clients and servers as
- well as any internal traffic among servers, with the exception of traffic determined to
- be within a localhost network connection. Encryption is enabled by default whenever it can
- be determined that both the client and server support the feature.
-
-* Kudu 1.3 adds coarse-grained service-level authorization of access to the cluster.
- The operator may set up lists of permitted users who may act as administrators and
- as clients of the cluster. Combined with the strong authentication feature described
- above, this can enable a secure environment for some use cases. Note that fine-grained
- access control (e.g. table-level or column-level) is not yet supported.
-
-* Kudu 1.3 adds a background task to tablet servers which removes historical versions of
- data which have fallen behind the configured data retention time. This reduces disk space
- usage in all workloads, but particularly in those with a higher volume of updates or
- upserts.
-
-* Kudu now incorporates Google Breakpad, a library which writes crash reports in
- the case of a server crash. These reports can be found within the configured log directory,
- and can be useful during bug diagnosis.
-
== Optimizations and improvements
-* Kudu servers will now change the file permissions of data directories and contained
- data files based on a new configuration flag `--umask`. As a result, after upgrading,
- permissions on disk may be more restrictive than in previous versions. The new default
- configuration improves data security.
-
-* Kudu's web UI will now redact strings which may include sensitive user data. For example,
- the monitoring page which shows in-progress scans no longer includes the scanner predicate
- values. The tracing and RPC diagnostics endpoints no longer include contents of RPCs which
- may include table data.
-
-* By default, Kudu now reserves 1% of each configured data volume as free space. If a volume
- is seen to have less than 1% of disk space free, Kudu will stop writing to that volume
- to avoid completely filling up the disk.
-
-* The default encoding for numeric columns (int, float, and double) has been changed
- to `BIT_SHUFFLE`. The default encoding for binary and string columns has been
- changed to `DICT_ENCODING`. Dictionary encoding automatically falls back to the old
- default (`PLAIN`) when cardinality is too high to be effectively encoded.
+* Kudu servers, by default, will now only allow unencrypted or unauthenticated connections
+ from trusted subnets, which are private networks (127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,
+ 192.168.0.0/16,169.254.0.0/16) and local subnets of all local network interfaces.
+ Unencrypted or unauthenticated connections from publicly routable IPs will be rejected,
+ even if encryption and authentication are not configured.
+
-These new defaults match the default behavior of other storage mechanisms such as
- Apache Parquet and are likely to perform better out of the box.
-
-* Kudu now uses `LZ4` compression when writing its Write Ahead Log (WAL). This improves
- write performance and stability for many use cases.
-
-* Kudu now uses `LZ4` compression when writing delta files. This can improve both
- read and write performance as well as save substantial disk usage, especially
- for workloads involving a high number of updates or upserts containing compressible
- data.
-
-* The Kudu API now supports the ability to express `IS NULL` and `IS NOT NULL` predicates
- on scanners. The Spark DataSource integration will take advantage of these new
- predicates when possible.
-
-* Both {cpp} and Java clients have been optimized to prune partitions more effectively
- when performing scans using the `IN (...)` predicate.
-
-* The exception messages produced by the Java client are now truncated to a maximum length
- of 32KB.
-
-[[rn_1.3.0_fixed_issues]]
+The trusted subnets can be configured using the `--trusted_subnets` flag, which can be set
+ to IP blocks represented in CIDR notation separated by comma. Set it to '0.0.0.0/0' to
+ allow unauthenticated connections from all remote IP addresses. However, if network access
+ is not otherwise restricted by a firewall, malicious users may be able to gain unauthorized
+ access. This can be mitigated if authentication and encryption are configured to be
+ required.
+
+[[rn_1.4.0_fixed_issues]]
== Fixed Issues
-* link:https://issues.apache.org/jira/browse/KUDU-1893[KUDU-1893]
- Fixed a critical bug in which wrong results would be returned when evaluating
- predicates applied to columns added using the `ALTER TABLE` operation.
-
-* link:https://issues.apache.org/jira/browse/KUDU-1905[KUDU-1905]
- Fixed a crash after inserting a row sharing a primary key with a recently-deleted
- row in tables where the primary key is comprised of all of the columns.
-
-* link:https://issues.apache.org/jira/browse/KUDU-1899[KUDU-1899]
- Fixed a crash after inserting a row with an empty string as the single-column
- primary key.
-
-* link:https://issues.apache.org/jira/browse/KUDU-1904[KUDU-1904]
- Fixed a potential crash when performing random reads against a column using RLE
- encoding and containing long runs of NULL values.
-
-* link:https://issues.apache.org/jira/browse/KUDU-1853[KUDU-1853]
- Fixed an issue where disk space could be leaked on servers which experienced an error
- during the process of copying tablet data from another server.
-
-* link:https://issues.apache.org/jira/browse/KUDU-1856[KUDU-1856]
- Fixed an issue in which disk space could be leaked by Kudu servers storing data on
- partitions using the XFS file system. Any leaked disk space will be automatically
- recovered upon upgrade.
-
-* link:https://issues.apache.org/jira/browse/KUDU-1888[KUDU-1888],
- link:https://issues.apache.org/jira/browse/KUDU-1906[KUDU-1906]
- Fixed multiple issues in the Java client where operation callbacks would never be
- triggered, causing the client to hang.
-
-
-[[rn_1.3.0_wire_compatibility]]
+[[rn_1.4.0_wire_compatibility]]
== Wire Protocol compatibility
-Kudu 1.3.0 is wire-compatible with previous versions of Kudu:
-* Kudu 1.3 clients may connect to servers running Kudu 1.0. If the client uses features
- that are not available on the target server, an error will be returned.
-* Kudu 1.0 clients may connect to servers running Kudu 1.3 with the exception of the
- below-mentioned restrictions regarding secure clusters.
-* Rolling upgrade between Kudu 1.2 and Kudu 1.3 servers is believed to be possible
- though has not been sufficiently tested. Users are encouraged to shut down all nodes
- in the cluster, upgrade the software, and then restart the daemons on the new version.
-
-The authentication features newly introduced in Kudu 1.3 place the following limitations
-on wire compatibility with older versions:
-
-* If a Kudu 1.3 cluster is configured with authentication or encryption set to "required",
- older clients will be unable to connect.
-* If a Kudu 1.3 cluster is configured with authentication and encryption set to "optional"
- or "disabled", older clients will still be able to connect.
-
-
-[[rn_1.3.0_incompatible_changes]]
+[[rn_1.4.0_incompatible_changes]]
== Incompatible Changes in Kudu 1.3.0
-* Due to storage format changes in Kudu 1.3, downgrade from Kudu 1.3 to earlier versions
- is not supported. After upgrading to Kudu 1.3, attempting to restart with an earlier
- version will result in an error.
-* In order to support running MapReduce and Spark jobs on secure clusters, these
- frameworks now connect to the cluster at job submission time to retrieve authentication
- credentials which can later be used by the tasks to be spawned. This means that
- the process submitting jobs to Kudu clusters must have direct access to that cluster.
-
-* The embedded web servers in Kudu processes now specify the `X-Frame-Options: DENY` HTTP
- header which prevents embedding Kudu web pages in HTML `iframe` elements.
-
-[[rn_1.3.0_client_compatibility]]
+[[rn_1.4.0_client_compatibility]]
=== Client Library Compatibility
-* The Kudu 1.3 Java client library is API- and ABI-compatible with Kudu 1.2. Applications
- written against Kudu 1.2 will compile and run against the Kudu 1.3 client library and
- vice-versa, unless one of the following newly added APIs is used:
-** `[Async]KuduClient.exportAuthenticationCredentials(...)` (unstable API)
-** `[Async]KuduClient.importAuthenticationCredentials(...)` (unstable API)
-** `[Async]KuduClient.getMasterAddressesAsString()`
-** `KuduPredicate.newIsNotNullPredicate()`
-** `KuduPredicate.newIsNullPredicate()`
-
-* The Kudu 1.3 {cpp} client is API- and ABI-forward-compatible with Kudu 1.2.
- Applications written and compiled against the Kudu 1.2 client library will run without
- modification against the Kudu 1.3 client library. Applications written and compiled
- against the Kudu 1.3 client library will run without modification against the Kudu 1.2
- client library unless they use one of the following new APIs:
-** `kudu::DisableOpenSSLInitialization()`
-** `KuduClientBuilder::import_authentication_credentials(...)`
-** `KuduClient::ExportAuthenticationCredentials(...)`
-** `KuduClient::NewIsNotNullPredicate(...)`
-** `KuduClient::NewIsNullPredicate(...)`
-
-* The Kudu 1.3 Python client is API-compatible with Kudu 1.2. Applications
- written against Kudu 1.2 will continue to run against the Kudu 1.3 client
- and vice-versa.
-
-
-[[rn_1.3.0_known_issues]]
+
+[[rn_1.4.0_known_issues]]
== Known Issues and Limitations
http://git-wip-us.apache.org/repos/asf/kudu/blob/d354693c/docs/security.adoc
----------------------------------------------------------------------
diff --git a/docs/security.adoc b/docs/security.adoc
index 1b8cbf7..d55a347 100644
--- a/docs/security.adoc
+++ b/docs/security.adoc
@@ -46,9 +46,19 @@ Authentication can be configured on Kudu servers using the
`--rpc-authentication` flag, which can be set to `required`, `optional`, or
`disabled`. By default, the flag is set to `optional`. When `required`, Kudu
will reject connections from clients and servers who lack authentication
-credentials. When `optional`, Kudu will attempt to use strong authentication,
-but will allow unauthenticated connections. When `disabled`, Kudu will only
-allow unauthenticated connections.
+credentials. When `optional`, Kudu will attempt to use strong authentication.
+When `disabled` or strong authentication fails for 'optional', by default Kudu
+will only allow unauthenticated connections from trusted subnets, which are
+private networks (127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,
+169.254.0.0/16) and local subnets of all local network interfaces. Unauthenticated
+connections from publicly routable IPs will be rejected.
+
+The trusted subnets can be configured using the `--trusted_subnets` flag,
+which can be set to IP blocks in CIDR notation separated by comma. Set it to
+'0.0.0.0/0' to allow unauthenticated connections from all remote IP addresses.
+However, if network access is not otherwise restricted by a firewall,
+malicious users may be able to gain unauthorized access. This can be mitigated
+if authentication is configured to be required.
WARNING: When the `--rpc-authentication` flag is set to `optional`,
the cluster does not prevent access from unauthenticated users. To secure a
@@ -106,9 +116,10 @@ to be encrypted with TLS.
Encryption can be configured on Kudu servers using the `--rpc-encryption` flag,
which can be set to `required`, `optional`, or `disabled`. By default, the flag
is set to `optional`. When `required`, Kudu will reject unencrypted connections.
-When `optional`, Kudu will attempt to use encryption, but will allow unencrypted
-connections. When `disabled`, Kudu will never use encryption. To secure a
-cluster, use `--rpc-encryption=required`.
+When `optional`, Kudu will attempt to use encryption. Same as authentication,
+when `disabled` or encryption fails for `optional`, Kudu will only allow
+unencrypted connections from trusted subnets and reject any unencrypted connections
+from publicly routable IPs. To secure a cluster, use `--rpc-encryption=required`.
NOTE: Kudu will automatically turn off encryption on local loopback connections,
since traffic from these connections is never exposed externally. This allows