You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kudu.apache.org by ab...@apache.org on 2020/09/04 13:25:19 UTC
[kudu] branch branch-1.13.x updated (dcb9b71 -> f052657)
This is an automated email from the ASF dual-hosted git repository.
abukor pushed a change to branch branch-1.13.x
in repository https://gitbox.apache.org/repos/asf/kudu.git.
from dcb9b71 KUDU-3185 Add table owner to web UI and tools
new 931a4a5 Add release notes for 1.13.0
new 1e688c7 Fix RAT errors
new f052657 Bump version to 1.13.0
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
build-support/release/rat_exclude_files.txt | 1 +
docs/release_notes.adoc | 131 ++++++++++++++++++++-
examples/java/collectl/pom.xml | 2 +-
examples/java/insert-loadgen/pom.xml | 2 +-
examples/java/java-example/pom.xml | 2 +-
.../kudu/client/RowResultIteratorBenchmark.java | 19 ++-
version.txt | 2 +-
7 files changed, 149 insertions(+), 10 deletions(-)
[kudu] 01/03: Add release notes for 1.13.0
Posted by ab...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
abukor pushed a commit to branch branch-1.13.x
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit 931a4a5f053528a985483677e84fb37d54deeeb4
Author: Attila Bukor <ab...@apache.org>
AuthorDate: Thu Sep 3 14:15:10 2020 +0200
Add release notes for 1.13.0
Change-Id: Iac2a0ae740c3dabc5e7d8b7ef53312924c6e3532
Reviewed-on: http://gerrit.cloudera.org:8080/16410
Tested-by: Kudu Jenkins
Reviewed-by: Greg Solovyev <gs...@cloudera.com>
Reviewed-by: Alexey Serbin <as...@cloudera.com>
Reviewed-by: Grant Henke <gr...@apache.org>
---
docs/release_notes.adoc | 131 ++++++++++++++++++++++++++++++++++++++++++++++--
1 file changed, 126 insertions(+), 5 deletions(-)
diff --git a/docs/release_notes.adoc b/docs/release_notes.adoc
index f23e070..044d389 100644
--- a/docs/release_notes.adoc
+++ b/docs/release_notes.adoc
@@ -31,27 +31,134 @@
[[rn_1.13.0_upgrade_notes]]
== Upgrade Notes
-
-[[rn_1.13.0_obsoletions]]
-== Obsoletions
-
+* The Sentry integration has been removed and the Ranger integration should now
+ be used in its place for fine-grained authorization.
[[rn_1.13.0_deprecations]]
== Deprecations
-Support for Python 2.x and Python 3.4 and earlier is deprecated and may be removed in the next minor release.
+* Support for Python 2.x and Python 3.4 and earlier is deprecated and may be
+ removed in the next minor release.
+* The `kudu-mapreduce` integration has been deprecated and may be removed in the
+ next minor release. Similar functionality and capabilities now exist via the
+ Apache Spark, Apache Hive, Apache Impala, and Apache NiFi integrations.
[[rn_1.13.0_new_features]]
== New features
+* Added table ownership support. All newly created tables are automatically
+ owned by the user creating them. It is also possible to change the owner by
+ altering the table. You can also assign privileges to table owners via Apache
+ Ranger (see link:https://issues.apache.org/jira/browse/KUDU-3090[KUDU-3090]).
+* An experimental feature is added to Kudu that allows it to automatically
+ rebalance tablet replicas among tablet servers. The background task can be
+ enabled by setting the `--auto_rebalancing_enabled` flag on the Kudu masters.
+ Before starting auto-rebalancing on an existing cluster, the CLI rebalancer
+ tool should be run first (see
+ link:https://issues.apache.org/jira/browse/KUDU-2780[KUDU-2780]).
+* Bloom filter column predicate pushdown has been added to allow optimized
+ execution of filters which match on a set of column values with a
+ false-positive rate. Support for Impala queries utilizing Bloom filter
+ predicate is available yielding performance improvements of 19% to 30% in TPC-H
+ benchmarks and around 41% improvement for distributed joins across large
+ tables. Support for Spark is not yet available. (see
+ link:https://issues.apache.org/jira/browse/KUDU-2483[KUDU-2483]).
+* AArch64-based (ARM) architectures are now supported including published Docker
+ images.
+* The Java client now supports the columnar row format returned from the server
+ transparently. Using this format can reduce the server CPU and size of the
+ request over the network for scans. The columnar format can be enabled via the
+ setRowDataFormat() method on the KuduScanner.
+* An experimental feature that can be enabled by setting the
+ `--enable_workload_score_for_perf_improvement_ops` prioritizes flushing and
+ compacting hot tablets.
[[rn_1.13.0_improvements]]
== Optimizations and improvements
+* Hive metastore synchronization now supports Hive 3 and later.
+* The Spark KuduContext accumulator metrics now track operation counts per table
+ instead of cumulatively for all tables.
+* The `kudu local_replica delete` CLI tool now accepts multiple tablet
+ identifiers. Along with the newly added `--ignore_nonexistent` flag, this
+ helps with scripting scenarios when removing multiple tablet replicas from a
+ particular Tablet Server.
+* Both Master’s and Tablet Server’s web UI now displays the name for a service
+ thread pool group at the `/threadz` page
+* Introduced `queue_overflow_rejections_` metrics for both Masters and Tablet
+ Servers: number of RPC requests of a particular type dropped due to RPC
+ service queue overflow.
+* Introduced a CoDel-like queue control mechanism for the apply queue. This
+ helps to avoid accumulating too many write requests and timing them out in
+ case of seek-bound workloads (e.g., uniform random inserts). The newly
+ introduced queue control mechanism is disabled by default. To enable it, set
+ the `--tablet_apply_pool_overload_threshold_ms` Tablet Server’s flag to
+ appropriate value, e.g. 250 (see
+ link:https://issues.apache.org/jira/browse/KUDU-1587[KUDU-1587]).
+* Operation accumulators in Spark KuduContext are now tracked on a per-table
+ basis.
+* Java client’s error collector can be resized (see
+ link:https://issues.apache.org/jira/browse/KUDU-1422[KUDU-1422]).
+* Calls to the Kudu master server are now drastically reduced when using scan
+ tokens. Previously deserializing a scan token would result in a GetTableSchema
+ request and potentially a GetTableLocations request. Now the table schema and
+ location information is serialized into the scan token itself avoiding the
+ need for any requests to the master when processing them.
+* The default size of Master’s RPC queue is now 100 (it was 50 in earlier
+ releases). This is to optimize for use cases where a Kudu cluster has many
+ clients working concurrently.
+* Masters now have an option to cache table location responses. This is
+ targeted for Kudu clusters which have many clients working concurrently. By
+ default, the caching of table location responses is disabled. To enable table
+ location caching, set the proper capacity of the table location cache using
+ Master’s `--table_locations_cache_capacity_mb` flag (setting to 0 disables the
+ caching). Up to 17% of improvement is observed in GetTableLocations request
+ rate when enabling the caching.
+* Removed lock contention on Raft consensus lock in Tablet Servers while
+ processing a write request. This helps to avoid RPC queue overflows when
+ handling concurrent write requests to the same tablet from multiple clients
+ (see link:https://issues.apache.org/jira/browse/KUDU-2727[KUDU-2727]).
+* Master’s performance for handling concurrent GetTableSchema requests has been
+ improved. End-to-end tests indicated up to 15% improvement in sustained
+ request rate for high concurrency scenarios.
+* Kudu servers now use protobuf Arena objects to perform all RPC
+ request/response-related memory allocations. This gives a boost for overall
+ RPC performance, and with further optimization the result request rate
+ was increased significantly for certain methods. For example, the result request
+ rate increased up to 25% for Master’s GetTabletLocations() RPC in case of
+ highly concurrent scenarios (see
+ link:https://issues.apache.org/jira/browse/KUDU-636[KUDU-636]).
+* Tablet Servers now use protobuf Arena for allocating Raft-related runtime
+ structures. This results in substantial reduction of CPU cycles used and
+ increases write throughput (see
+ link:https://issues.apache.org/jira/browse/KUDU-636[KUDU-636]).
+* Tablet Servers now use protobuf Arena for allocating EncodedKeys to reduce
+ allocator contention and improve memory locality (see
+ link:https://issues.apache.org/jira/browse/KUDU-636[KUDU-636]).
+* Bloom filter predicate evaluation for scans can be computationally expensive.
+ A heuristic has been added that verifies rejection rate of the supplied Bloom
+ filter predicate below which the Bloom filter predicate is automatically
+ disabled. This helped reduce regression observed with Bloom filter predicate
+ in TPC-H benchmark query #9 (see
+ link:https://issues.apache.org/jira/browse/KUDU-3140[KUDU-3140]).
+* Improved scan performance of dictionary and plain-encoded string columns by
+ avoiding copying them (see
+ link:https://issues.apache.org/jira/browse/KUDU-2844[KUDU-2844]).
+* Improved maintenance manager's heuristics to prioritize larger memstores
+ (see link:https://issues.apache.org/jira/browse/KUDU-3180[KUDU-3180]).
+* Spark client's KuduReadOptions now supports setting a snapshot timestamp for
+ repeatable reads with READ_AT_SNAPSHOT consistency mode (see
+ link:https://issues.apache.org/jira/browse/KUDU-3177[KUDU-3177]).
[[rn_1.13.0_fixed_issues]]
== Fixed Issues
+* Kudu scans now honor location assignments when multiple tablet servers are
+ co-located with the client.
+* Fixed a bug that caused IllegalArgumentException to be thrown when trying to
+ create a predicate for a DATE column in Kudu Java client (see
+ link:https://issues.apache.org/jira/browse/KUDU-3152[KUDU-3152]).
+* Fixed a potential race when multiple RPCs work on the same scanner object.
[[rn_1.13.0_wire_compatibility]]
== Wire Protocol compatibility
@@ -104,6 +211,20 @@ documentation.
[[rn_1.13.0_contributors]]
== Contributors
+Kudu 1.13.0 includes contributions from 22 people, including 9 first-time
+contributors:
+
+* Jim Apple
+* Kevin J McCarthy
+* Li Zhiming
+* Mahesh Reddy
+* Romain Rigaux
+* RuiChen
+* Shuping Zhou
+* ningw
+* wenjie
+
+
[[resources_and_next_steps]]
== Resources
[kudu] 02/03: Fix RAT errors
Posted by ab...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
abukor pushed a commit to branch branch-1.13.x
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit 1e688c7ef21bbec9afe295c485e8dec427295716
Author: Attila Bukor <ab...@apache.org>
AuthorDate: Thu Sep 3 23:05:05 2020 +0200
Fix RAT errors
RowResultIteratorBenchmark didn't have a license and
gutil/atomicops-internals-arm64.h has a 3-clause BSD license that has to
be excluded.
Change-Id: I95fbc790cde0d325b7bef0c2424b04bef3682303
Reviewed-on: http://gerrit.cloudera.org:8080/16413
Tested-by: Kudu Jenkins
Reviewed-by: Grant Henke <gr...@apache.org>
---
build-support/release/rat_exclude_files.txt | 1 +
.../kudu/client/RowResultIteratorBenchmark.java | 19 ++++++++++++++++++-
2 files changed, 19 insertions(+), 1 deletion(-)
diff --git a/build-support/release/rat_exclude_files.txt b/build-support/release/rat_exclude_files.txt
index ca6c4f3..6d33ef3 100644
--- a/build-support/release/rat_exclude_files.txt
+++ b/build-support/release/rat_exclude_files.txt
@@ -50,6 +50,7 @@ python/version.txt
src/kudu/client/symbols.map
src/kudu/gutil/arm_instruction_set_select.h
src/kudu/gutil/atomic_refcount.h
+src/kudu/gutil/atomicops-internals-arm64.h
src/kudu/gutil/atomicops-internals-macosx.h
src/kudu/gutil/atomicops-internals-tsan.h
src/kudu/gutil/atomicops.h
diff --git a/java/kudu-client/src/jmh/java/org/apache/kudu/client/RowResultIteratorBenchmark.java b/java/kudu-client/src/jmh/java/org/apache/kudu/client/RowResultIteratorBenchmark.java
index 4ede40c..1c37820 100644
--- a/java/kudu-client/src/jmh/java/org/apache/kudu/client/RowResultIteratorBenchmark.java
+++ b/java/kudu-client/src/jmh/java/org/apache/kudu/client/RowResultIteratorBenchmark.java
@@ -1,3 +1,20 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
package org.apache.kudu.client;
import java.util.ArrayList;
@@ -181,4 +198,4 @@ public class RowResultIteratorBenchmark {
System.arraycopy(data, 0, b, offset, data.length);
return data.length;
}
-}
\ No newline at end of file
+}
[kudu] 03/03: Bump version to 1.13.0
Posted by ab...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
abukor pushed a commit to branch branch-1.13.x
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit f0526578aef16a91f6c74de80419738a9c45acf9
Author: Attila Bukor <ab...@apache.org>
AuthorDate: Fri Sep 4 11:16:13 2020 +0200
Bump version to 1.13.0
Change-Id: I3c56fe015213f2f83a057d2f2b73d41cc69825b0
Reviewed-on: http://gerrit.cloudera.org:8080/16416
Tested-by: Kudu Jenkins
Reviewed-by: Grant Henke <gr...@apache.org>
---
examples/java/collectl/pom.xml | 2 +-
examples/java/insert-loadgen/pom.xml | 2 +-
examples/java/java-example/pom.xml | 2 +-
version.txt | 2 +-
4 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/examples/java/collectl/pom.xml b/examples/java/collectl/pom.xml
index 9576530..ad2a2bf 100644
--- a/examples/java/collectl/pom.xml
+++ b/examples/java/collectl/pom.xml
@@ -28,7 +28,7 @@
<name>Kudu collectl Example</name>
<properties>
- <kudu-version>1.12.0</kudu-version>
+ <kudu-version>1.13.0</kudu-version>
</properties>
<build>
diff --git a/examples/java/insert-loadgen/pom.xml b/examples/java/insert-loadgen/pom.xml
index 8733acd..4f9614c 100644
--- a/examples/java/insert-loadgen/pom.xml
+++ b/examples/java/insert-loadgen/pom.xml
@@ -28,7 +28,7 @@
<name>Random Insert Load Generator for Kudu</name>
<properties>
- <kudu-version>1.12.0</kudu-version>
+ <kudu-version>1.13.0</kudu-version>
</properties>
<build>
diff --git a/examples/java/java-example/pom.xml b/examples/java/java-example/pom.xml
index 007fef0..b50644e 100644
--- a/examples/java/java-example/pom.xml
+++ b/examples/java/java-example/pom.xml
@@ -28,7 +28,7 @@
<name>Kudu Java Client Examples</name>
<properties>
- <kudu-version>1.12.0</kudu-version>
+ <kudu-version>1.13.0</kudu-version>
</properties>
<build>
diff --git a/version.txt b/version.txt
index e9172f4..feaae22 100644
--- a/version.txt
+++ b/version.txt
@@ -1 +1 @@
-1.13.0-SNAPSHOT
+1.13.0