You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kudu.apache.org by ad...@apache.org on 2018/03/06 23:10:18 UTC
[1/4] kudu git commit: [docs] Improvements to multi-master migration
doc
Repository: kudu
Updated Branches:
refs/heads/master 9f93e97a6 -> c8724c615
[docs] Improvements to multi-master migration doc
- Add extra reminder to run as the Kudu user.
- Note that the copy_from_remote command requires authenticating
to the remote service as the Kudu user.
- Note that the workflow can be used to migrate 2->3 masters by
making straightforward adjustments to the procedure.
- Move steps for verifying the migration was successful to a new
section so they are more noticeable.
Change-Id: I77ef796f8b35729871ef8ddf2b635989278c2ebc
Reviewed-on: http://gerrit.cloudera.org:8080/9466
Reviewed-by: Adar Dembo <ad...@cloudera.com>
Tested-by: Will Berkeley <wd...@gmail.com>
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/e7c7d4a1
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/e7c7d4a1
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/e7c7d4a1
Branch: refs/heads/master
Commit: e7c7d4a1203d4869713ec2d912dadb3f43c60bdb
Parents: 9f93e97
Author: Will Berkeley <wd...@apache.org>
Authored: Thu Mar 1 11:31:37 2018 -0800
Committer: Will Berkeley <wd...@gmail.com>
Committed: Tue Mar 6 22:58:15 2018 +0000
----------------------------------------------------------------------
docs/administration.adoc | 84 +++++++++++++++++++++++--------------------
1 file changed, 46 insertions(+), 38 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/kudu/blob/e7c7d4a1/docs/administration.adoc
----------------------------------------------------------------------
diff --git a/docs/administration.adoc b/docs/administration.adoc
index ab78aa1..6f64d7a 100644
--- a/docs/administration.adoc
+++ b/docs/administration.adoc
@@ -210,13 +210,14 @@ The frequency with which metrics are dumped to the diagnostics log is configured
For high availability and to avoid a single point of failure, Kudu clusters should be created with
multiple masters. Many Kudu clusters were created with just a single master, either for simplicity
or because Kudu multi-master support was still experimental at the time. This workflow demonstrates
-how to migrate to a multi-master configuration.
+how to migrate to a multi-master configuration. It can also be used to migrate from two masters to
+three, with straightforward modifications.
-WARNING: The workflow is unsafe for adding new masters to an existing multi-master configuration.
-Do not use it for that purpose.
+WARNING: The workflow is unsafe for adding new masters to an existing configuration that already has
+three or more masters. Do not use it for that purpose.
-WARNING: All of the command line steps below should be executed as the Kudu UNIX user, typically
-`kudu`.
+WARNING: All of the command line steps below should be executed as the Kudu UNIX user. The example
+commands assume the Kudu Unix user is `kudu`, which is typical.
WARNING: The workflow presupposes at least basic familiarity with Kudu configuration management. If
using Cloudera Manager (CM), the workflow also presupposes familiarity with it.
@@ -230,11 +231,11 @@ using Cloudera Manager (CM), the workflow also presupposes familiarity with it.
configurations are recommended; they can tolerate one or two failures respectively.
. Perform the following preparatory steps for the existing master:
-* Identify and record the directory where the master's data lives. If using Kudu system packages,
- the default value is /var/lib/kudu/master, but it may be customized via the `fs_wal_dir` and
- `fs_data_dirs` configuration parameters. Please note if you've set `fs_data_dirs` to some directories
- other than the value of `fs_wal_dir`, it should be explicitly included in every command below where
- `fs_wal_dir` is also included. For more information on configuring these directories, see the
+* Identify and record the directories where the master's write-ahead log (WAL) and data live. If
+ using Kudu system packages, their default locations are /var/lib/kudu/master, but they may be
+ customized via the `fs_wal_dir` and `fs_data_dirs` configuration parameters. The commands below
+ assume that `fs_wal_dir` is /data/kudu/master/wal and `fs_data_dirs` is /data/kudu/master/data.
+ Your configuration may differ. For more information on configuring these directories, see the
link:configuration.html#directory_configuration[Kudu Configuration docs].
* Identify and record the port the master is using for RPCs. The default port value is 7051, but it
may have been customized using the `rpc_bind_addresses` configuration parameter.
@@ -242,7 +243,7 @@ using Cloudera Manager (CM), the workflow also presupposes familiarity with it.
+
[source,bash]
----
-$ kudu fs dump uuid --fs_wal_dir=<master_wal_dir> [--fs_data_dirs=<master_data_dirs>] 2>/dev/null
+$ sudo -u kudu kudu fs dump uuid --fs_wal_dir=<master_wal_dir> [--fs_data_dirs=<master_data_dirs>] 2>/dev/null
----
master_data_dir:: existing master's previously recorded data directory
+
@@ -250,7 +251,7 @@ master_data_dir:: existing master's previously recorded data directory
Example::
+
----
-$ kudu fs dump uuid --fs_wal_dir=/var/lib/kudu/master 2>/dev/null
+$ sudo -u kudu kudu fs dump uuid --fs_wal_dir=/data/kudu/master/wal --fs_data_dirs=/data/kudu/master/data 2>/dev/null
4aab798a69e94fab8d77069edff28ce0
----
+
@@ -297,8 +298,8 @@ the migration section for updating HMS.
+
[source,bash]
----
-$ kudu fs format --fs_wal_dir=<master_wal_dir> [--fs_data_dirs=<master_data_dirs>]
-$ kudu fs dump uuid --fs_wal_dir=<master_wal_dir> [--fs_data_dirs=<master_data_dirs>] 2>/dev/null
+$ sudo -u kudu kudu fs format --fs_wal_dir=<master_wal_dir> [--fs_data_dirs=<master_data_dirs>]
+$ sudo -u kudu kudu fs dump uuid --fs_wal_dir=<master_wal_dir> [--fs_data_dirs=<master_data_dirs>] 2>/dev/null
----
+
master_data_dir:: new master's previously recorded data directory
@@ -307,8 +308,8 @@ master_data_dir:: new master's previously recorded data directory
Example::
+
----
-$ kudu fs format --fs_wal_dir=/var/lib/kudu/master
-$ kudu fs dump uuid --fs_wal_dir=/var/lib/kudu/master 2>/dev/null
+$ sudo -u kudu kudu fs format --fs_wal_dir=/data/kudu/master/wal --fs_data_dirs=/data/kudu/master/data
+$ sudo -u kudu kudu fs dump uuid --fs_wal_dir=/data/kudu/master/wal --fs_data_dirs=/data/kudu/master/data 2>/dev/null
f5624e05f40649b79a757629a69d061e
----
@@ -322,7 +323,7 @@ f5624e05f40649b79a757629a69d061e
+
[source,bash]
----
-$ kudu local_replica cmeta rewrite_raft_config --fs_wal_dir=<master_wal_dir> [--fs_data_dirs=<master_data_dirs>] <tablet_id> <all_masters>
+$ sudo -u kudu kudu local_replica cmeta rewrite_raft_config --fs_wal_dir=<master_wal_dir> [--fs_data_dirs=<master_data_dirs>] <tablet_id> <all_masters>
----
+
master_data_dir:: existing master's previously recorded data directory
@@ -337,7 +338,7 @@ port::: master's previously recorded RPC port number
Example::
+
----
-$ kudu local_replica cmeta rewrite_raft_config --fs_wal_dir=/var/lib/kudu/master 00000000000000000000000000000000 4aab798a69e94fab8d77069edff28ce0:master-1:7051 f5624e05f40649b79a757629a69d061e:master-2:7051 988d8ac6530f426cbe180be5ba52033d:master-3:7051
+$ sudo -u kudu kudu local_replica cmeta rewrite_raft_config --fs_wal_dir=/data/kudu/master/wal --fs_data_dirs=/data/kudu/master/data 00000000000000000000000000000000 4aab798a69e94fab8d77069edff28ce0:master-1:7051 f5624e05f40649b79a757629a69d061e:master-2:7051 988d8ac6530f426cbe180be5ba52033d:master-3:7051
----
. Modify the value of the `master_addresses` configuration parameter for both existing master and new masters.
@@ -348,11 +349,14 @@ port:: master's previously recorded RPC port number
. Start the existing master.
. Copy the master data to each new master with the following command, executed on each new master
- machine:
+ machine.
++
+WARNING: If your Kudu cluster is secure, in addition to running as the Kudu UNIX user, you must
+ authenticate as the Kudu service user prior to running this command.
+
[source,bash]
----
-$ kudu local_replica copy_from_remote --fs_wal_dir=<master_wal_dir> [--fs_data_dirs=<master_data_dirs>] <tablet_id> <existing_master>
+$ sudo -u kudu kudu local_replica copy_from_remote --fs_wal_dir=<master_wal_dir> [--fs_data_dirs=<master_data_dirs>] <tablet_id> <existing_master>
----
+
master_data_dir:: new master's previously recorded data directory
@@ -366,7 +370,7 @@ port::: existing master's previously recorded RPC port number
Example::
+
----
-$ kudu local_replica copy_from_remote --fs_wal_dir=/var/lib/kudu/master 00000000000000000000000000000000 master-1:7051
+$ sudo -u kudu kudu local_replica copy_from_remote --fs_wal_dir=/data/kudu/master/wal --fs_data_dirs=/data/kudu/master/data 00000000000000000000000000000000 master-1:7051
----
. Start all of the new masters.
@@ -401,9 +405,9 @@ INVALIDATE METADATA;
----
+
+==== Verify the migration was successful
-Congratulations, the cluster has now been migrated to multiple masters! To verify that all masters
-are working properly, consider performing the following sanity checks:
+To verify that all masters are working properly, perform the following sanity checks:
* Using a browser, visit each master's web UI. Look at the /masters page. All of the masters should
be listed there with one master in the LEADER role and the others in the FOLLOWER role. The
@@ -468,7 +472,7 @@ WARNING: All of the command line steps below should be executed as the Kudu UNIX
+
[source,bash]
----
-$ kudu fs dump uuid --fs_wal_dir=<master_wal_dir> [--fs_data_dirs=<master_data_dirs>] 2>/dev/null
+$ sudo -u kudu kudu fs dump uuid --fs_wal_dir=<master_wal_dir> [--fs_data_dirs=<master_data_dirs>] 2>/dev/null
----
master_data_dir:: live master's previously recorded data directory
+
@@ -476,7 +480,7 @@ master_data_dir:: live master's previously recorded data directory
Example::
+
----
-$ kudu fs dump uuid --fs_wal_dir=/var/lib/kudu/master 2>/dev/null
+$ sudo -u kudu kudu fs dump uuid --fs_wal_dir=/data/kudu/master/wal --fs_data_dirs=/data/kudu/master/data 2>/dev/null
80a82c4b8a9f4c819bab744927ad765c
----
+
@@ -491,7 +495,7 @@ $ kudu fs dump uuid --fs_wal_dir=/var/lib/kudu/master 2>/dev/null
+
[source,bash]
----
-$ kudu local_replica cmeta print_replica_uuids --fs_wal_dir=<master_wal_dir> [--fs_data_dirs=<master_data_dirs>] <tablet_id> 2>/dev/null
+$ sudo -u kudu kudu local_replica cmeta print_replica_uuids --fs_wal_dir=<master_wal_dir> [--fs_data_dirs=<master_data_dirs>] <tablet_id> 2>/dev/null
----
master_data_dir:: reference master's previously recorded data directory
tablet_id:: must be the string `00000000000000000000000000000000`
@@ -500,7 +504,7 @@ tablet_id:: must be the string `00000000000000000000000000000000`
Example::
+
----
-$ kudu local_replica cmeta print_replica_uuids --fs_wal_dir=/var/lib/kudu/master 00000000000000000000000000000000 2>/dev/null
+$ sudo -u kudu kudu local_replica cmeta print_replica_uuids --fs_wal_dir=/data/kudu/master/wal --fs_data_dirs=/data/kudu/master/data 00000000000000000000000000000000 2>/dev/null
80a82c4b8a9f4c819bab744927ad765c 2a73eeee5d47413981d9a1c637cce170 1c3f3094256347528d02ec107466aef3
----
+
@@ -514,7 +518,7 @@ $ kudu local_replica cmeta print_replica_uuids --fs_wal_dir=/var/lib/kudu/master
+
[source,bash]
----
-$ kudu fs format --fs_wal_dir=<master_wal_dir> [--fs_data_dirs=<master_data_dirs>] --uuid=<uuid>
+$ sudo -u kudu kudu fs format --fs_wal_dir=<master_wal_dir> [--fs_data_dirs=<master_data_dirs>] --uuid=<uuid>
----
+
master_data_dir:: replacement master's previously recorded data directory
@@ -524,14 +528,17 @@ uuid:: dead master's previously recorded UUID
Example::
+
----
-$ kudu fs format --fs_wal_dir=/var/lib/kudu/master --uuid=80a82c4b8a9f4c819bab744927ad765c
+$ sudo -u kudu kudu fs format --fs_wal_dir=/data/kudu/master/wal --fs_data_dirs=/data/kudu/master/data --uuid=80a82c4b8a9f4c819bab744927ad765c
----
+
. Copy the master data to the replacement master with the following command:
+
+WARNING: If your Kudu cluster is secure, in addition to running as the Kudu UNIX user, you must
+ authenticate as the Kudu service user prior to running this command.
++
[source,bash]
----
-$ kudu local_replica copy_from_remote --fs_wal_dir=<master_wal_dir> [--fs_data_dirs=<master_data_dirs>] <tablet_id> <reference_master>
+$ sudo -u kudu kudu local_replica copy_from_remote --fs_wal_dir=<master_wal_dir> [--fs_data_dirs=<master_data_dirs>] <tablet_id> <reference_master>
----
+
master_data_dir:: replacement master's previously recorded data directory
@@ -545,7 +552,7 @@ port::: reference master's previously recorded RPC port number
Example::
+
----
-$ kudu local_replica copy_from_remote --fs_wal_dir=/var/lib/kudu/master 00000000000000000000000000000000 master-2:7051
+$ sudo -u kudu kudu local_replica copy_from_remote --fs_wal_dir=/data/kudu/master/wal --fs_data_dirs=/data/kudu/master/data 00000000000000000000000000000000 master-2:7051
----
+
. If using CM, add the replacement Kudu master role now, but do not start it.
@@ -621,8 +628,9 @@ remove any unwanted masters.
. Start all of the tablet servers.
-Congratulations, the masters have now been removed! To verify that all masters are working properly,
-consider performing the following sanity checks:
+==== Verify the migration was successful
+
+To verify that all masters are working properly, perform the following sanity checks:
* Using a browser, visit each master's web UI. Look at the /masters page. All of the masters should
be listed there with one master in the LEADER role and the others in the FOLLOWER role. The
@@ -644,7 +652,7 @@ addresses to be specified:
[source,bash]
----
-$ kudu cluster ksck master-01.example.com,master-02.example.com,master-03.example.com
+$ sudo -u kudu kudu cluster ksck master-01.example.com,master-02.example.com,master-03.example.com
----
To see a full list of the options available with `ksck`, use the `--help` flag.
@@ -694,7 +702,7 @@ be done with the following command:
[source,bash]
----
-$ kudu cluster ksck --checksum_scan --tables IntegrationTestBigLinkedList master-01.example.com,master-02.example.com,master-03.example.com
+$ sudo -u kudu kudu cluster ksck --checksum_scan --tables IntegrationTestBigLinkedList master-01.example.com,master-02.example.com,master-03.example.com
----
[[change_dir_config]]
@@ -744,7 +752,7 @@ UNIX user, typically `kudu`.
+
[source,bash]
----
-$ kudu fs update_dirs --fs_wal_dir=/wals --fs_data_dirs=/data/1,/data/2,/data/3
+$ sudo -u kudu kudu fs update_dirs --fs_wal_dir=/wals --fs_data_dirs=/data/1,/data/2,/data/3
----
+
@@ -864,7 +872,7 @@ diagnosing and fixing the problem is to examine the tablet's state using ksck:
[source,bash]
----
-$ kudu cluster ksck --tablets=e822cab6c0584bc0858219d1539a17e6 master-00,master-01,master-02
+$ sudo -u kudu kudu cluster ksck --tablets=e822cab6c0584bc0858219d1539a17e6 master-00,master-01,master-02
Connected to the Master
Fetched info from all 5 Tablet Servers
Tablet e822cab6c0584bc0858219d1539a17e6 of table 'my_table' is unavailable: 2 replica(s) not RUNNING
@@ -898,7 +906,7 @@ will be rewritten to include only the healthy replica.
[source,bash]
----
-$ kudu remote_replica unsafe_change_config tserver-00:7150 <tablet-id> <tserver-00-uuid>
+$ sudo -u kudu kudu remote_replica unsafe_change_config tserver-00:7150 <tablet-id> <tserver-00-uuid>
----
where `<tablet-id>` is `e822cab6c0584bc0858219d1539a17e6` and
[4/4] kudu git commit: decimal_util: do not produce warnings when
building with gcc
Posted by ad...@apache.org.
decimal_util: do not produce warnings when building with gcc
When I build with gcc5, I get the following warning:
../../src/kudu/util/decimal_util.cc:31:45: warning: ‘no_sanitize’ attribute directive ignored [-Wattributes]
int128_t MaxUnscaledDecimal(int8_t precision) {
This is because gcc doesn't recognize the "no_sanitize" attribute, so let's
deal with this as we've dealt with other sanitizer suppressions: compile it
conditionally using ASAN support as a proxy.
Change-Id: Iac9b145a914d551e3a18d62a0984aad34b95f4dd
Reviewed-on: http://gerrit.cloudera.org:8080/9499
Tested-by: Kudu Jenkins
Reviewed-by: Grant Henke <gr...@gmail.com>
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/c8724c61
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/c8724c61
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/c8724c61
Branch: refs/heads/master
Commit: c8724c61501a0e940fbcda6cc4d13716b65f8a8a
Parents: 940e60f
Author: Adar Dembo <ad...@cloudera.com>
Authored: Mon Mar 5 14:22:30 2018 -0800
Committer: Adar Dembo <ad...@cloudera.com>
Committed: Tue Mar 6 23:09:51 2018 +0000
----------------------------------------------------------------------
src/kudu/gutil/port.h | 13 +++++++++++++
src/kudu/util/decimal_util.cc | 6 ++++--
2 files changed, 17 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/kudu/blob/c8724c61/src/kudu/gutil/port.h
----------------------------------------------------------------------
diff --git a/src/kudu/gutil/port.h b/src/kudu/gutil/port.h
index 900df65..9c66721 100644
--- a/src/kudu/gutil/port.h
+++ b/src/kudu/gutil/port.h
@@ -420,6 +420,19 @@ inline void* memrchr(const void* bytes, int find_char, size_t len) {
#define ATTRIBUTE_NO_SANITIZE_THREAD
#endif
+// Tell UBSAN to ignore a given function completely. There is no
+// __has_feature(undefined_sanitizer) or equivalent, so ASAN support is used as
+// a proxy.
+#if defined(__has_feature)
+# if __has_feature(address_sanitizer)
+# define ATTRIBUTE_NO_SANITIZE_UNDEFINED \
+ __attribute__((no_sanitize("undefined")))
+# endif
+#endif
+#ifndef ATTRIBUTE_NO_SANITIZE_UNDEFINED
+#define ATTRIBUTE_NO_SANITIZE_UNDEFINED
+#endif
+
// Tell UBSAN to ignore integer overflows in a given function. There is no
// __has_feature(undefined_sanitizer) or equivalent, so ASAN support is used as
// a proxy.
http://git-wip-us.apache.org/repos/asf/kudu/blob/c8724c61/src/kudu/util/decimal_util.cc
----------------------------------------------------------------------
diff --git a/src/kudu/util/decimal_util.cc b/src/kudu/util/decimal_util.cc
index d4f7e70..0e04494 100644
--- a/src/kudu/util/decimal_util.cc
+++ b/src/kudu/util/decimal_util.cc
@@ -21,13 +21,15 @@
#include <glog/logging.h>
+#include "kudu/gutil/port.h"
+
namespace kudu {
using std::string;
// Workaround for an ASAN build issue documented here:
// https://bugs.llvm.org/show_bug.cgi?id=16404
-__attribute__((no_sanitize("undefined")))
+ATTRIBUTE_NO_SANITIZE_UNDEFINED
int128_t MaxUnscaledDecimal(int8_t precision) {
DCHECK_GE(precision, kMinDecimalPrecision);
DCHECK_LE(precision, kMaxDecimalPrecision);
@@ -44,7 +46,7 @@ int128_t MinUnscaledDecimal(int8_t precision) {
// Workaround for an ASAN build issue documented here:
// https://bugs.llvm.org/show_bug.cgi?id=16404
-__attribute__((no_sanitize("undefined")))
+ATTRIBUTE_NO_SANITIZE_UNDEFINED
string DecimalToString(int128_t d, int8_t scale) {
// 38 digits, 1 extra leading zero, decimal point,
// and sign are good for 128-bit or smaller decimals.
[3/4] kudu git commit: build: stop generating md5 file
Posted by ad...@apache.org.
build: stop generating md5 file
New guidance from the ASF is to not generate md5 files when producing source
releases. Instead we just generate the sha checksum.
Change-Id: I6b0359d63b660b285c0853cd9b88654a400f14c2
Reviewed-on: http://gerrit.cloudera.org:8080/9504
Tested-by: Kudu Jenkins
Reviewed-by: Dan Burkert <da...@apache.org>
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/940e60f0
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/940e60f0
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/940e60f0
Branch: refs/heads/master
Commit: 940e60f0f844782fa01ba449ce6a38e08f721d06
Parents: 1c1d3ba
Author: Todd Lipcon <to...@apache.org>
Authored: Mon Mar 5 17:19:50 2018 -0800
Committer: Todd Lipcon <to...@apache.org>
Committed: Tue Mar 6 22:59:17 2018 +0000
----------------------------------------------------------------------
build-support/build_source_release.py | 25 +++++++++++--------------
1 file changed, 11 insertions(+), 14 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/kudu/blob/940e60f0/build-support/build_source_release.py
----------------------------------------------------------------------
diff --git a/build-support/build_source_release.py b/build-support/build_source_release.py
index 6d922d4..3006a68 100755
--- a/build-support/build_source_release.py
+++ b/build-support/build_source_release.py
@@ -78,7 +78,7 @@ def create_tarball():
"--prefix=%s/" % artifact_name,
"--output=%s" % tarball_path,
"HEAD"])
- print(Colors.GREEN + "Generated tarball:\t" + Colors.RESET, tarball_path)
+ print(Colors.GREEN + "Generated tarball:\t" + Colors.RESET + tarball_path)
return tarball_path
@@ -118,21 +118,18 @@ def checksum_file(summer, path):
return summer.hexdigest()
-def gen_checksum_files(tarball_path):
+def gen_sha_file(tarball_path):
"""
- Create md5 and sha files of the tarball.
+ Create a sha checksum file of the tarball.
- The output format is compatible with command line tools like 'sha1sum'
- and 'md5sum' so they may be used to verify the checksums.
+ The output format is compatible with command line tools like 'sha1sum' so it
+ can be used to verify the checksum.
"""
- hashes = [(hashlib.sha1, "sha"),
- (hashlib.md5, "md5")]
- for hash_func, extension in hashes:
- digest = checksum_file(hash_func(), tarball_path)
- path = tarball_path + "." + extension
- with open(path, "w") as f:
- f.write("%s\t%s\n" % (digest, os.path.basename(tarball_path)))
- print(Colors.GREEN + ("Generated %s:\t" % extension) + Colors.RESET, path)
+ digest = checksum_file(hashlib.sha1(), tarball_path)
+ path = tarball_path + ".sha"
+ with open(path, "w") as f:
+ f.write("%s\t%s\n" % (digest, os.path.basename(tarball_path)))
+ print(Colors.GREEN + "Generated sha:\t\t" + Colors.RESET + path)
def run_rat(tarball_path):
@@ -182,7 +179,7 @@ def main():
check_repo_not_dirty()
check_no_local_commits()
tarball_path = create_tarball()
- gen_checksum_files(tarball_path)
+ gen_sha_file(tarball_path)
sign_tarball(tarball_path)
run_rat(tarball_path)
[2/4] kudu git commit: build: adjust the number of processors for
various tests
Posted by ad...@apache.org.
build: adjust the number of processors for various tests
Many of our tests are multithreaded and can utilize many cores worth of
CPU during execution. Other tests are somewhat timing-sensitive and
might time out if run in coordination with a stress test, particularly
on a small machine which might only have four CPUs total.
This patch adjusts the core count property for a bunch of tests based on
some empirical analysis using a new 'perf-stat' wrapper capability
provided in run-test.sh. The wrapper produces periodic samples of the
CPU usage for each test. Measurements are written into log files which
can then be analyzed with a command like the following:
for x in build/latest/test-logs/*.perf.txt ; do
echo -n $x
grep task-clock $x | sort -nk2 | tail -1
done | sort -nk3
This resulted in a list of tests along with their peak CPU usage (from
their busiest 1000ms period). I used this as a rough guide to find tests
with inappropriate settings in CMakeLists.txt. The net effect of this
change is that 'ctest -j' might run slightly longer, but that we'll have
fewer flaky tests in such a run configuration because we don't have
stress tests perturbing timing of unrelated non-stress tests.
Change-Id: I922031af69178b98459d0d59287442b692425afa
Reviewed-on: http://gerrit.cloudera.org:8080/9503
Tested-by: Kudu Jenkins
Reviewed-by: Adar Dembo <ad...@cloudera.com>
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/1c1d3ba1
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/1c1d3ba1
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/1c1d3ba1
Branch: refs/heads/master
Commit: 1c1d3ba1cebb6b02cea9603c952931fd8261200a
Parents: e7c7d4a
Author: Todd Lipcon <to...@apache.org>
Authored: Mon Mar 5 16:56:11 2018 -0800
Committer: Todd Lipcon <to...@apache.org>
Committed: Tue Mar 6 22:59:06 2018 +0000
----------------------------------------------------------------------
CMakeLists.txt | 18 ++++++++++
build-support/run-test.sh | 18 ++++++++--
src/kudu/cfile/CMakeLists.txt | 2 +-
src/kudu/client/CMakeLists.txt | 4 +--
src/kudu/clock/CMakeLists.txt | 2 +-
src/kudu/consensus/CMakeLists.txt | 4 +--
src/kudu/integration-tests/CMakeLists.txt | 50 +++++++++++++-------------
src/kudu/mini-cluster/CMakeLists.txt | 2 +-
src/kudu/rpc/CMakeLists.txt | 4 +--
src/kudu/security/CMakeLists.txt | 2 +-
src/kudu/tablet/CMakeLists.txt | 4 +--
src/kudu/tools/CMakeLists.txt | 6 ++--
src/kudu/tserver/CMakeLists.txt | 2 +-
src/kudu/util/CMakeLists.txt | 10 +++---
14 files changed, 80 insertions(+), 48 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/kudu/blob/1c1d3ba1/CMakeLists.txt
----------------------------------------------------------------------
diff --git a/CMakeLists.txt b/CMakeLists.txt
index ac960ff..753da44 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -693,6 +693,24 @@ endfunction()
# parallel on a single machine using ctest, they will still run in
# parallel across separate machines using dist-test.
#
+# PROCESSORS <num processors>
+# RUN_SERIAL true
+# These built-in CMake properties should be used when a test is
+# a heavy consumer of CPU.
+#
+# The PROCESSORS flag allows ctest to ensure that tests are only
+# scheduled in parallel when an appropriate number of cores are
+# available. The default is to assume that a test uses a single core
+# (i.e. equivalent to PROCESSORS 1).
+#
+# RUN_SERIAL ensures that the test does not run in parallel with any
+# other test. This should be used for stress tests which start many
+# threads and attempt to monopolize the machine.
+#
+# In order to determine an appropriate setting, you can run the test
+# with the KUDU_MEASURE_TEST_CPU_CONSUMPTION environment variable
+# set. See build-support/run-test.sh for more details.
+#
# Any other arguments will be passed to set_tests_properties().
function(ADD_KUDU_TEST REL_TEST_NAME)
# Parse out properties for which we have special handling.
http://git-wip-us.apache.org/repos/asf/kudu/blob/1c1d3ba1/build-support/run-test.sh
----------------------------------------------------------------------
diff --git a/build-support/run-test.sh b/build-support/run-test.sh
index b3fc285..c1b7647 100755
--- a/build-support/run-test.sh
+++ b/build-support/run-test.sh
@@ -31,6 +31,12 @@
#
# If KUDU_REPORT_TEST_RESULTS is non-zero, then tests are reported to the
# central test server.
+#
+# If KUDU_MEASURE_TEST_CPU_CONSUMPTION is non-zero, then tests will be wrapped
+# in a 'perf stat' command which measures the CPU consumption on a periodic
+# basis, writing the output into '<test>.<shard>.perf.txt' logs in the test log
+# directory. This is handy in order to determine an appropriate value for the
+# 'PROCESSORS' property passed to 'ADD_KUDU_TEST' in CMakeLists.txt.
# Path to the test executable or script to be run.
# May be relative or absolute.
@@ -162,8 +168,14 @@ for ATTEMPT_NUMBER in $(seq 1 $TEST_EXECUTION_ATTEMPTS) ; do
addr2line_filter="$SOURCE_ROOT/build-support/stacktrace_addr2line.pl $ABS_TEST_PATH"
fi
echo "Running $TEST_NAME, redirecting output into $LOGFILE" \
- "(attempt ${ATTEMPT_NUMBER}/$TEST_EXECUTION_ATTEMPTS)"
- $ABS_TEST_PATH "$@" --test_timeout_after $KUDU_TEST_TIMEOUT 2>&1 \
+ "(attempt ${ATTEMPT_NUMBER}/$TEST_EXECUTION_ATTEMPTS)"
+ test_command=$ABS_TEST_PATH
+ if [ -n "$KUDU_MEASURE_TEST_CPU_CONSUMPTION" ] && [ "$KUDU_MEASURE_TEST_CPU_CONSUMPTION" -ne 0 ] ; then
+ perf_file=$TEST_LOGDIR/$TEST_NAME.perf.txt
+ echo "Will wrap test in perf-stat, output to ${perf_file}..."
+ test_command="perf stat -I1000 -e task-clock -o $perf_file $test_command"
+ fi
+ $test_command "$@" --test_timeout_after $KUDU_TEST_TIMEOUT 2>&1 \
| $addr2line_filter \
| $pipe_cmd > $LOGFILE
STATUS=$?
@@ -202,7 +214,7 @@ for ATTEMPT_NUMBER in $(seq 1 $TEST_EXECUTION_ATTEMPTS) ; do
done
fi
- if [ -n "$KUDU_REPORT_TEST_RESULTS" ]; then
+ if [ -n "$KUDU_REPORT_TEST_RESULTS" ] && [ "$KUDU_REPORT_TEST_RESULTS" -ne 0 ]; then
echo Reporting results
$SOURCE_ROOT/build-support/report-test.sh "$ABS_TEST_PATH" "$LOGFILE" "$STATUS" &
http://git-wip-us.apache.org/repos/asf/kudu/blob/1c1d3ba1/src/kudu/cfile/CMakeLists.txt
----------------------------------------------------------------------
diff --git a/src/kudu/cfile/CMakeLists.txt b/src/kudu/cfile/CMakeLists.txt
index c42e589..7eac936 100644
--- a/src/kudu/cfile/CMakeLists.txt
+++ b/src/kudu/cfile/CMakeLists.txt
@@ -60,5 +60,5 @@ ADD_KUDU_TEST(index-test)
ADD_KUDU_TEST(cfile-test NUM_SHARDS 4)
ADD_KUDU_TEST(encoding-test LABELS no_tsan)
ADD_KUDU_TEST(bloomfile-test)
-ADD_KUDU_TEST(mt-bloomfile-test)
+ADD_KUDU_TEST(mt-bloomfile-test RUN_SERIAL true)
ADD_KUDU_TEST(block_cache-test)
http://git-wip-us.apache.org/repos/asf/kudu/blob/1c1d3ba1/src/kudu/client/CMakeLists.txt
----------------------------------------------------------------------
diff --git a/src/kudu/client/CMakeLists.txt b/src/kudu/client/CMakeLists.txt
index 68d07b3..776acdb 100644
--- a/src/kudu/client/CMakeLists.txt
+++ b/src/kudu/client/CMakeLists.txt
@@ -261,7 +261,7 @@ set(KUDU_TEST_LINK_LIBS
kudu_client
mini_cluster
${KUDU_MIN_TEST_LIBS})
-ADD_KUDU_TEST(client-test NUM_SHARDS 8)
+ADD_KUDU_TEST(client-test NUM_SHARDS 8 PROCESSORS 2)
ADD_KUDU_TEST(client-unittest)
ADD_KUDU_TEST(predicate-test)
-ADD_KUDU_TEST(scan_token-test)
+ADD_KUDU_TEST(scan_token-test PROCESSORS 2)
http://git-wip-us.apache.org/repos/asf/kudu/blob/1c1d3ba1/src/kudu/clock/CMakeLists.txt
----------------------------------------------------------------------
diff --git a/src/kudu/clock/CMakeLists.txt b/src/kudu/clock/CMakeLists.txt
index 46ebe3b..7371aeb 100644
--- a/src/kudu/clock/CMakeLists.txt
+++ b/src/kudu/clock/CMakeLists.txt
@@ -33,5 +33,5 @@ target_link_libraries(clock
set(KUDU_TEST_LINK_LIBS ${KUDU_MIN_TEST_LIBS}
clock)
-ADD_KUDU_TEST(hybrid_clock-test)
+ADD_KUDU_TEST(hybrid_clock-test PROCESSORS 3)
ADD_KUDU_TEST(logical_clock-test)
http://git-wip-us.apache.org/repos/asf/kudu/blob/1c1d3ba1/src/kudu/consensus/CMakeLists.txt
----------------------------------------------------------------------
diff --git a/src/kudu/consensus/CMakeLists.txt b/src/kudu/consensus/CMakeLists.txt
index ef2f9a7..11463a6 100644
--- a/src/kudu/consensus/CMakeLists.txt
+++ b/src/kudu/consensus/CMakeLists.txt
@@ -135,9 +135,9 @@ ADD_KUDU_TEST(consensus_queue-test)
ADD_KUDU_TEST(leader_election-test)
ADD_KUDU_TEST(log-test)
ADD_KUDU_TEST(log_anchor_registry-test)
-ADD_KUDU_TEST(log_cache-test)
+ADD_KUDU_TEST(log_cache-test PROCESSORS 2)
ADD_KUDU_TEST(log_index-test)
-ADD_KUDU_TEST(mt-log-test)
+ADD_KUDU_TEST(mt-log-test PROCESSORS 5)
ADD_KUDU_TEST(quorum_util-test)
ADD_KUDU_TEST(raft_consensus_quorum-test)
ADD_KUDU_TEST(time_manager-test)
http://git-wip-us.apache.org/repos/asf/kudu/blob/1c1d3ba1/src/kudu/integration-tests/CMakeLists.txt
----------------------------------------------------------------------
diff --git a/src/kudu/integration-tests/CMakeLists.txt b/src/kudu/integration-tests/CMakeLists.txt
index 21d842a..49bcad3 100644
--- a/src/kudu/integration-tests/CMakeLists.txt
+++ b/src/kudu/integration-tests/CMakeLists.txt
@@ -54,55 +54,57 @@ add_dependencies(itest_util
set(KUDU_TEST_LINK_LIBS itest_util ${KUDU_MIN_TEST_LIBS})
ADD_KUDU_TEST(all_types-itest
RESOURCE_LOCK "master-rpc-ports"
+ PROCESSORS 4
NUM_SHARDS 8)
-ADD_KUDU_TEST(alter_table-randomized-test)
-ADD_KUDU_TEST(alter_table-test)
+ADD_KUDU_TEST(alter_table-randomized-test
+ PROCESSORS 4)
+ADD_KUDU_TEST(alter_table-test PROCESSORS 3)
ADD_KUDU_TEST(authn_token_expire-itest)
-ADD_KUDU_TEST(catalog_manager_tsk-itest)
+ADD_KUDU_TEST(catalog_manager_tsk-itest PROCESSORS 2)
ADD_KUDU_TEST(client_failover-itest)
ADD_KUDU_TEST(client-negotiation-failover-itest
RESOURCE_LOCK "master-rpc-ports")
ADD_KUDU_TEST(client-stress-test
RESOURCE_LOCK "master-rpc-ports"
RUN_SERIAL true)
-ADD_KUDU_TEST(consistency-itest)
-ADD_KUDU_TEST(create-table-itest)
-ADD_KUDU_TEST(create-table-stress-test)
+ADD_KUDU_TEST(consistency-itest PROCESSORS 5)
+ADD_KUDU_TEST(create-table-itest PROCESSORS 3)
+ADD_KUDU_TEST(create-table-stress-test RUN_SERIAL true)
ADD_KUDU_TEST(decimal-itest)
-ADD_KUDU_TEST(delete_table-itest NUM_SHARDS 8)
-ADD_KUDU_TEST(delete_tablet-itest)
-ADD_KUDU_TEST(disk_failure-itest)
+ADD_KUDU_TEST(delete_table-itest NUM_SHARDS 8 PROCESSORS 4)
+ADD_KUDU_TEST(delete_tablet-itest PROCESSORS 2)
+ADD_KUDU_TEST(disk_failure-itest PROCESSORS 2)
ADD_KUDU_TEST(disk_reservation-itest)
ADD_KUDU_TEST(exactly_once_writes-itest)
-ADD_KUDU_TEST(flex_partitioning-itest TIMEOUT 1800 NUM_SHARDS 8)
+ADD_KUDU_TEST(flex_partitioning-itest TIMEOUT 1800 NUM_SHARDS 8 PROCESSORS 2)
ADD_KUDU_TEST(full_stack-insert-scan-test RUN_SERIAL true)
ADD_KUDU_TEST(fuzz-itest RUN_SERIAL true)
ADD_KUDU_TEST(heavy-update-compaction-itest RUN_SERIAL true)
-ADD_KUDU_TEST(linked_list-test RESOURCE_LOCK "master-rpc-ports")
+ADD_KUDU_TEST(linked_list-test RESOURCE_LOCK "master-rpc-ports" RUN_SERIAL true)
ADD_KUDU_TEST(log-rolling-itest)
-ADD_KUDU_TEST(master_cert_authority-itest RESOURCE_LOCK "master-rpc-ports")
-ADD_KUDU_TEST(master_failover-itest RESOURCE_LOCK "master-rpc-ports")
+ADD_KUDU_TEST(master_cert_authority-itest RESOURCE_LOCK "master-rpc-ports" PROCESSORS 2)
+ADD_KUDU_TEST(master_failover-itest RESOURCE_LOCK "master-rpc-ports" PROCESSORS 3)
ADD_KUDU_TEST(master_migration-itest RESOURCE_LOCK "master-rpc-ports")
ADD_KUDU_TEST_DEPENDENCIES(master_migration-itest
kudu)
ADD_KUDU_TEST(master_replication-itest RESOURCE_LOCK "master-rpc-ports")
-ADD_KUDU_TEST(master-stress-test RESOURCE_LOCK "master-rpc-ports")
+ADD_KUDU_TEST(master-stress-test RESOURCE_LOCK "master-rpc-ports" RUN_SERIAL true)
ADD_KUDU_TEST(multidir_cluster-itest)
-ADD_KUDU_TEST(open-readonly-fs-itest)
+ADD_KUDU_TEST(open-readonly-fs-itest PROCESSORS 4)
ADD_KUDU_TEST(raft_config_change-itest)
-ADD_KUDU_TEST(raft_consensus_election-itest)
+ADD_KUDU_TEST(raft_consensus_election-itest PROCESSORS 4)
ADD_KUDU_TEST(raft_consensus_failure_detector-imc-itest)
-ADD_KUDU_TEST(raft_consensus_nonvoter-itest)
-ADD_KUDU_TEST(raft_consensus_stress-itest)
+ADD_KUDU_TEST(raft_consensus_nonvoter-itest PROCESSORS 3)
+ADD_KUDU_TEST(raft_consensus_stress-itest RUN_SERIAL true)
ADD_KUDU_TEST(raft_consensus-itest RUN_SERIAL true NUM_SHARDS 6)
ADD_KUDU_TEST(registration-test RESOURCE_LOCK "master-web-port")
ADD_KUDU_TEST(security-faults-itest)
ADD_KUDU_TEST(security-itest RESOURCE_LOCK "master-rpc-ports")
ADD_KUDU_TEST(security-master-auth-itest RESOURCE_LOCK "master-rpc-ports")
-ADD_KUDU_TEST(security-unknown-tsk-itest)
-ADD_KUDU_TEST(stop_tablet-itest)
+ADD_KUDU_TEST(security-unknown-tsk-itest PROCESSORS 4)
+ADD_KUDU_TEST(stop_tablet-itest PROCESSORS 4)
ADD_KUDU_TEST(table_locations-itest)
-ADD_KUDU_TEST(tablet_copy-itest NUM_SHARDS 6)
+ADD_KUDU_TEST(tablet_copy-itest NUM_SHARDS 6 PROCESSORS 4)
ADD_KUDU_TEST(tablet_copy_client_session-itest)
ADD_KUDU_TEST(tablet_history_gc-itest)
ADD_KUDU_TEST(tablet_replacement-itest)
@@ -110,10 +112,10 @@ ADD_KUDU_TEST(tombstoned_voting-imc-itest)
ADD_KUDU_TEST(tombstoned_voting-itest)
ADD_KUDU_TEST(tombstoned_voting-stress-test RUN_SERIAL true)
ADD_KUDU_TEST(token_signer-itest RESOURCE_LOCK "master-rpc-ports")
-ADD_KUDU_TEST(ts_recovery-itest)
+ADD_KUDU_TEST(ts_recovery-itest PROCESSORS 4)
ADD_KUDU_TEST(ts_tablet_manager-itest)
ADD_KUDU_TEST(update_scan_delta_compact-test RUN_SERIAL true)
-ADD_KUDU_TEST(webserver-stress-itest RESOURCE_LOCK "master-rpc-ports")
+ADD_KUDU_TEST(webserver-stress-itest RESOURCE_LOCK "master-rpc-ports" RUN_SERIAL true)
ADD_KUDU_TEST(write_throttling-itest)
if (NOT APPLE)
@@ -121,7 +123,7 @@ if (NOT APPLE)
# Opens too many files on macOS because we're not using a file cache for
# reading log segments.
# TODO(wdb): Re-enable once a file cache is being used.
- ADD_KUDU_TEST(dense_node-itest)
+ ADD_KUDU_TEST(dense_node-itest RUN_SERIAL true)
endif()
# Tests that should not be run automatically by ctest.
http://git-wip-us.apache.org/repos/asf/kudu/blob/1c1d3ba1/src/kudu/mini-cluster/CMakeLists.txt
----------------------------------------------------------------------
diff --git a/src/kudu/mini-cluster/CMakeLists.txt b/src/kudu/mini-cluster/CMakeLists.txt
index c3be379..9d2e394 100644
--- a/src/kudu/mini-cluster/CMakeLists.txt
+++ b/src/kudu/mini-cluster/CMakeLists.txt
@@ -49,4 +49,4 @@ add_dependencies(mini_cluster
# Tests
set(KUDU_TEST_LINK_LIBS mini_cluster kudu_hms ${KUDU_MIN_TEST_LIBS})
-ADD_KUDU_TEST(external_mini_cluster-test RESOURCE_LOCK "master-rpc-ports")
+ADD_KUDU_TEST(external_mini_cluster-test RESOURCE_LOCK "master-rpc-ports" PROCESSORS 3)
http://git-wip-us.apache.org/repos/asf/kudu/blob/1c1d3ba1/src/kudu/rpc/CMakeLists.txt
----------------------------------------------------------------------
diff --git a/src/kudu/rpc/CMakeLists.txt b/src/kudu/rpc/CMakeLists.txt
index 5a3a2b0..f8cdb02 100644
--- a/src/kudu/rpc/CMakeLists.txt
+++ b/src/kudu/rpc/CMakeLists.txt
@@ -126,7 +126,7 @@ set(KUDU_TEST_LINK_LIBS
rtest_krpc
security_test_util
${KUDU_MIN_TEST_LIBS})
-ADD_KUDU_TEST(exactly_once_rpc-test)
+ADD_KUDU_TEST(exactly_once_rpc-test PROCESSORS 10)
ADD_KUDU_TEST(mt-rpc-test RUN_SERIAL true)
ADD_KUDU_TEST(negotiation-test)
ADD_KUDU_TEST(periodic-test)
@@ -135,4 +135,4 @@ ADD_KUDU_TEST(request_tracker-test)
ADD_KUDU_TEST(rpc-bench RUN_SERIAL true)
ADD_KUDU_TEST(rpc-test)
ADD_KUDU_TEST(rpc_stub-test)
-ADD_KUDU_TEST(service_queue-test)
+ADD_KUDU_TEST(service_queue-test RUN_SERIAL true)
http://git-wip-us.apache.org/repos/asf/kudu/blob/1c1d3ba1/src/kudu/security/CMakeLists.txt
----------------------------------------------------------------------
diff --git a/src/kudu/security/CMakeLists.txt b/src/kudu/security/CMakeLists.txt
index 509d01b..3af155d 100644
--- a/src/kudu/security/CMakeLists.txt
+++ b/src/kudu/security/CMakeLists.txt
@@ -127,6 +127,6 @@ if (NOT NO_TESTS)
ADD_KUDU_TEST(crypto-test)
ADD_KUDU_TEST(test/mini_kdc-test)
ADD_KUDU_TEST(tls_handshake-test)
- ADD_KUDU_TEST(tls_socket-test)
+ ADD_KUDU_TEST(tls_socket-test PROCESSORS 2)
ADD_KUDU_TEST(token-test)
endif()
http://git-wip-us.apache.org/repos/asf/kudu/blob/1c1d3ba1/src/kudu/tablet/CMakeLists.txt
----------------------------------------------------------------------
diff --git a/src/kudu/tablet/CMakeLists.txt b/src/kudu/tablet/CMakeLists.txt
index 99c7de0..771e22a 100644
--- a/src/kudu/tablet/CMakeLists.txt
+++ b/src/kudu/tablet/CMakeLists.txt
@@ -96,7 +96,7 @@ ADD_KUDU_TEST(compaction_policy-test
# Can't use dist-test because it relies on a data file.
LABELS no_dist_test)
-ADD_KUDU_TEST(all_types-scan-correctness-test NUM_SHARDS 8)
+ADD_KUDU_TEST(all_types-scan-correctness-test NUM_SHARDS 8 PROCESSORS 2)
ADD_KUDU_TEST(diskrowset-test)
ADD_KUDU_TEST(mt-diskrowset-test RUN_SERIAL true)
ADD_KUDU_TEST(memrowset-test)
@@ -114,7 +114,7 @@ ADD_KUDU_TEST(lock_manager-test)
ADD_KUDU_TEST(rowset_tree-test NUM_SHARDS 6)
ADD_KUDU_TEST(composite-pushdown-test)
ADD_KUDU_TEST(delta_compaction-test)
-ADD_KUDU_TEST(mt-rowset_delta_compaction-test)
+ADD_KUDU_TEST(mt-rowset_delta_compaction-test PROCESSORS 2)
ADD_KUDU_TEST(major_delta_compaction-test)
ADD_KUDU_TEST(transactions/transaction_tracker-test)
ADD_KUDU_TEST(tablet_history_gc-test)
http://git-wip-us.apache.org/repos/asf/kudu/blob/1c1d3ba1/src/kudu/tools/CMakeLists.txt
----------------------------------------------------------------------
diff --git a/src/kudu/tools/CMakeLists.txt b/src/kudu/tools/CMakeLists.txt
index 12f5e40..06fe484 100644
--- a/src/kudu/tools/CMakeLists.txt
+++ b/src/kudu/tools/CMakeLists.txt
@@ -141,11 +141,11 @@ set(KUDU_TEST_LINK_LIBS
mini_cluster
${KUDU_MIN_TEST_LIBS})
ADD_KUDU_TEST(ksck-test)
-ADD_KUDU_TEST(ksck_remote-test RESOURCE_LOCK "master-rpc-ports")
-ADD_KUDU_TEST(kudu-admin-test)
+ADD_KUDU_TEST(ksck_remote-test RESOURCE_LOCK "master-rpc-ports" PROCESSORS 3)
+ADD_KUDU_TEST(kudu-admin-test PROCESSORS 3)
ADD_KUDU_TEST_DEPENDENCIES(kudu-admin-test
kudu)
-ADD_KUDU_TEST(kudu-tool-test NUM_SHARDS 4)
+ADD_KUDU_TEST(kudu-tool-test NUM_SHARDS 4 PROCESSORS 3)
ADD_KUDU_TEST_DEPENDENCIES(kudu-tool-test
kudu)
ADD_KUDU_TEST(kudu-ts-cli-test)
http://git-wip-us.apache.org/repos/asf/kudu/blob/1c1d3ba1/src/kudu/tserver/CMakeLists.txt
----------------------------------------------------------------------
diff --git a/src/kudu/tserver/CMakeLists.txt b/src/kudu/tserver/CMakeLists.txt
index e062e2d..3a67f44 100644
--- a/src/kudu/tserver/CMakeLists.txt
+++ b/src/kudu/tserver/CMakeLists.txt
@@ -175,7 +175,7 @@ ADD_KUDU_TEST(mini_tablet_server-test)
ADD_KUDU_TEST(tablet_copy_client-test)
ADD_KUDU_TEST(tablet_copy_source_session-test)
ADD_KUDU_TEST(tablet_copy_service-test)
-ADD_KUDU_TEST(tablet_server-test)
+ADD_KUDU_TEST(tablet_server-test PROCESSORS 3)
ADD_KUDU_TEST(tablet_server-stress-test RUN_SERIAL true)
ADD_KUDU_TEST(scanners-test)
ADD_KUDU_TEST(ts_tablet_manager-test)
http://git-wip-us.apache.org/repos/asf/kudu/blob/1c1d3ba1/src/kudu/util/CMakeLists.txt
----------------------------------------------------------------------
diff --git a/src/kudu/util/CMakeLists.txt b/src/kudu/util/CMakeLists.txt
index 46a0d31..c91a780 100644
--- a/src/kudu/util/CMakeLists.txt
+++ b/src/kudu/util/CMakeLists.txt
@@ -382,23 +382,23 @@ ADD_KUDU_TEST(random-test)
ADD_KUDU_TEST(random_util-test)
ADD_KUDU_TEST(rle-test)
ADD_KUDU_TEST(rolling_log-test)
-ADD_KUDU_TEST(rw_mutex-test)
+ADD_KUDU_TEST(rw_mutex-test RUN_SERIAL true)
ADD_KUDU_TEST(rw_semaphore-test)
-ADD_KUDU_TEST(rwc_lock-test)
+ADD_KUDU_TEST(rwc_lock-test RUN_SERIAL true)
ADD_KUDU_TEST(safe_math-test)
ADD_KUDU_TEST(scoped_cleanup-test)
ADD_KUDU_TEST(slice-test)
ADD_KUDU_TEST(sorted_disjoint_interval_list-test)
ADD_KUDU_TEST(spinlock_profiling-test)
-ADD_KUDU_TEST(stack_watchdog-test)
+ADD_KUDU_TEST(stack_watchdog-test PROCESSORS 2)
ADD_KUDU_TEST(status-test)
ADD_KUDU_TEST(string_case-test)
-ADD_KUDU_TEST(striped64-test)
+ADD_KUDU_TEST(striped64-test RUN_SERIAL true)
ADD_KUDU_TEST(subprocess-test)
ADD_KUDU_TEST(thread-test)
ADD_KUDU_TEST(threadpool-test)
ADD_KUDU_TEST(throttler-test)
-ADD_KUDU_TEST(trace-test)
+ADD_KUDU_TEST(trace-test PROCESSORS 4)
ADD_KUDU_TEST(url-coding-test)
ADD_KUDU_TEST(user-test)