You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kudu.apache.org by al...@apache.org on 2020/03/13 15:34:22 UTC
[kudu] branch master updated (de2bf21 -> 84745c3)
This is an automated email from the ASF dual-hosted git repository.
alexey pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git.
from de2bf21 KUDU-3063: Set a ratio as multiplication factor for memkind_malloc_usable_size
new 30c8287 [backup] fix log info
new 3343144 [tests] use 768-bit RSA keys for crypto in tests
new 84745c3 [gradle] remove failOnNoMatchingTests
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
java/gradle/tests.gradle | 7 -------
.../main/scala/org/apache/kudu/backup/KuduBackup.scala | 2 +-
.../org/apache/kudu/test/cluster/MiniKuduCluster.java | 9 +++++++++
src/kudu/mini-cluster/external_mini_cluster.cc | 18 ++++++++----------
src/kudu/tools/tool_test_util.cc | 4 ++--
src/kudu/util/test_util.cc | 17 ++++++++---------
6 files changed, 28 insertions(+), 29 deletions(-)
[kudu] 02/03: [tests] use 768-bit RSA keys for crypto in tests
Posted by al...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
alexey pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit 3343144fefaad5a30e95e21297c64c78e308fa1f
Author: Alexey Serbin <al...@apache.org>
AuthorDate: Thu Mar 12 14:27:50 2020 -0700
[tests] use 768-bit RSA keys for crypto in tests
With default Java security policies on CentOS 8, RSA keys shorter than
2048 bits are considered lax and cannot be used while establishing a TLS
connection, server certificates are rejected, etc.
To allow smaller RSA keys to be accepted by Java security, this patch
overrides MiniKuduCluster's custom security settings for the
corresponding properties:
* jdk.certpath.disabledAlgorithms
* jdk.tls.disabledAlgorithms
In addition, this patch lowers security level for the OpenSSL library
to 0 (applicable starting OpenSSL 1.1.0) to allow for using of 768-bit
RSA keys. This is to spare CPU resources while running Kudu tests.
This is a follow-up to 0ee93e31a1febb987c72e7392a69b2584e6f38ed.
Change-Id: I6fa0794ff8f8dac6c58b1765539e3fc16857013f
Reviewed-on: http://gerrit.cloudera.org:8080/15426
Tested-by: Kudu Jenkins
Reviewed-by: Grant Henke <gr...@apache.org>
---
.../org/apache/kudu/test/cluster/MiniKuduCluster.java | 9 +++++++++
src/kudu/mini-cluster/external_mini_cluster.cc | 18 ++++++++----------
src/kudu/tools/tool_test_util.cc | 4 ++--
src/kudu/util/test_util.cc | 17 ++++++++---------
4 files changed, 27 insertions(+), 21 deletions(-)
diff --git a/java/kudu-test-utils/src/main/java/org/apache/kudu/test/cluster/MiniKuduCluster.java b/java/kudu-test-utils/src/main/java/org/apache/kudu/test/cluster/MiniKuduCluster.java
index bd19625..09ac649 100644
--- a/java/kudu-test-utils/src/main/java/org/apache/kudu/test/cluster/MiniKuduCluster.java
+++ b/java/kudu-test-utils/src/main/java/org/apache/kudu/test/cluster/MiniKuduCluster.java
@@ -27,6 +27,7 @@ import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.nio.file.Paths;
+import java.security.Security;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
@@ -140,6 +141,14 @@ public final class MiniKuduCluster implements AutoCloseable {
} else {
this.clusterRoot = clusterRoot;
}
+
+ // Default Java security settings are restrictive with regard to RSA key
+ // length. Since Kudu masters and tablet servers in MiniKuduCluster use
+ // smaller RSA keys to shorten runtime of tests, it's necessary to override
+ // those default security settings to allow for using relaxed cryptography,
+ // particularly smaller RSA keys.
+ Security.setProperty("jdk.certpath.disabledAlgorithms", "MD2, RC4, MD5");
+ Security.setProperty("jdk.tls.disabledAlgorithms", "SSLv3, RC4, MD5");
}
/**
diff --git a/src/kudu/mini-cluster/external_mini_cluster.cc b/src/kudu/mini-cluster/external_mini_cluster.cc
index 66da128..23c40c5 100644
--- a/src/kudu/mini-cluster/external_mini_cluster.cc
+++ b/src/kudu/mini-cluster/external_mini_cluster.cc
@@ -988,21 +988,19 @@ Status ExternalDaemon::StartProcess(const vector<string>& user_flags) {
// rely on forcefully cutting power to a machine or equivalent.
"--never_fsync",
- // Generate smaller RSA keys -- generating a 1024-bit key is faster
+ // Generate smaller RSA keys -- generating a 768-bit key is faster
// than generating the default 2048-bit key, and we don't care about
// strong encryption in tests. Setting it lower (e.g. 512 bits) results
// in OpenSSL errors RSA_sign:digest too big for rsa key:rsa_sign.c:122
// since we are using strong/high TLS v1.2 cipher suites, so the minimum
// size of TLS-related RSA key is 768 bits (due to the usage of
- // the ECDHE-RSA-AES256-GCM-SHA384 suite). However, to work with Java
- // client it's necessary to have at least 1024 bits for certificate RSA key
- // due to Java security policies.
- "--ipki_server_key_size=1024",
+ // the ECDHE-RSA-AES256-GCM-SHA384 suite).
+ "--ipki_server_key_size=768",
- // The RSA key of 1024 bits is too short if OpenSSL security level is set to
- // 2 or higher (applicable for OpenSSL 1.1.0 and newer). Lowering the
- // security level to 1 makes possible ot use shorter keys in such cases.
- "--openssl_security_level_override=1",
+ // The RSA key of 768 bits is too short if OpenSSL security level is set to
+ // 1 or higher (applicable for OpenSSL 1.1.0 and newer). Lowering the
+ // security level to 0 makes possible ot use shorter keys in such cases.
+ "--openssl_security_level_override=0",
// Disable minidumps by default since many tests purposely inject faults.
"--enable_minidumps=false",
@@ -1469,7 +1467,7 @@ const vector<string>& ExternalMaster::GetCommonFlags() {
static const vector<string> kFlags = {
// See the in-line comment for "--ipki_server_key_size" flag in
// ExternalDaemon::StartProcess() method.
- "--ipki_ca_key_size=1024",
+ "--ipki_ca_key_size=768",
// As for the TSK keys, 512 bits is the minimum since we are using
// SHA256 digest for token signing/verification.
diff --git a/src/kudu/tools/tool_test_util.cc b/src/kudu/tools/tool_test_util.cc
index 59c3696..22abba5 100644
--- a/src/kudu/tools/tool_test_util.cc
+++ b/src/kudu/tools/tool_test_util.cc
@@ -69,9 +69,9 @@ Status RunKuduTool(const vector<string>& args, string* out, string* err,
// Kudu masters and tablet servers run as a part of external mini-cluster use
// shorter keys. Newer OS distros have OpenSSL built with the default security
- // level higher than 1, so it's necessary to override it on the client
+ // level higher than 0, so it's necessary to override it on the client
// side as well to allow clients to accept and verify TLS certificates.
- total_args.emplace_back("--openssl_security_level_override=1");
+ total_args.emplace_back("--openssl_security_level_override=0");
total_args.insert(total_args.end(), args.begin(), args.end());
return Subprocess::Call(total_args, in, out, err);
diff --git a/src/kudu/util/test_util.cc b/src/kudu/util/test_util.cc
index 4a74a8b..76fe872 100644
--- a/src/kudu/util/test_util.cc
+++ b/src/kudu/util/test_util.cc
@@ -102,18 +102,17 @@ KuduTest::KuduTest()
{"redact", "none"},
// Reduce default RSA key length for faster tests. We are using strong/high
// TLS v1.2 cipher suites, so minimum possible for TLS-related RSA keys is
- // 768 bits. However, for the external mini cluster we use 1024 bits because
- // Java default security policies require at least 1024 bits for RSA keys
- // used in certificates. For uniformity, here 1024 RSA bit keys are used
- // as well. As for the TSK keys, 512 bits is the minimum since the SHA256
- // digest is used for token signing/verification.
- {"ipki_server_key_size", "1024"},
- {"ipki_ca_key_size", "1024"},
+ // 768 bits. Java security policies in tests tweaked appropriately to allow
+ // for using smaller RSA keys in certificates. As for the TSK keys, 512 bits
+ // is the minimum since the SHA256 digest is used for token
+ // signing/verification.
+ {"ipki_server_key_size", "768"},
+ {"ipki_ca_key_size", "768"},
{"tsk_num_rsa_bits", "512"},
- // Some OS distros set the default security level higher than 1, so it's
+ // Some OS distros set the default security level higher than 0, so it's
// necessary to override it to use the key length specified above (which are
// considered lax and don't work in case of security level 2 or higher).
- {"openssl_security_level_override", "1"},
+ {"openssl_security_level_override", "0"},
// For a generic Kudu test, the local wall-clock time is good enough even
// if it's not synchronized by NTP. All test components are run at the same
// node, so there aren't multiple time sources to synchronize.
[kudu] 03/03: [gradle] remove failOnNoMatchingTests
Posted by al...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
alexey pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit 84745c30253aee48da4f4a05da1fb187281bf0e6
Author: Alexey Serbin <al...@apache.org>
AuthorDate: Fri Mar 13 00:03:34 2020 -0700
[gradle] remove failOnNoMatchingTests
I found it confusing when gradle reports success on executing test
targets which don't exist. Prior to this patch, the following returned
success and claimed successfully executing tasks:
$ ./gradlew :kudu-client:test --tests org.apache.kudu.client.Nonsense
BUILD SUCCESSFUL in 1s
15 actionable tasks: 1 executed, 14 up-to-date
This patch fixes it by removing failOnNoMatchingTests.
It seems there were some ramifications on the pattern matching for
dist-test some time ago, but I verified prior and after this patch
and saw that the following works and breaks the same way:
works:
../build-support/dist_test.py java loop -n 8 '*ITClient'
../build-support/dist_test.py java loop -n 8 '*ITClient*'
../build-support/dist_test.py java loop -n 8 org/apache/kudu/client/TestConnectToCluster
doesn't work:
../build-support/dist_test.py java loop -n 8 '*ITClient.class'
../build-support/dist_test.py java loop -n 8 '*ITClient*X'
../build-support/dist_test.py java loop -n 8 org/apache/kudu/client/TestConnectToCluster.class
Change-Id: Iabb0f72fb9481c775ebcc9881a39180fb0413154
Reviewed-on: http://gerrit.cloudera.org:8080/15428
Tested-by: Alexey Serbin <as...@cloudera.com>
Reviewed-by: Grant Henke <gr...@apache.org>
---
java/gradle/tests.gradle | 7 -------
1 file changed, 7 deletions(-)
diff --git a/java/gradle/tests.gradle b/java/gradle/tests.gradle
index eaf083b..51a3d7f 100644
--- a/java/gradle/tests.gradle
+++ b/java/gradle/tests.gradle
@@ -80,13 +80,6 @@ tasks.withType(Test) {
systemProperty "testRandomSeed", propertyWithDefault("testRandomSeed", null)
}
- // Don't fail the build if a --tests filter doesn't match any tests.
- // This is useful for filtering tests from the top of the project where some
- // subprojects won't match the filter. Without this, those subprojects would fail.
- filter {
- failOnNoMatchingTests = false
- }
-
// Force the tests to be rerun if the rerunTests property is set.
if (propertyExists("rerunTests")) {
outputs.upToDateWhen { false }
[kudu] 01/03: [backup] fix log info
Posted by al...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
alexey pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit 30c8287d7b1c8790fda0b6acadbb649178c212a3
Author: zhangyifan27 <ch...@163.com>
AuthorDate: Fri Mar 13 19:02:21 2020 +0800
[backup] fix log info
Change-Id: I4eae75374c21253acfcbf410617b6b39cfbed7bf
Reviewed-on: http://gerrit.cloudera.org:8080/15429
Tested-by: Kudu Jenkins
Reviewed-by: Grant Henke <gr...@apache.org>
---
java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala b/java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala
index f881c38..2c7347a 100644
--- a/java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala
+++ b/java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala
@@ -140,7 +140,7 @@ object KuduBackup {
val backupResult = Try(doBackup(tableName, context, session, io, options, backupMap))
backupResult match {
case Success(()) =>
- log.info(s"Successfully backed up up table $tableName")
+ log.info(s"Successfully backed up table $tableName")
case Failure(ex) =>
if (options.numParallelBackups == 1 && options.failOnFirstError)
throw ex