You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kudu.apache.org by aw...@apache.org on 2020/12/02 20:32:19 UTC
[kudu] branch branch-1.13.x updated (07eb5a9 -> 2acc4d6)
This is an automated email from the ASF dual-hosted git repository.
awong pushed a change to branch branch-1.13.x
in repository https://gitbox.apache.org/repos/asf/kudu.git.
from 07eb5a9 [kudu-tool-test] fix ClusterNameResolverFileCorrupt with glibc 2.31
new 5a66fff [dist_test] Ship security libraries to dist_test
new 1e630db KUDU-2700: Add a LSAN suppression
new 2acc4d6 [test] Use /tmp for mini-chronyd sockets
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
build-support/dist_test.py | 63 +++++++++++++++++++++++++++++++++++--
build-support/run_dist_test.py | 6 ++++
cmake_modules/FindCyrusSASL.cmake | 2 +-
src/kudu/clock/test/mini_chronyd.cc | 7 ++---
src/kudu/util/sanitizer_options.cc | 4 +++
5 files changed, 75 insertions(+), 7 deletions(-)
[kudu] 02/03: KUDU-2700: Add a LSAN suppression
Posted by aw...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch branch-1.13.x
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit 1e630db0348c44793f55513eddd23e8f4ab9b2c4
Author: Grant Henke <gr...@apache.org>
AuthorDate: Tue Nov 24 11:18:56 2020 -0600
KUDU-2700: Add a LSAN suppression
OpenSSL 1.1 has a leak in libcrypto.so when running tests that use the CLI
to connect to a remote server. Previously this didn’t impact CI runs
because Jenkins and dist_test were using Ubuntu 14 and therefore
OpenSSL 1.0. Now that we are using Ubuntu 18 and OpenSSL 1.1
this leak is reproducable 100% of the time.
This patch suppresses the leak to unbreak the ASAN builds.
Change-Id: I5d1378a820997d3eabfa7e5cda53c07dd4df917d
Reviewed-on: http://gerrit.cloudera.org:8080/16781
Reviewed-by: Andrew Wong <aw...@cloudera.com>
Reviewed-by: Alexey Serbin <as...@cloudera.com>
Tested-by: Kudu Jenkins
Reviewed-on: http://gerrit.cloudera.org:8080/16804
Reviewed-by: Grant Henke <gr...@apache.org>
Tested-by: Andrew Wong <aw...@cloudera.com>
---
src/kudu/util/sanitizer_options.cc | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/src/kudu/util/sanitizer_options.cc b/src/kudu/util/sanitizer_options.cc
index 8c8c0d3..bd77459 100644
--- a/src/kudu/util/sanitizer_options.cc
+++ b/src/kudu/util/sanitizer_options.cc
@@ -194,6 +194,10 @@ SANITIZER_HOOK_ATTRIBUTE const char *__lsan_default_suppressions() {
// Fixed by upstream commit 379d39c17b8930718e98185a5b32a0f7f3e3b4b6
"leak:krb5_authdata_import_attributes\n"
+ // KUDU-2700: OpenSSL 1.1 has a leak in libcrypto.so when running tests
+ // that use the CLI to connect to a remote server.
+ "leak:libcrypto.so.1.1\n"
+
// KUDU-2653: Memory leak in libgssapi_krb5 [1]. Exists in certain patched
// versions of krb5-1.12 (such as krb5 in Debian 8).
//
[kudu] 01/03: [dist_test] Ship security libraries to dist_test
Posted by aw...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch branch-1.13.x
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit 5a66fffcdcfbb97ea471d6db05d61dcf4b1b7c22
Author: Grant Henke <gr...@apache.org>
AuthorDate: Thu Nov 12 19:05:11 2020 -0600
[dist_test] Ship security libraries to dist_test
This patch adjusts the lib whitelist to allow shipping the security libraries
to dist_test and allow more flexibility when versions do not match the
dist_test images versions.
This was already happening for rhel6 installs due to the rhel6 workaround script
linked below. With this change the libraries will be shipped even when not
in thirdparty.
https://github.com/apache/kudu/blob/master/thirdparty/install-openssl-el6-workaround.sh
I also needed to adjust run_dist_test.py in order to set the SASL_PATH
environment variable if SASL modules are present. Otherwise the
system modules were still used.
Change-Id: Id10afab6e9c48b9ffcf0da905993c7f2a1e606a6
Reviewed-on: http://gerrit.cloudera.org:8080/16716
Tested-by: Kudu Jenkins
Reviewed-by: Andrew Wong <aw...@cloudera.com>
Reviewed-by: Alexey Serbin <as...@cloudera.com>
(cherry picked from commit fab3a38d1da0d5d13d8e9c91fd306e03b65da4e6)
Reviewed-on: http://gerrit.cloudera.org:8080/16802
Reviewed-by: Grant Henke <gr...@apache.org>
Tested-by: Andrew Wong <aw...@cloudera.com>
---
build-support/dist_test.py | 63 +++++++++++++++++++++++++++++++++++++--
build-support/run_dist_test.py | 6 ++++
cmake_modules/FindCyrusSASL.cmake | 2 +-
3 files changed, 68 insertions(+), 3 deletions(-)
diff --git a/build-support/dist_test.py b/build-support/dist_test.py
index 1696ea8..89e53bf 100755
--- a/build-support/dist_test.py
+++ b/build-support/dist_test.py
@@ -240,6 +240,11 @@ def get_test_executions(tests_regex, extra_args=None):
def is_lib_whitelisted(lib):
# No need to ship things like libc, libstdcxx, etc.
if lib.startswith("/lib") or lib.startswith("/usr"):
+ # Ship the dynamically linked security libraries from
+ # OpenSSL and Cyrus SASL to better support submitting
+ # installed versions different from the dist_test image.
+ if "libcrypto" in lib or "libsasl2" in lib or "libssl" in lib:
+ return True
return False
return True
@@ -262,8 +267,41 @@ def get_base_deps(dep_extractor):
# of the test executable. We must include those dependencies in the archive
# for the binaries to be usable.
deps.extend(dep_extractor.extract_deps(d))
+
+ add_sasl_module_deps(deps)
return deps
+def add_sasl_module_deps(deps):
+ """
+ The SASL module dependencies are used at runtime but are not discovered
+ via ldd in the dep_extractor. This method finds the sasl2 directory
+ relative to the libsasl2 library and adds all the libraries in that
+ directory.
+ """
+ # Find the libsasl2 module in the dependencies.
+ sasl_lib = None
+ for dep in deps:
+ if "libsasl2" in dep:
+ sasl_lib = dep
+ break
+
+ # Look for libplain in potential sasl2 module paths, which is required for
+ # Kudu's basic operation.
+ sasl_path = None
+ if sasl_lib:
+ path = os.path.join(os.path.dirname(sasl_lib), "sasl2")
+ if os.path.exists(path):
+ children = os.listdir(path)
+ for child in children:
+ if "libplain" in child:
+ sasl_path = path
+ break
+
+ if sasl_path:
+ for dirpath, subdirs, files in os.walk(sasl_path):
+ for f in files:
+ dep = os.path.join(dirpath, f)
+ deps.append(dep)
def is_outside_of_tree(path):
repo_dir = rel_to_abs("./")
@@ -282,7 +320,18 @@ def copy_system_library(lib):
sys_lib_dir = rel_to_abs("build/dist-test-system-libs")
if not os.path.exists(sys_lib_dir):
os.makedirs(sys_lib_dir)
- dst = os.path.join(sys_lib_dir, os.path.basename(lib))
+
+ sasl_dir = os.path.join(sys_lib_dir, "sasl2")
+ if not os.path.exists(sasl_dir):
+ os.makedirs(sasl_dir)
+
+ # If the library is a SASL module keep it in its own directory so
+ # we can set the SASL_PATH environment variable in run_dist_test.py.
+ if "/sasl2/" in lib:
+ dst = os.path.join(sasl_dir, os.path.basename(lib))
+ else:
+ dst = os.path.join(sys_lib_dir, os.path.basename(lib))
+
# Copy if it doesn't exist, or the mtimes don't match.
# Using shutil.copy2 preserves the mtime after the copy (like cp -p)
if not os.path.exists(dst) or os.stat(dst).st_mtime != os.stat(lib).st_mtime:
@@ -664,7 +713,17 @@ def add_java_subparser(subparsers):
loop.set_defaults(func=loop_java_test)
def dump_base_deps(parser, options):
- print(json.dumps(get_base_deps(create_dependency_extractor())))
+ deps = get_base_deps(create_dependency_extractor())
+ relocated_deps = []
+ # Deduplicate dependencies included via DEPS_FOR_ALL.
+ for d in set(deps):
+ # System libraries will end up being relative paths out
+ # of the build tree. We need to copy those into the build
+ # tree somewhere.
+ if is_outside_of_tree(d):
+ d = copy_system_library(d)
+ relocated_deps.append(d)
+ print(json.dumps(relocated_deps))
def add_internal_commands(subparsers):
p = subparsers.add_parser('internal', help="[Internal commands not for users]")
diff --git a/build-support/run_dist_test.py b/build-support/run_dist_test.py
index cca701b..8d8a5a5 100755
--- a/build-support/run_dist_test.py
+++ b/build-support/run_dist_test.py
@@ -183,6 +183,12 @@ def main():
[os.path.join(ROOT, "build/dist-test-system-libs/")] +
glob.glob(os.path.abspath(os.path.join(ROOT, "build/*/lib"))))
+ # If SASL modules are included in the dist-test-system-libs, set the
+ # SASL_PATH environment variable to use them instead of the system ones.
+ sasl_dir = os.path.join(ROOT, "build/dist-test-system-libs/sasl2")
+ if os.path.exists(sasl_dir):
+ env['SASL_PATH'] = sasl_dir
+
# Don't pollute /tmp in dist-test setting. If a test crashes, the dist-test slave
# will clear up our working directory but won't be able to find and clean up things
# left in /tmp.
diff --git a/cmake_modules/FindCyrusSASL.cmake b/cmake_modules/FindCyrusSASL.cmake
index 8f92cf5..24192b7 100644
--- a/cmake_modules/FindCyrusSASL.cmake
+++ b/cmake_modules/FindCyrusSASL.cmake
@@ -24,7 +24,7 @@
#
# N.B: we do _not_ include sasl in thirdparty, for a fairly subtle reason. The
# TLDR version is that newer versions of cyrus-sasl (>=2.1.26) have a bug fix
-# for https://bugzilla.cyrusimap.org/show_bug.cgi?id=3590, but that bug fix
+# for https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=728332, but that bug fix
# relied on a change both on the plugin side and on the library side. If you
# then try to run the new version of sasl (e.g from our thirdparty tree) with
# an older version of a plugin (eg from RHEL6 install), you'll get a SASL_NOMECH
[kudu] 03/03: [test] Use /tmp for mini-chronyd sockets
Posted by aw...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
awong pushed a commit to branch branch-1.13.x
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit 2acc4d66d5fefff42f532b0e10899c66920c46f0
Author: Grant Henke <gr...@apache.org>
AuthorDate: Wed Dec 2 10:55:43 2020 -0600
[test] Use /tmp for mini-chronyd sockets
To ensure the path is less than 100 bytes this patch uses /tmp instead
of the test directory for chronyd sockets.
Change-Id: I567ff640fca5c56fef3f4400216fc8b23532713e
Reviewed-on: http://gerrit.cloudera.org:8080/16809
Reviewed-by: Alexey Serbin <as...@cloudera.com>
Tested-by: Andrew Wong <aw...@cloudera.com>
---
src/kudu/clock/test/mini_chronyd.cc | 7 +++----
1 file changed, 3 insertions(+), 4 deletions(-)
diff --git a/src/kudu/clock/test/mini_chronyd.cc b/src/kudu/clock/test/mini_chronyd.cc
index 0d2cf78..5f4cf6d 100644
--- a/src/kudu/clock/test/mini_chronyd.cc
+++ b/src/kudu/clock/test/mini_chronyd.cc
@@ -422,10 +422,8 @@ $0
// The path to Unix domain socket file cannot be longer than ~100 bytes,
// so it's necessary to create a directory with shorter absolute path.
// TODO(aserbin): use some synthetic mount point instead?
- string dir;
- RETURN_NOT_OK(Env::Default()->GetTestDirectory(&dir));
- dir = JoinPathSegments(dir, Substitute("$0.$1", Env::Default()->NowMicros(),
- getpid()));
+ string dir = JoinPathSegments("/tmp", Substitute("$0.$1",
+ Env::Default()->NowMicros(), getpid()));
const auto s = Env::Default()->CreateDir(dir);
if (!s.ok() && !s.IsAlreadyPresent()) {
return s;
@@ -433,6 +431,7 @@ $0
RETURN_NOT_OK(CorrectOwnership(dir));
options_.bindcmdaddress = Substitute("$0/chronyd.$1.sock",
dir, options_.index);
+ // Set cmd_socket_dir_ so we can cleanup the unix domain sockets.
cmd_socket_dir_ = std::move(dir);
}
string username;