You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kudu.apache.org by to...@apache.org on 2016/01/26 18:13:59 UTC
[1/4] incubator-kudu git commit: Use 'kqueue' instead of 'select' for
the rpc event loop in Mac OS X
Repository: incubator-kudu
Updated Branches:
refs/heads/master a724818c5 -> f09025432
Use 'kqueue' instead of 'select' for the rpc event loop in Mac OS X
It appears that, in Mac OS X, 'select' can't handle connections when 1024 or
more file descriptors are open. This breaks rpc-test consistently and makes
tablet servers crash when serving a lot of/large tablets in OS X.
This patch makes it so we use 'kqueue' in OS X, which doesn't have the same
problem.
I tested this on my local laptop, rpc-test failed before and passes now.
Change-Id: I8daa3d02c7e28952eecd9e2ce56cadb8b14c418c
Reviewed-on: http://gerrit.cloudera.org:8080/1897
Reviewed-by: Todd Lipcon <to...@apache.org>
Tested-by: Kudu Jenkins
Project: http://git-wip-us.apache.org/repos/asf/incubator-kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-kudu/commit/21df2bab
Tree: http://git-wip-us.apache.org/repos/asf/incubator-kudu/tree/21df2bab
Diff: http://git-wip-us.apache.org/repos/asf/incubator-kudu/diff/21df2bab
Branch: refs/heads/master
Commit: 21df2bab4c2ba3101fa00571745386f07627964f
Parents: a724818
Author: David Alves <da...@cloudera.com>
Authored: Mon Jan 25 16:36:01 2016 -0800
Committer: David Ribeiro Alves <da...@cloudera.com>
Committed: Tue Jan 26 04:20:54 2016 +0000
----------------------------------------------------------------------
src/kudu/rpc/reactor.cc | 11 ++++++++++-
1 file changed, 10 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-kudu/blob/21df2bab/src/kudu/rpc/reactor.cc
----------------------------------------------------------------------
diff --git a/src/kudu/rpc/reactor.cc b/src/kudu/rpc/reactor.cc
index 40cb112..2e6c4db 100644
--- a/src/kudu/rpc/reactor.cc
+++ b/src/kudu/rpc/reactor.cc
@@ -51,6 +51,15 @@
#include "kudu/util/status.h"
#include "kudu/util/net/socket.h"
+// When compiling on Mac OS X, use 'kqueue' instead of the default, 'select', for the event loop.
+// Otherwise we run into problems because 'select' can't handle connections when more than 1024
+// file descriptors are open by the process.
+#if defined(__APPLE__)
+static const int kDefaultLibEvFlags = ev::KQUEUE;
+#else
+static const int kDefaultLibEvFlags = ev::AUTO;
+#endif
+
using std::string;
using std::shared_ptr;
@@ -72,7 +81,7 @@ Status ShutdownError(bool aborted) {
} // anonymous namespace
ReactorThread::ReactorThread(Reactor *reactor, const MessengerBuilder &bld)
- : loop_(0),
+ : loop_(kDefaultLibEvFlags),
cur_time_(MonoTime::Now(MonoTime::COARSE)),
last_unused_tcp_scan_(cur_time_),
reactor_(reactor),
[4/4] incubator-kudu git commit: KUDU-1316. Fix parsing of available
CPUs on single-core machines
Posted by to...@apache.org.
KUDU-1316. Fix parsing of available CPUs on single-core machines
This also changes the startup process to FATAL if procfs is unavailable.
Previously, we could continue but would probably just crash in the
parts of code where we depend on the CPU count.
Change-Id: I0dda1aab4735edeb4ebff901db627427d44ef555
Reviewed-on: http://gerrit.cloudera.org:8080/1908
Reviewed-by: Jean-Daniel Cryans
Tested-by: Kudu Jenkins
Project: http://git-wip-us.apache.org/repos/asf/incubator-kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-kudu/commit/f0902543
Tree: http://git-wip-us.apache.org/repos/asf/incubator-kudu/tree/f0902543
Diff: http://git-wip-us.apache.org/repos/asf/incubator-kudu/diff/f0902543
Branch: refs/heads/master
Commit: f090254322c5445f0d1d7098cadb279ad78250e0
Parents: 242e5da
Author: Todd Lipcon <to...@apache.org>
Authored: Tue Jan 26 08:46:18 2016 -0800
Committer: Todd Lipcon <to...@apache.org>
Committed: Tue Jan 26 17:13:16 2016 +0000
----------------------------------------------------------------------
src/kudu/gutil/sysinfo.cc | 14 ++++++++------
1 file changed, 8 insertions(+), 6 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-kudu/blob/f0902543/src/kudu/gutil/sysinfo.cc
----------------------------------------------------------------------
diff --git a/src/kudu/gutil/sysinfo.cc b/src/kudu/gutil/sysinfo.cc
index 73afa27..6006b6d 100644
--- a/src/kudu/gutil/sysinfo.cc
+++ b/src/kudu/gutil/sysinfo.cc
@@ -148,7 +148,13 @@ static bool ReadIntFromFile(const char *file, int *value) {
static int ReadMaxCPUIndex() {
char buf[1024];
CHECK(SlurpSmallTextFile("/sys/devices/system/cpu/present", buf, arraysize(buf)));
- // 'buf' should contain a string like '0-7'.
+
+ // On a single-core machine, 'buf' will contain the string '0' with a newline.
+ if (strcmp(buf, "0\n") == 0) {
+ return 0;
+ }
+
+ // On multi-core, it will have a CPU range like '0-7'.
CHECK_EQ(0, memcmp(buf, "0-", 2)) << "bad list of possible CPUs: " << buf;
char* max_cpu_str = &buf[2];
@@ -216,11 +222,7 @@ static void InitializeSystemInfo() {
const char* pname = "/proc/cpuinfo";
int fd = open(pname, O_RDONLY);
if (fd == -1) {
- perror(pname);
- if (!saw_mhz) {
- cpuinfo_cycles_per_second = EstimateCyclesPerSecond(1000);
- }
- return; // TODO: use generic tester instead?
+ PLOG(FATAL) << "Unable to read CPU info from /proc. procfs must be mounted.";
}
double bogo_clock = 1.0;
[3/4] incubator-kudu git commit: client-stress-test: fix TSAN build
flakiness
Posted by to...@apache.org.
client-stress-test: fix TSAN build flakiness
On TSAN builds, this has been failing reasonably often on GCE slaves.
It seems like the new slaves just run a bit slower than the old ones,
and therefore we need to make the test a little more lenient in TSAN
builds.
Change-Id: I5e9beec3a8a75cf5a83c8ec3ae6e04396303a662
Reviewed-on: http://gerrit.cloudera.org:8080/1893
Tested-by: Todd Lipcon <to...@apache.org>
Reviewed-by: Jean-Daniel Cryans
Project: http://git-wip-us.apache.org/repos/asf/incubator-kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-kudu/commit/242e5da4
Tree: http://git-wip-us.apache.org/repos/asf/incubator-kudu/tree/242e5da4
Diff: http://git-wip-us.apache.org/repos/asf/incubator-kudu/diff/242e5da4
Branch: refs/heads/master
Commit: 242e5da4652b0a8808240278d5a86556d4e59164
Parents: 3df123a
Author: Todd Lipcon <to...@apache.org>
Authored: Mon Jan 25 11:58:27 2016 -0800
Committer: Jean-Daniel Cryans <jd...@gerrit.cloudera.org>
Committed: Tue Jan 26 17:05:06 2016 +0000
----------------------------------------------------------------------
src/kudu/integration-tests/client-stress-test.cc | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-kudu/blob/242e5da4/src/kudu/integration-tests/client-stress-test.cc
----------------------------------------------------------------------
diff --git a/src/kudu/integration-tests/client-stress-test.cc b/src/kudu/integration-tests/client-stress-test.cc
index 4a6e80e..9be5f71 100644
--- a/src/kudu/integration-tests/client-stress-test.cc
+++ b/src/kudu/integration-tests/client-stress-test.cc
@@ -222,7 +222,14 @@ class ClientStressTest_LowMemory : public ClientStressTest {
// Stress test where, due to absurdly low memory limits, many client requests
// are rejected, forcing the client to retry repeatedly.
TEST_F(ClientStressTest_LowMemory, TestMemoryThrottling) {
+#ifdef THREAD_SANITIZER
+ // TSAN tests run much slower, so we don't want to wait for as many
+ // rejections before declaring the test to be passed.
+ const int64_t kMinRejections = 20;
+#else
const int64_t kMinRejections = 100;
+#endif
+
const MonoDelta kMaxWaitTime = MonoDelta::FromSeconds(60);
TestWorkload work(cluster_.get());
@@ -265,7 +272,8 @@ TEST_F(ClientStressTest_LowMemory, TestMemoryThrottling) {
if (total_num_rejections >= kMinRejections) {
break;
} else if (deadline.ComesBefore(MonoTime::Now(MonoTime::FINE))) {
- FAIL() << "Ran for " << kMaxWaitTime.ToString() << ", deadline expired";
+ FAIL() << "Ran for " << kMaxWaitTime.ToString() << ", deadline expired and only saw "
+ << total_num_rejections << " memory rejections";
}
SleepFor(MonoDelta::FromMilliseconds(200));
}
[2/4] incubator-kudu git commit: KUDU-1305: do not allow building
with gcc 4.6 and 4.7
Posted by to...@apache.org.
KUDU-1305: do not allow building with gcc 4.6 and 4.7
They produce broken code in release mode ala KUDU-1030.
I tested this by running cmake with gcc 4.7, 4.8, and 5.2.
Change-Id: I0a6c3724f19cac207787b00e3b471f982f620c50
Reviewed-on: http://gerrit.cloudera.org:8080/1904
Tested-by: Kudu Jenkins
Reviewed-by: Todd Lipcon <to...@apache.org>
Project: http://git-wip-us.apache.org/repos/asf/incubator-kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-kudu/commit/3df123ac
Tree: http://git-wip-us.apache.org/repos/asf/incubator-kudu/tree/3df123ac
Diff: http://git-wip-us.apache.org/repos/asf/incubator-kudu/diff/3df123ac
Branch: refs/heads/master
Commit: 3df123ac7f71b44f64393c6637cebfffa38e623d
Parents: 21df2ba
Author: Adar Dembo <ad...@cloudera.com>
Authored: Mon Jan 25 18:53:35 2016 -0800
Committer: Todd Lipcon <to...@apache.org>
Committed: Tue Jan 26 16:34:14 2016 +0000
----------------------------------------------------------------------
CMakeLists.txt | 9 +++++++++
1 file changed, 9 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-kudu/blob/3df123ac/CMakeLists.txt
----------------------------------------------------------------------
diff --git a/CMakeLists.txt b/CMakeLists.txt
index 81b56be..6bdf7e0 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -181,6 +181,15 @@ if ("${COMPILER_FAMILY}" STREQUAL "clang")
else()
message("Running without a controlling terminal or in a dumb terminal")
endif()
+elseif("${COMPILER_FAMILY}" STREQUAL "gcc")
+ # Blacklist gcc versions known to generate broken optimized code.
+ #
+ # See KUDU-1030 for more details.
+ if ("${COMPILER_VERSION}" MATCHES "^4.[67]")
+ message(FATAL_ERROR "Building with gcc version ${COMPILER_VERSION} is "
+ "forbidden as it is known to produce broken code in release mode. "
+ "Upgrade gcc, or build with clang from the thirdparty directory.")
+ endif()
endif()
# Sanity check linking option.