You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kudu.apache.org by ad...@apache.org on 2016/11/09 23:49:32 UTC

[1/3] kudu git commit: Add a macro to LOG and return on a non-OK status

Repository: kudu
Updated Branches:
  refs/heads/master 29f1543cc -> af9fc13cf


Add a macro to LOG and return on a non-OK status

We often see the following pattern:

Status s = someOperation();

if (!s.ok()) {
  LOG(ERROR) << "A critical error occurred at blah. Status: " << s.ToString();
  return s;
}

This is cumbersome. This macro allows to do the same the following way:

RETURN_NOT_OK_LOG(someOperation(), ERROR, "A critical error occurred at blah.");

The macro also prints the status (though at the beginning of the
message and not at the end so that we can have arbirarily long
debugging information but the status is easy to find).

Change-Id: Ibb906e7240d7a289a7822444bc9cbf748a555efb
Reviewed-on: http://gerrit.cloudera.org:8080/4927
Reviewed-by: Alexey Serbin <as...@cloudera.com>
Tested-by: Kudu Jenkins


Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/dbac80fd
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/dbac80fd
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/dbac80fd

Branch: refs/heads/master
Commit: dbac80fd373c782d60f7dbb031832dc81f5024a2
Parents: 29f1543
Author: David Alves <dr...@apache.org>
Authored: Thu Oct 27 17:52:35 2016 -0700
Committer: David Ribeiro Alves <dr...@apache.org>
Committed: Wed Nov 9 22:28:24 2016 +0000

----------------------------------------------------------------------
 src/kudu/util/status.h | 10 ++++++++++
 1 file changed, 10 insertions(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/kudu/blob/dbac80fd/src/kudu/util/status.h
----------------------------------------------------------------------
diff --git a/src/kudu/util/status.h b/src/kudu/util/status.h
index af02c73..289d4e1 100644
--- a/src/kudu/util/status.h
+++ b/src/kudu/util/status.h
@@ -62,6 +62,15 @@
     return _s; \
   } while (0);
 
+/// @brief If the given status is not OK, log it and 'msg' at 'level' and return the status.
+#define KUDU_RETURN_NOT_OK_LOG(s, level, msg) do { \
+    const ::kudu::Status& _s = (s);             \
+    if (PREDICT_FALSE(!_s.ok())) { \
+      KUDU_LOG(level) << "Status: " << _s.ToString() << " " << (msg); \
+      return _s;     \
+    } \
+  } while (0);
+
 /// @brief If @c to_call returns a bad status, CHECK immediately with
 ///   a logged message of @c msg followed by the status.
 #define KUDU_CHECK_OK_PREPEND(to_call, msg) do { \
@@ -92,6 +101,7 @@
 #define RETURN_NOT_OK_RET     KUDU_RETURN_NOT_OK_RET
 #define WARN_NOT_OK           KUDU_WARN_NOT_OK
 #define LOG_AND_RETURN        KUDU_LOG_AND_RETURN
+#define RETURN_NOT_OK_LOG     KUDU_RETURN_NOT_OK_LOG
 #define CHECK_OK_PREPEND      KUDU_CHECK_OK_PREPEND
 #define CHECK_OK              KUDU_CHECK_OK
 


[2/3] kudu git commit: C++ client: hide deflate symbols

Posted by ad...@apache.org.
C++ client: hide deflate symbols

These snuck in via commit 8a4bb33. Oddly, client_symbol-test.sh was only
observed to be failing on el6 (not on Ubuntu 14.04 or 16.04). The running
theory is it's because the devtoolset linker is old enough that it can't
figure out that these symbols aren't callable by any client code path.

Change-Id: Ibb5b6cf186a9a18583c4695b134c0729b81a4c01
Reviewed-on: http://gerrit.cloudera.org:8080/5023
Reviewed-by: Todd Lipcon <to...@apache.org>
Tested-by: Kudu Jenkins


Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/388908c0
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/388908c0
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/388908c0

Branch: refs/heads/master
Commit: 388908c0d5f2fc810a92ccc3ab0adca17f79358e
Parents: dbac80f
Author: Adar Dembo <ad...@cloudera.com>
Authored: Wed Nov 9 14:49:50 2016 -0800
Committer: Adar Dembo <ad...@cloudera.com>
Committed: Wed Nov 9 23:48:55 2016 +0000

----------------------------------------------------------------------
 src/kudu/client/symbols.map | 1 +
 1 file changed, 1 insertion(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/kudu/blob/388908c0/src/kudu/client/symbols.map
----------------------------------------------------------------------
diff --git a/src/kudu/client/symbols.map b/src/kudu/client/symbols.map
index 7c6db4d..fe11596 100644
--- a/src/kudu/client/symbols.map
+++ b/src/kudu/client/symbols.map
@@ -14,6 +14,7 @@
     # zlib
     adler32*;
     crc32*;
+    deflate*;
     get_crc_table;
     inflate*;
     zError;


[3/3] kudu git commit: log block manager: fix when append exceeds preallocated space

Posted by ad...@apache.org.
log block manager: fix when append exceeds preallocated space

In commit abea8c6 I added a DCHECK to enforce the assumption that a
container's preallocation offset is always ahead of its append offset. This
turns out to be wrong when very large values are written, or when the
preallocation size is tuned down (as it is in disk_reservation-itest.cc).
The effect: disk_reservation-itest.cc failed from time to time because the
tserver crashed too early.

So let's throw that away, and let's also adjust the preallocation algorithm
slightly: when we preallocate, we take the write offset into account. If
it's greater than the preallocation offset, we preallocate again, and we do
so from the write offset.

The new test crashes 100% of the time without the fix.

Change-Id: I99460d35e355abf843c965d3c2b4309cba24eadd
Reviewed-on: http://gerrit.cloudera.org:8080/5024
Reviewed-by: Jean-Daniel Cryans <jd...@apache.org>
Tested-by: Kudu Jenkins


Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/af9fc13c
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/af9fc13c
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/af9fc13c

Branch: refs/heads/master
Commit: af9fc13cfc277faa0078b2502f4baa25fd0d9102
Parents: 388908c
Author: Adar Dembo <ad...@cloudera.com>
Authored: Wed Nov 9 14:22:25 2016 -0800
Committer: Adar Dembo <ad...@cloudera.com>
Committed: Wed Nov 9 23:48:59 2016 +0000

----------------------------------------------------------------------
 src/kudu/fs/block_manager-test.cc | 19 +++++++++++++++++++
 src/kudu/fs/log_block_manager.cc  | 18 ++++++++++--------
 2 files changed, 29 insertions(+), 8 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/kudu/blob/af9fc13c/src/kudu/fs/block_manager-test.cc
----------------------------------------------------------------------
diff --git a/src/kudu/fs/block_manager-test.cc b/src/kudu/fs/block_manager-test.cc
index 9aeb29e..fc65fd1 100644
--- a/src/kudu/fs/block_manager-test.cc
+++ b/src/kudu/fs/block_manager-test.cc
@@ -1007,6 +1007,25 @@ TEST_F(LogBlockManagerTest, TestMetadataTruncation) {
                                      false));
 }
 
+// Regression test for a crash when a container's append offset exceeded its
+// preallocation offset.
+TEST_F(LogBlockManagerTest, TestAppendExceedsPreallocation) {
+  RETURN_NOT_LOG_BLOCK_MANAGER();
+
+  FLAGS_log_container_preallocate_bytes = 1;
+
+  // Create a container, preallocate it by one byte, and append more than one.
+  gscoped_ptr<WritableBlock> writer;
+  ASSERT_OK(bm_->CreateBlock(&writer));
+  ASSERT_OK(writer->Append("hello world"));
+  ASSERT_OK(writer->Close());
+
+  // On second append, don't crash just because the append offset is ahead of
+  // the preallocation offset!
+  ASSERT_OK(bm_->CreateBlock(&writer));
+  ASSERT_OK(writer->Append("hello world"));
+}
+
 TYPED_TEST(BlockManagerTest, TestDiskSpaceCheck) {
   // Reopen the block manager with metrics enabled.
   MetricRegistry registry;

http://git-wip-us.apache.org/repos/asf/kudu/blob/af9fc13c/src/kudu/fs/log_block_manager.cc
----------------------------------------------------------------------
diff --git a/src/kudu/fs/log_block_manager.cc b/src/kudu/fs/log_block_manager.cc
index 129766b..91609da 100644
--- a/src/kudu/fs/log_block_manager.cc
+++ b/src/kudu/fs/log_block_manager.cc
@@ -576,8 +576,10 @@ Status LogBlockContainer::AppendData(const Slice& data) {
   }
   total_bytes_written_ += data.size();
 
-  if (PREDICT_FALSE(!FLAGS_log_container_preallocate_bytes)) {
-    // Without preallocation, every call grows the container.
+  // This append may have changed the container size if:
+  // 1. It was large enough that it blew out the preallocated space.
+  // 2. Preallocation was disabled.
+  if (total_bytes_written_ > preallocated_offset_) {
     RETURN_NOT_OK(data_dir_->RefreshIsFull(DataDir::RefreshMode::ALWAYS));
   }
   return Status::OK();
@@ -630,12 +632,12 @@ Status LogBlockContainer::EnsurePreallocated(size_t next_append_bytes) {
   if (!FLAGS_log_container_preallocate_bytes) {
     return Status::OK();
   }
-  DCHECK_GE(preallocated_offset_, total_bytes_written_);
 
-  // If the next write exceeds the remaining preallocated window, we need to
-  // preallocate another chunk.
-  if (next_append_bytes > preallocated_offset_ - total_bytes_written_) {
-    int64_t off = preallocated_offset_;
+  // If the last write blew out the preallocation window, or if the next write
+  // exceeds it, we need to preallocate another chunk.
+  if (total_bytes_written_ > preallocated_offset_ ||
+      next_append_bytes > preallocated_offset_ - total_bytes_written_) {
+    int64_t off = std::max(preallocated_offset_, total_bytes_written_);
     int64_t len = FLAGS_log_container_preallocate_bytes;
     {
       std::lock_guard<Mutex> l(data_file_lock_);
@@ -645,7 +647,7 @@ Status LogBlockContainer::EnsurePreallocated(size_t next_append_bytes) {
     VLOG(2) << Substitute("Preallocated $0 bytes at offset $1 in container $2",
                           len, off, ToString());
 
-    preallocated_offset_ += len;
+    preallocated_offset_ = off + len;
   }
 
   return Status::OK();