You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kudu.apache.org by ad...@apache.org on 2019/03/21 01:05:15 UTC
[kudu] branch master updated (39f4637 -> 28c7067)
This is an automated email from the ASF dual-hosted git repository.
adar pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git.
from 39f4637 [util] added cache with FIFO eviction policy
new 7a8c210 [macos] fix build
new 28c7067 KUDU-2748 Leader master erroneously tries to tablet copy to a follower master due to race at startup
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
src/kudu/cfile/block_cache.cc | 4 ++++
src/kudu/tserver/tablet_service.cc | 11 +++++++++--
src/kudu/util/cache-test.cc | 4 ++++
3 files changed, 17 insertions(+), 2 deletions(-)
[kudu] 02/02: KUDU-2748 Leader master erroneously tries to tablet
copy to a follower master due to race at startup
Posted by ad...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
adar pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit 28c706722891d20aada5d8bee4cfafe456c89561
Author: Will Berkeley <wd...@gmail.com>
AuthorDate: Fri Mar 15 14:38:38 2019 -0700
KUDU-2748 Leader master erroneously tries to tablet copy to a follower master due to race at startup
The initialization of the master works as follows:
1. Register RPC services.
2. Init catalog manager asynchronously.
As a result, if a master in a multimaster cluster with a healthy leader
starts, there is a brief period of time when a call to UpdateConsensus
from the leader master will hit a CatalogManager and SysTable that are
not initialized. The initializing master will respond TABLET_NOT_FOUND
to the leader, which will cause the leader master to initiate the tablet
copy process. This is a dead end because masters don't support tablet
copy. Things are stuck until there is a leadership change or the
"orphaned" master is restarted again.
Tablets on tablet servers are not vulnerable to this because their
startup order is
1. Init the ts tablet manager synchronously.
2. Register RPC services.
So it is not possible for an UpdateConsensus call to query a ts tablet
manager that hasn't loaded all of the initial tablets.
The fix is pretty simple: recognize and return the StatusUnavailable
returned by the tablet lookup for the master tablet, instead of
TABLET_NOT_FOUND. This will cause the leader master to retry until the
initializing master has finished initializing.
This was the cause of flakiness in KUDU-2734. Without the fix, about 8%
of runs fail on TSAN with 8 stress threads. With the fix, about 0.3% do
(and in 2000 runs with 6 failures I verified that none of the 6 were due
to this issue).
Change-Id: Ib86548085e45ed5cd987d99e227a1af84bf801e7
Reviewed-on: http://gerrit.cloudera.org:8080/12770
Tested-by: Kudu Jenkins
Reviewed-by: Adar Dembo <ad...@cloudera.com>
Reviewed-by: Grant Henke <gr...@apache.org>
Reviewed-by: Alexey Serbin <as...@cloudera.com>
---
src/kudu/tserver/tablet_service.cc | 11 +++++++++--
1 file changed, 9 insertions(+), 2 deletions(-)
diff --git a/src/kudu/tserver/tablet_service.cc b/src/kudu/tserver/tablet_service.cc
index 5d89ea1..010b35b 100644
--- a/src/kudu/tserver/tablet_service.cc
+++ b/src/kudu/tserver/tablet_service.cc
@@ -230,8 +230,15 @@ bool LookupTabletReplicaOrRespond(TabletReplicaLookupIf* tablet_manager,
scoped_refptr<TabletReplica>* replica) {
Status s = tablet_manager->GetTabletReplica(tablet_id, replica);
if (PREDICT_FALSE(!s.ok())) {
- SetupErrorAndRespond(resp->mutable_error(), s,
- TabletServerErrorPB::TABLET_NOT_FOUND, context);
+ if (s.IsServiceUnavailable()) {
+ // If the tablet manager isn't initialized, the remote should check again
+ // soon.
+ SetupErrorAndRespond(resp->mutable_error(), s,
+ TabletServerErrorPB::UNKNOWN_ERROR, context);
+ } else {
+ SetupErrorAndRespond(resp->mutable_error(), s,
+ TabletServerErrorPB::TABLET_NOT_FOUND, context);
+ }
return false;
}
return true;
[kudu] 01/02: [macos] fix build
Posted by ad...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
adar pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit 7a8c21048f0aad9ecab686782c60eeb4985643e8
Author: Alexey Serbin <al...@apache.org>
AuthorDate: Wed Mar 20 16:12:24 2019 -0700
[macos] fix build
This is a follow-up to 39f4637559ac29791cc385308bd63d77a5f18e56.
Change-Id: I6566d08de063676a8f810b63bcb8dfc7863c090b
Reviewed-on: http://gerrit.cloudera.org:8080/12818
Reviewed-by: Adar Dembo <ad...@cloudera.com>
Tested-by: Kudu Jenkins
---
src/kudu/cfile/block_cache.cc | 4 ++++
src/kudu/util/cache-test.cc | 4 ++++
2 files changed, 8 insertions(+)
diff --git a/src/kudu/cfile/block_cache.cc b/src/kudu/cfile/block_cache.cc
index 32587c4..bddaa08 100644
--- a/src/kudu/cfile/block_cache.cc
+++ b/src/kudu/cfile/block_cache.cc
@@ -76,8 +76,12 @@ Cache* CreateCache(int64_t capacity) {
return NewCache<Cache::EvictionPolicy::LRU, Cache::MemoryType::DRAM>(
capacity, "block_cache");
case Cache::MemoryType::NVM:
+#if defined(HAVE_LIB_VMEM)
return NewCache<Cache::EvictionPolicy::LRU, Cache::MemoryType::NVM>(
capacity, "block_cache");
+#else
+ LOG(FATAL) << "cache of NVM memory type is not supported";
+#endif // #if defined(HAVE_LIB_VMEM) ... #else ...
default:
LOG(FATAL) << "unsupported LRU cache memory type: " << mem_type;
return nullptr;
diff --git a/src/kudu/util/cache-test.cc b/src/kudu/util/cache-test.cc
index 6869296..5c2b526 100644
--- a/src/kudu/util/cache-test.cc
+++ b/src/kudu/util/cache-test.cc
@@ -140,9 +140,13 @@ class CacheBaseTest : public KuduTest,
"cache_test"));
break;
case Cache::MemoryType::NVM:
+#if defined(HAVE_LIB_VMEM)
cache_.reset(NewCache<Cache::EvictionPolicy::LRU,
Cache::MemoryType::NVM>(cache_size(),
"cache_test"));
+#else
+ FAIL() << "cache of NVM memory type is not supported";
+#endif // #if defined(HAVE_LIB_VMEM) ... #else ...
break;
default:
FAIL() << mem_type << ": unrecognized cache memory type";