You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kudu.apache.org by al...@apache.org on 2018/03/20 17:46:10 UTC
[1/5] kudu git commit: Minor updates to RELEASING.adoc
Repository: kudu
Updated Branches:
refs/heads/master 9f6b7f168 -> 5d7b68d59
Minor updates to RELEASING.adoc
A few minor updates to RELEASING.adoc while going
through the process:
* Update asf git repo
* Update gerrit ssh steps
* Add link to Apache GPG guide
* Adjust staging and release svn commands
* Fixed doc site links
Change-Id: Ie03c38126cb96e0dd30aa56cbbe0f42843e15e7d
Reviewed-on: http://gerrit.cloudera.org:8080/9692
Tested-by: Kudu Jenkins
Reviewed-by: Dan Burkert <da...@apache.org>
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/1bfb0c02
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/1bfb0c02
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/1bfb0c02
Branch: refs/heads/master
Commit: 1bfb0c025fcdf5497006ce67a21aed75238d13cf
Parents: 9f6b7f1
Author: Grant Henke <gr...@apache.org>
Authored: Thu Mar 15 21:36:57 2018 -0500
Committer: Grant Henke <gr...@apache.org>
Committed: Mon Mar 19 16:09:59 2018 +0000
----------------------------------------------------------------------
RELEASING.adoc | 48 ++++++++++++++++++++++++++++++------------------
1 file changed, 30 insertions(+), 18 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/kudu/blob/1bfb0c02/RELEASING.adoc
----------------------------------------------------------------------
diff --git a/RELEASING.adoc b/RELEASING.adoc
index 3646069..4857927 100644
--- a/RELEASING.adoc
+++ b/RELEASING.adoc
@@ -50,7 +50,7 @@ in `master`.
----
. Push the branch to public remotes https://github.com/cloudera/kudu.git and
-http://git-wip-us.apache.org/repos/asf?p=kudu.git. The following example
+https://git-wip-us.apache.org/repos/asf/kudu.git. The following example
assumes they are called `cloudera` and `apache`.
+
----
@@ -67,15 +67,17 @@ branch with the same name and the previously-noted SHA1.
have to perform this step because SSH access is behind a firewall. The steps
are as follows:
1. Ensure your public SSH key is in `~gerrit/.ssh/authorized_keys` on gerrit.cloudera.org
- 2. From behind the firewall, `ssh gerrit@gerrit.cloudera.org` to log in.
- 3. Back up the existing replication configuration file by executing
- `cp etc/replication.config etc/replication.config.bak.\`date '+%Y%m%d.%H%M%S'\``
- 4. Edit `etc/replication.config` to add a line for the new branch, such as `branch-1.x.y`
- 5. Send email to the dev lists for Kudu and Impala (dev@kudu.apache.org and
+ 2. From behind the firewall, `ssh gerrit.cloudera.org` to log in.
+ 3. Change to the gerrit user, `sudo su gerrit`
+ 4. Back up the existing replication configuration file by executing
+ `cp ~/etc/replication.config ~/etc/replication.config.bak.`date '+%Y%m%d.%H%M%S'``
+ 5. Edit `etc/replication.config` to add a line for the new branch, such as `branch-1.x.y`
+ 6. Send email to the dev lists for Kudu and Impala (dev@kudu.apache.org and
dev@impala.apache.org) indicating that you are going to restart Gerrit
(link:https://s.apache.org/2Wj7[example]). It is best to do the restart at
some time of day when you don't expect many people to be using the system,
since Gerrit can take a few minutes to restart.
+ 7: Restart Gerrit, `~/bin/gerrit.sh restart`
. As needed, patches can be cherry-picked to the new branch.
@@ -102,6 +104,10 @@ branch with the same name and the previously-noted SHA1.
== Preparing A Release Candidate
+. Before building a release candidate, make sure you have followed the
+Apache committer guide for setting up your GPG keys
+(link:https://www.apache.org/dev/new-committers-guide.html#set-up-security-and-pgp-keys[here]).
+
. When close to building a release candidate, try building a source tarball
(on a supported platform):
+
@@ -154,8 +160,8 @@ you can link:http://maven.apache.org/guides/mini/guide-encryption.html[encrypt i
. When ready, create a new lightweight tag and push it to the Apache Git repository.
+
----
- git tag 1.x.0-RC1
- git push apache 1.x.0-RC1
+ git tag 1.x.y-RC1
+ git push apache 1.x.y-RC1
----
. Build a source tarball against the RC branch.
@@ -164,8 +170,12 @@ you can link:http://maven.apache.org/guides/mini/guide-encryption.html[encrypt i
Copy the artifacts to this folder and commit (it is Subversion).
+
----
- svn co --depth=immediates https://dist.apache.org/repos/dist/dev/kudu/
- svn commit -m "Adding Kudu 1.x.0 RC1"
+ svn co --depth=immediates https://dist.apache.org/repos/dist/dev/kudu/ kudu-dev-release
+ cd kudu-dev-release
+ mkdir 1.x.y-RC1
+ cp <path_to_kudu>/build/apache-kudu-1.x.y.tar.* 1.x.y-RC1
+ svn add 1.x.y-RC1/*
+ svn commit -m "Adding Kudu 1.x.y RC1"
----
. Create a Maven staging repository for the RC.
@@ -180,10 +190,10 @@ you can link:http://maven.apache.org/guides/mini/guide-encryption.html[encrypt i
Go to the link:https://repository.apache.org/\#stagingRepositories[staging repository]
and look for ‘orgapachekudu-####’ in the staging repositories list. You can
check the ‘content’ tab at the bottom to make sure you have all of the expected
-stuff (client, various integrations, both versions of Spark) Hit the checkbox
-next to your new staging repo and hit “close”. Enter “Apache Kudu 1.1.0-RC1” or
-whatever into that box. Wait a minute or two and hit refresh, and your staging
-repo should now have a URL shown in its summary tab (eg
+stuff (client, various integrations, etc). Hit the checkbox next to your new
+staging repo and hit “close”. Enter something similar to “Apache Kudu 1.x.y-RC1”
+into the description box and confirm. Wait a minute or two and hit refresh, and
+your staging repo should now have a URL shown in its summary tab (eg
`https://repository.apache.org/content/repositories/orgapachekudu-1005`)
. Add your PGP key to the KEYS file:
@@ -218,11 +228,11 @@ svn commit -m "Adding my key to the KEYS file"
copy the files from the release candidate’s folder in dist/dev/kudu/, then commit.
+
----
- cd kudu
+ cd kudu-dist-release
mkdir 1.x.y
cp <path_to_rc_artifacts>/* 1.x.y
svn add 1.x.y
- svn commit -m "Adding files for Kudu 1.x.y RC"
+ svn commit -m "Adding files for Kudu 1.x.y"
----
. In the Kudu git repo, create a signed tag from the RC’s tag, and push it to the
@@ -238,7 +248,7 @@ Apache Git repository:
and click 'Release'.
. Generate the version-specific documentation from that branch following these
- link:https://github.com/apache/kudu/#updating-the-documentation-on-the-kudu-web-site[instructions].
+ link:https://github.com/apache/kudu/#updating-the-kudu-web-site-documentation[instructions].
. Replace the `apidocs` and `docs` folders in the `gh-pages` branch with the new documentation.
@@ -250,7 +260,8 @@ Apache Git repository:
. Create a new review for all those new and updated files in `gh-pages`.
-. Once the review is finished and the commit is pushed, update the website.
+. Once the review is finished and the commit is pushed, update the website following
+ these link:https://github.com/apache/kudu/#deploying-changes-to-the-apache-kudu-web-site[instructions].
. About 24 hours after the first step was completed, send an email to
user@kudu.apache.org, dev@kudu.apache.org, and announce@apache.org
@@ -259,6 +270,7 @@ Apache Git repository:
. About another 24 hours later, delete the previous minor version in the branch
you released from, from SVN. For example, if you released 1.2.1, delete `1.2.0`.
+ Also delete any release candidates from the dev SVN.
. Update the version number on the branch you released from back to a SNAPSHOT
for the next patch release, such as `1.6.1-SNAPSHOT` after the `1.6.0` release.
[3/5] kudu git commit: [rpc-test] macOS-specific fix for
TestRpcSidecarLimits
Posted by al...@apache.org.
[rpc-test] macOS-specific fix for TestRpcSidecarLimits
As Will noticed, after TestRpcSidecarLimits was updated with 2b0c1c0,
on macOS the following new error surfaced while running the test
(which is completely legit in accordance with 'man sendmsg'):
Network error: sendmsg error: Invalid argument (error 22)
This patch updates the pattern of the expected error message for the
case if sendmsg() returns EINVAL when the sum of the iov_len values
overflows an ssize_t.
This is a follow-up for 2b0c1c019921e485f06c4be280fedba3d5279672.
Change-Id: I17681546fd942724025dfde81eaa412ee7e0acbf
Reviewed-on: http://gerrit.cloudera.org:8080/9719
Tested-by: Alexey Serbin <as...@cloudera.com>
Reviewed-by: Will Berkeley <wd...@gmail.com>
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/bc8a5029
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/bc8a5029
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/bc8a5029
Branch: refs/heads/master
Commit: bc8a5029675e30962ec5126afe3cf12f3143d4d3
Parents: 250375f
Author: Alexey Serbin <as...@cloudera.com>
Authored: Mon Mar 19 15:55:16 2018 -0700
Committer: Alexey Serbin <as...@cloudera.com>
Committed: Tue Mar 20 17:42:23 2018 +0000
----------------------------------------------------------------------
src/kudu/rpc/rpc-test.cc | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/kudu/blob/bc8a5029/src/kudu/rpc/rpc-test.cc
----------------------------------------------------------------------
diff --git a/src/kudu/rpc/rpc-test.cc b/src/kudu/rpc/rpc-test.cc
index 0960dd2..03c47e6 100644
--- a/src/kudu/rpc/rpc-test.cc
+++ b/src/kudu/rpc/rpc-test.cc
@@ -742,7 +742,9 @@ TEST_P(TestRpc, TestRpcSidecarLimits) {
// macOS, while reading from socket.
"|got EOF from remote"
// macOS, while writing to socket.
- "|Protocol wrong type for socket");
+ "|Protocol wrong type for socket"
+ // macOS, sendmsg(): the sum of the iov_len values overflows an ssize_t
+ "|sendmsg error: Invalid argument");
}
}
[2/5] kudu git commit: Upgrade gperftools to 2.6.90
Posted by al...@apache.org.
Upgrade gperftools to 2.6.90
2.6.90 is a release candidate for 2.7. It contains basically the same
bits that we are already using plus another small patch that I wrote
upstream that might make 1MB allocations slightly more efficient.
Change-Id: I62d832801160b25952669b7f4b49f5501bf0b112
Reviewed-on: http://gerrit.cloudera.org:8080/9721
Tested-by: Kudu Jenkins
Reviewed-by: Alexey Serbin <as...@cloudera.com>
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/250375f9
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/250375f9
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/250375f9
Branch: refs/heads/master
Commit: 250375f9783bd5558b4cdb5460dca030e9099168
Parents: 1bfb0c0
Author: Todd Lipcon <to...@apache.org>
Authored: Mon Mar 19 19:46:22 2018 -0700
Committer: Todd Lipcon <to...@apache.org>
Committed: Tue Mar 20 15:01:15 2018 +0000
----------------------------------------------------------------------
thirdparty/download-thirdparty.sh | 4 +-
...ed-O-log-n-searching-among-large-spans.patch | 494 -------------------
...namespace-base-with-namespace-tcmalloc.patch | 80 +--
.../gperftools-sized-alloc-build-fix.patch | 36 --
thirdparty/vars.sh | 2 +-
5 files changed, 20 insertions(+), 596 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/kudu/blob/250375f9/thirdparty/download-thirdparty.sh
----------------------------------------------------------------------
diff --git a/thirdparty/download-thirdparty.sh b/thirdparty/download-thirdparty.sh
index 5ef1618..d3c8704 100755
--- a/thirdparty/download-thirdparty.sh
+++ b/thirdparty/download-thirdparty.sh
@@ -124,15 +124,13 @@ if [ ! -d $GFLAGS_SOURCE ]; then
fetch_and_expand gflags-${GFLAGS_VERSION}.tar.gz
fi
-GPERFTOOLS_PATCHLEVEL=4
+GPERFTOOLS_PATCHLEVEL=1
delete_if_wrong_patchlevel $GPERFTOOLS_SOURCE $GPERFTOOLS_PATCHLEVEL
if [ ! -d $GPERFTOOLS_SOURCE ]; then
fetch_and_expand gperftools-${GPERFTOOLS_VERSION}.tar.gz
pushd $GPERFTOOLS_SOURCE
- patch -p1 < $TP_DIR/patches/gperftools-Implemented-O-log-n-searching-among-large-spans.patch
patch -p1 < $TP_DIR/patches/gperftools-Replace-namespace-base-with-namespace-tcmalloc.patch
- patch -p1 < $TP_DIR/patches/gperftools-sized-alloc-build-fix.patch
touch patchlevel-$GPERFTOOLS_PATCHLEVEL
autoreconf -fvi
popd
http://git-wip-us.apache.org/repos/asf/kudu/blob/250375f9/thirdparty/patches/gperftools-Implemented-O-log-n-searching-among-large-spans.patch
----------------------------------------------------------------------
diff --git a/thirdparty/patches/gperftools-Implemented-O-log-n-searching-among-large-spans.patch b/thirdparty/patches/gperftools-Implemented-O-log-n-searching-among-large-spans.patch
deleted file mode 100644
index 9a840ee..0000000
--- a/thirdparty/patches/gperftools-Implemented-O-log-n-searching-among-large-spans.patch
+++ /dev/null
@@ -1,494 +0,0 @@
-From 071cadad7a752c5678c18174a74b82858404f762 Mon Sep 17 00:00:00 2001
-From: Aliaksey Kandratsenka <al...@gmail.com>
-Date: Sat, 3 Jun 2017 15:31:06 -0700
-Subject: [PATCH 1/1] Implemented O(log n) searching among large spans
-
-This is implemented via std::set with custom STL allocator that
-delegates to PageHeapAllocator. Free large spans are not linked
-together via linked list, but inserted into std::set. Spans also store
-iterators to std::set positions pointing to them. So that removing
-span from set is fast too.
-
-Patch implemented by Aliaksey Kandratsenka and Todd Lipcon based on
-earlier research and experimentation by James Golick.
-
-Addresses issue #535
----
- src/page_heap.cc | 152 +++++++++++++++++++++++++---------------
- src/page_heap.h | 18 ++++--
- src/page_heap_allocator.h | 64 +++++++++++++++++
- src/span.h | 61 ++++++++++++++++-
- src/tests/tcmalloc_unittest.cc | 2 +-
- 5 files changed, 234 insertions(+), 63 deletions(-)
-
-diff --git a/src/page_heap.cc b/src/page_heap.cc
-index 0dda58f..712732a 100644
---- a/src/page_heap.cc
-+++ b/src/page_heap.cc
-@@ -69,8 +69,6 @@ PageHeap::PageHeap()
- release_index_(kMaxPages),
- aggressive_decommit_(false) {
- COMPILE_ASSERT(kClassSizesMax <= (1 << PageMapCache::kValuebits), valuebits);
-- DLL_Init(&large_.normal);
-- DLL_Init(&large_.returned);
- for (int i = 0; i < kMaxPages; i++) {
- DLL_Init(&free_[i].normal);
- DLL_Init(&free_[i].returned);
-@@ -168,45 +166,38 @@ Span* PageHeap::New(Length n) {
- }
-
- Span* PageHeap::AllocLarge(Length n) {
-- // find the best span (closest to n in size).
-- // The following loops implements address-ordered best-fit.
- Span *best = NULL;
-+ Span *best_normal = NULL;
-
-- // Search through normal list
-- for (Span* span = large_.normal.next;
-- span != &large_.normal;
-- span = span->next) {
-- if (span->length >= n) {
-- if ((best == NULL)
-- || (span->length < best->length)
-- || ((span->length == best->length) && (span->start < best->start))) {
-- best = span;
-- ASSERT(best->location == Span::ON_NORMAL_FREELIST);
-- }
-- }
-- }
-+ // Create a Span to use as an upper bound.
-+ Span bound;
-+ bound.start = 0;
-+ bound.length = n;
-
-- Span *bestNormal = best;
-+ // First search the NORMAL spans..
-+ SpanSet::iterator place = large_normal_.upper_bound(SpanPtrWithLength(&bound));
-+ if (place != large_normal_.end()) {
-+ best = place->span;
-+ best_normal = best;
-+ ASSERT(best->location == Span::ON_NORMAL_FREELIST);
-+ }
-
-- // Search through released list in case it has a better fit
-- for (Span* span = large_.returned.next;
-- span != &large_.returned;
-- span = span->next) {
-- if (span->length >= n) {
-- if ((best == NULL)
-- || (span->length < best->length)
-- || ((span->length == best->length) && (span->start < best->start))) {
-- best = span;
-- ASSERT(best->location == Span::ON_RETURNED_FREELIST);
-- }
-- }
-+ // Try to find better fit from RETURNED spans.
-+ place = large_returned_.upper_bound(SpanPtrWithLength(&bound));
-+ if (place != large_returned_.end()) {
-+ Span *c = place->span;
-+ ASSERT(c->location == Span::ON_RETURNED_FREELIST);
-+ if (best_normal == NULL
-+ || c->length < best->length
-+ || (c->length == best->length && c->start < best->start))
-+ best = place->span;
- }
-
-- if (best == bestNormal) {
-+ if (best == best_normal) {
- return best == NULL ? NULL : Carve(best, n);
- }
-
-- // best comes from returned list.
-+ // best comes from RETURNED set.
-
- if (EnsureLimit(n, false)) {
- return Carve(best, n);
-@@ -214,13 +205,13 @@ Span* PageHeap::AllocLarge(Length n) {
-
- if (EnsureLimit(n, true)) {
- // best could have been destroyed by coalescing.
-- // bestNormal is not a best-fit, and it could be destroyed as well.
-+ // best_normal is not a best-fit, and it could be destroyed as well.
- // We retry, the limit is already ensured:
- return AllocLarge(n);
- }
-
-- // If bestNormal existed, EnsureLimit would succeeded:
-- ASSERT(bestNormal == NULL);
-+ // If best_normal existed, EnsureLimit would succeeded:
-+ ASSERT(best_normal == NULL);
- // We are not allowed to take best from returned list.
- return NULL;
- }
-@@ -406,12 +397,27 @@ void PageHeap::MergeIntoFreeList(Span* span) {
-
- void PageHeap::PrependToFreeList(Span* span) {
- ASSERT(span->location != Span::IN_USE);
-- SpanList* list = (span->length < kMaxPages) ? &free_[span->length] : &large_;
-- if (span->location == Span::ON_NORMAL_FREELIST) {
-+ if (span->location == Span::ON_NORMAL_FREELIST)
- stats_.free_bytes += (span->length << kPageShift);
-+ else
-+ stats_.unmapped_bytes += (span->length << kPageShift);
-+
-+ if (span->length >= kMaxPages) {
-+ SpanSet *set = &large_normal_;
-+ if (span->location == Span::ON_RETURNED_FREELIST)
-+ set = &large_returned_;
-+ std::pair<SpanSet::iterator, bool> p =
-+ set->insert(SpanPtrWithLength(span));
-+ ASSERT(p.second); // We never have duplicates since span->start is unique.
-+ span->rev_ptr.set_iterator(p.first);
-+ ASSERT(span->rev_ptr.get_iterator()->span == span);
-+ return;
-+ }
-+
-+ SpanList* list = &free_[span->length];
-+ if (span->location == Span::ON_NORMAL_FREELIST) {
- DLL_Prepend(&list->normal, span);
- } else {
-- stats_.unmapped_bytes += (span->length << kPageShift);
- DLL_Prepend(&list->returned, span);
- }
- }
-@@ -423,7 +429,16 @@ void PageHeap::RemoveFromFreeList(Span* span) {
- } else {
- stats_.unmapped_bytes -= (span->length << kPageShift);
- }
-- DLL_Remove(span);
-+ if (span->length >= kMaxPages) {
-+ SpanSet *set = &large_normal_;
-+ if (span->location == Span::ON_RETURNED_FREELIST)
-+ set = &large_returned_;
-+ ASSERT(span->rev_ptr.get_iterator()->span == span);
-+ ASSERT(set->find(SpanPtrWithLength(span)) == span->rev_ptr.get_iterator());
-+ set->erase(span->rev_ptr.get_iterator());
-+ } else {
-+ DLL_Remove(span);
-+ }
- }
-
- void PageHeap::IncrementalScavenge(Length n) {
-@@ -459,8 +474,7 @@ void PageHeap::IncrementalScavenge(Length n) {
- }
- }
-
--Length PageHeap::ReleaseLastNormalSpan(SpanList* slist) {
-- Span* s = slist->normal.prev;
-+Length PageHeap::ReleaseSpan(Span* s) {
- ASSERT(s->location == Span::ON_NORMAL_FREELIST);
-
- if (DecommitSpan(s)) {
-@@ -477,21 +491,35 @@ Length PageHeap::ReleaseLastNormalSpan(SpanList* slist) {
- Length PageHeap::ReleaseAtLeastNPages(Length num_pages) {
- Length released_pages = 0;
-
-- // Round robin through the lists of free spans, releasing the last
-- // span in each list. Stop after releasing at least num_pages
-+ // Round robin through the lists of free spans, releasing a
-+ // span from each list. Stop after releasing at least num_pages
- // or when there is nothing more to release.
- while (released_pages < num_pages && stats_.free_bytes > 0) {
- for (int i = 0; i < kMaxPages+1 && released_pages < num_pages;
- i++, release_index_++) {
-+ Span *s;
- if (release_index_ > kMaxPages) release_index_ = 0;
-- SpanList* slist = (release_index_ == kMaxPages) ?
-- &large_ : &free_[release_index_];
-- if (!DLL_IsEmpty(&slist->normal)) {
-- Length released_len = ReleaseLastNormalSpan(slist);
-- // Some systems do not support release
-- if (released_len == 0) return released_pages;
-- released_pages += released_len;
-+
-+ if (release_index_ == kMaxPages) {
-+ if (large_normal_.empty()) {
-+ continue;
-+ }
-+ s = (large_normal_.begin())->span;
-+ } else {
-+ SpanList* slist = &free_[release_index_];
-+ if (DLL_IsEmpty(&slist->normal)) {
-+ continue;
-+ }
-+ s = slist->normal.prev;
- }
-+ // TODO(todd) if the remaining number of pages to release
-+ // is significantly smaller than s->length, and s is on the
-+ // large freelist, should we carve s instead of releasing?
-+ // the whole thing?
-+ Length released_len = ReleaseSpan(s);
-+ // Some systems do not support release
-+ if (released_len == 0) return released_pages;
-+ released_pages += released_len;
- }
- }
- return released_pages;
-@@ -544,12 +572,12 @@ void PageHeap::GetLargeSpanStats(LargeSpanStats* result) {
- result->spans = 0;
- result->normal_pages = 0;
- result->returned_pages = 0;
-- for (Span* s = large_.normal.next; s != &large_.normal; s = s->next) {
-- result->normal_pages += s->length;;
-+ for (SpanSet::iterator it = large_normal_.begin(); it != large_normal_.end(); ++it) {
-+ result->normal_pages += it->length;
- result->spans++;
- }
-- for (Span* s = large_.returned.next; s != &large_.returned; s = s->next) {
-- result->returned_pages += s->length;
-+ for (SpanSet::iterator it = large_returned_.begin(); it != large_returned_.end(); ++it) {
-+ result->returned_pages += it->length;
- result->spans++;
- }
- }
-@@ -664,8 +692,8 @@ bool PageHeap::Check() {
-
- bool PageHeap::CheckExpensive() {
- bool result = Check();
-- CheckList(&large_.normal, kMaxPages, 1000000000, Span::ON_NORMAL_FREELIST);
-- CheckList(&large_.returned, kMaxPages, 1000000000, Span::ON_RETURNED_FREELIST);
-+ CheckSet(&large_normal_, kMaxPages, Span::ON_NORMAL_FREELIST);
-+ CheckSet(&large_returned_, kMaxPages, Span::ON_RETURNED_FREELIST);
- for (Length s = 1; s < kMaxPages; s++) {
- CheckList(&free_[s].normal, s, s, Span::ON_NORMAL_FREELIST);
- CheckList(&free_[s].returned, s, s, Span::ON_RETURNED_FREELIST);
-@@ -685,4 +713,16 @@ bool PageHeap::CheckList(Span* list, Length min_pages, Length max_pages,
- return true;
- }
-
-+bool PageHeap::CheckSet(SpanSet* spanset, Length min_pages,int freelist) {
-+ for (SpanSet::iterator it = spanset->begin(); it != spanset->end(); ++it) {
-+ Span* s = it->span;
-+ CHECK_CONDITION(s->length == it->length);
-+ CHECK_CONDITION(s->location == freelist); // NORMAL or RETURNED
-+ CHECK_CONDITION(s->length >= min_pages);
-+ CHECK_CONDITION(GetDescriptor(s->start) == s);
-+ CHECK_CONDITION(GetDescriptor(s->start+s->length-1) == s);
-+ }
-+ return true;
-+}
-+
- } // namespace tcmalloc
-diff --git a/src/page_heap.h b/src/page_heap.h
-index b688512..34f18bb 100644
---- a/src/page_heap.h
-+++ b/src/page_heap.h
-@@ -196,6 +196,7 @@ class PERFTOOLS_DLL_DECL PageHeap {
- bool CheckExpensive();
- bool CheckList(Span* list, Length min_pages, Length max_pages,
- int freelist); // ON_NORMAL_FREELIST or ON_RETURNED_FREELIST
-+ bool CheckSet(SpanSet *s, Length min_pages, int freelist);
-
- // Try to release at least num_pages for reuse by the OS. Returns
- // the actual number of pages released, which may be less than
-@@ -264,8 +265,12 @@ class PERFTOOLS_DLL_DECL PageHeap {
- Span returned;
- };
-
-- // List of free spans of length >= kMaxPages
-- SpanList large_;
-+ // Sets of spans with length >= kMaxPages.
-+ //
-+ // Rather than using a linked list, we use sets here for efficient
-+ // best-fit search.
-+ SpanSet large_normal_;
-+ SpanSet large_returned_;
-
- // Array mapping from span length to a doubly linked list of free spans
- SpanList free_[kMaxPages];
-@@ -317,9 +322,12 @@ class PERFTOOLS_DLL_DECL PageHeap {
- // IncrementalScavenge(n) is called whenever n pages are freed.
- void IncrementalScavenge(Length n);
-
-- // Release the last span on the normal portion of this list.
-- // Return the length of that span or zero if release failed.
-- Length ReleaseLastNormalSpan(SpanList* slist);
-+ // Attempts to decommit 's' and move it to the returned freelist.
-+ //
-+ // Returns the length of the Span or zero if release failed.
-+ //
-+ // REQUIRES: 's' must be on the NORMAL freelist.
-+ Length ReleaseSpan(Span *s);
-
- // Checks if we are allowed to take more memory from the system.
- // If limit is reached and allowRelease is true, tries to release
-diff --git a/src/page_heap_allocator.h b/src/page_heap_allocator.h
-index 892d1c1..440b8ad 100644
---- a/src/page_heap_allocator.h
-+++ b/src/page_heap_allocator.h
-@@ -109,6 +109,70 @@ class PageHeapAllocator {
- int inuse_;
- };
-
-+// STL-compatible allocator which forwards allocations to a PageHeapAllocator.
-+//
-+// Like PageHeapAllocator, this requires external synchronization. To avoid multiple
-+// separate STLPageHeapAllocator<T> from sharing the same underlying PageHeapAllocator<T>,
-+// the |LockingTag| template argument should be used. Template instantiations with
-+// different locking tags can safely be used concurrently.
-+template <typename T, class LockingTag>
-+class STLPageHeapAllocator {
-+ public:
-+ typedef size_t size_type;
-+ typedef ptrdiff_t difference_type;
-+ typedef T* pointer;
-+ typedef const T* const_pointer;
-+ typedef T& reference;
-+ typedef const T& const_reference;
-+ typedef T value_type;
-+
-+ template <class T1> struct rebind {
-+ typedef STLPageHeapAllocator<T1, LockingTag> other;
-+ };
-+
-+ STLPageHeapAllocator() { }
-+ STLPageHeapAllocator(const STLPageHeapAllocator&) { }
-+ template <class T1> STLPageHeapAllocator(const STLPageHeapAllocator<T1, LockingTag>&) { }
-+ ~STLPageHeapAllocator() { }
-+
-+ pointer address(reference x) const { return &x; }
-+ const_pointer address(const_reference x) const { return &x; }
-+
-+ size_type max_size() const { return size_t(-1) / sizeof(T); }
-+
-+ void construct(pointer p, const T& val) { ::new(p) T(val); }
-+ void construct(pointer p) { ::new(p) T(); }
-+ void destroy(pointer p) { p->~T(); }
-+
-+ // There's no state, so these allocators are always equal
-+ bool operator==(const STLPageHeapAllocator&) const { return true; }
-+
-+ pointer allocate(size_type n, const void* = 0) {
-+ if (!underlying_.initialized) {
-+ underlying_.allocator.Init();
-+ underlying_.initialized = true;
-+ }
-+
-+ CHECK_CONDITION(n == 1);
-+ return underlying_.allocator.New();
-+ }
-+ void deallocate(pointer p, size_type n) {
-+ CHECK_CONDITION(n == 1);
-+ underlying_.allocator.Delete(p);
-+ }
-+
-+ private:
-+ struct Storage {
-+ explicit Storage(base::LinkerInitialized x) {}
-+ PageHeapAllocator<T> allocator;
-+ bool initialized;
-+ };
-+ static Storage underlying_;
-+};
-+
-+template<typename T, class LockingTag>
-+typename STLPageHeapAllocator<T, LockingTag>::Storage STLPageHeapAllocator<T, LockingTag>::underlying_(base::LINKER_INITIALIZED);
-+
- } // namespace tcmalloc
-
- #endif // TCMALLOC_PAGE_HEAP_ALLOCATOR_H_
-diff --git a/src/span.h b/src/span.h
-index 83feda1..4e343ba 100644
---- a/src/span.h
-+++ b/src/span.h
-@@ -37,17 +37,63 @@
- #define TCMALLOC_SPAN_H_
-
- #include <config.h>
-+#include <set>
- #include "common.h"
-+#include "base/logging.h"
-+#include "page_heap_allocator.h"
-
- namespace tcmalloc {
-
-+struct SpanBestFitLess;
-+struct Span;
-+
-+// Store a pointer to a span along with a cached copy of its length.
-+// These are used as set elements to improve the performance of
-+// comparisons during tree traversal: the lengths are inline with the
-+// tree nodes and thus avoid expensive cache misses to dereference
-+// the actual Span objects in most cases.
-+struct SpanPtrWithLength {
-+ explicit SpanPtrWithLength(Span* s);
-+
-+ Span* span;
-+ Length length;
-+};
-+typedef std::set<SpanPtrWithLength, SpanBestFitLess, STLPageHeapAllocator<SpanPtrWithLength, void> > SpanSet;
-+
-+// Comparator for best-fit search, with address order as a tie-breaker.
-+struct SpanBestFitLess {
-+ bool operator()(SpanPtrWithLength a, SpanPtrWithLength b);
-+};
-+
-+// Wrapper which stores a SpanSet::iterator as a POD type.
-+// This allows the iterator to be stored in a union below.
-+struct SpanSetRevPtr {
-+ char data[sizeof(SpanSet::iterator)];
-+
-+ SpanSet::iterator get_iterator() {
-+ SpanSet::iterator ret;
-+ memcpy(&ret, this, sizeof(ret));
-+ return ret;
-+ }
-+
-+ void set_iterator(const SpanSet::iterator& val) {
-+ new (this) SpanSet::iterator(val);
-+ }
-+};
-+
- // Information kept for a span (a contiguous run of pages).
- struct Span {
-+ Span() {}
- PageID start; // Starting page number
- Length length; // Number of pages in span
- Span* next; // Used when in link list
- Span* prev; // Used when in link list
-- void* objects; // Linked list of free objects
-+ union {
-+ void* objects; // Linked list of free objects
-+ SpanSetRevPtr rev_ptr; // "pointer" (std::set iterator) to
-+ // SpanSet entry associated with this
-+ // Span.
-+ };
- unsigned int refcount : 16; // Number of non-free objects
- unsigned int sizeclass : 8; // Size-class for small objects (or 0)
- unsigned int location : 2; // Is the span on a freelist, and if so, which?
-@@ -71,6 +117,19 @@ void Event(Span* span, char op, int v = 0);
- #define Event(s,o,v) ((void) 0)
- #endif
-
-+inline SpanPtrWithLength::SpanPtrWithLength(Span* s)
-+ : span(s),
-+ length(s->length) {
-+}
-+
-+inline bool SpanBestFitLess::operator()(SpanPtrWithLength a, SpanPtrWithLength b) {
-+ if (a.length < b.length)
-+ return true;
-+ if (a.length > b.length)
-+ return false;
-+ return a.span->start < b.span->start;
-+}
-+
- // Allocator/deallocator for spans
- Span* NewSpan(PageID p, Length len);
- void DeleteSpan(Span* span);
-diff --git a/src/tests/tcmalloc_unittest.cc b/src/tests/tcmalloc_unittest.cc
-index 25b2e41..a9c6429 100644
---- a/src/tests/tcmalloc_unittest.cc
-+++ b/src/tests/tcmalloc_unittest.cc
-@@ -1422,7 +1422,7 @@ static int RunAllTests(int argc, char** argv) {
-
- // Try strdup(), which the system allocates but we must free. If
- // all goes well, libc will use our malloc!
-- p2 = strdup("test");
-+ p2 = strdup("in memory of James Golick");
- CHECK(p2 != NULL);
- VerifyNewHookWasCalled();
- free(p2);
---
-1.7.1
-
http://git-wip-us.apache.org/repos/asf/kudu/blob/250375f9/thirdparty/patches/gperftools-Replace-namespace-base-with-namespace-tcmalloc.patch
----------------------------------------------------------------------
diff --git a/thirdparty/patches/gperftools-Replace-namespace-base-with-namespace-tcmalloc.patch b/thirdparty/patches/gperftools-Replace-namespace-base-with-namespace-tcmalloc.patch
index eca38d7..44e974d 100644
--- a/thirdparty/patches/gperftools-Replace-namespace-base-with-namespace-tcmalloc.patch
+++ b/thirdparty/patches/gperftools-Replace-namespace-base-with-namespace-tcmalloc.patch
@@ -1,52 +1,11 @@
-From 181843b0ecddcc8c8be912cf3c035e27e2f63842 Mon Sep 17 00:00:00 2001
-From: Todd Lipcon <to...@cloudera.com>
-Date: Mon, 5 Feb 2018 21:19:54 -0800
-Subject: [PATCH 3/3] Substitute namespaces
+commit afca5dcb9a9e88487523d7e0d7ff917555dc1a51
+Author: Todd Lipcon <to...@cloudera.com>
+Date: Mon Mar 19 19:44:05 2018 -0700
-perl -p -i -e 's,base::,tcmalloc::,g' $(find . -name *.h -o -name \*.cc)
-perl -p -i -e 's,namespace base,namespace tcmalloc,g' $(find . -name *.h -o -name \*.cc)
----
- src/base/atomicops-internals-arm-generic.h | 6 +-
- src/base/atomicops-internals-arm-v6plus.h | 4 +-
- src/base/atomicops-internals-gcc.h | 6 +-
- src/base/atomicops-internals-linuxppc.h | 6 +-
- src/base/atomicops-internals-macosx.h | 8 +-
- src/base/atomicops-internals-mips.h | 6 +-
- src/base/atomicops-internals-windows.h | 14 ++--
- src/base/atomicops-internals-x86.h | 6 +-
- src/base/atomicops.h | 92 ++++++++++++++--------------
- src/base/basictypes.h | 8 +-
- src/base/elf_mem_image.cc | 4 +-
- src/base/elf_mem_image.h | 4 +-
- src/base/spinlock.cc | 16 +++---
- src/base/spinlock.h | 14 ++--
- src/base/spinlock_internal.cc | 18 +++---
- src/base/spinlock_internal.h | 4 +-
- src/base/spinlock_linux-inl.h | 6 +-
- src/base/spinlock_posix-inl.h | 6 +-
- src/base/spinlock_win32-inl.h | 6 +-
- src/base/vdso_support.cc | 4 +-
- src/base/vdso_support.h | 4 +-
- src/central_freelist.h | 2 +-
- src/emergency_malloc.cc | 2 +-
- src/gperftools/malloc_extension.h | 8 +-
- src/heap-profile-table.cc | 2 +-
- src/internal_logging.cc | 2 +-
- src/malloc_hook-inl.h | 44 +++++++-------
- src/malloc_hook.cc | 60 +++++++++---------
- src/page_heap.cc | 10 ++--
- src/page_heap.h | 4 +-
- src/page_heap_allocator.h | 4 +-
- src/raw_printer.cc | 2 +-
- src/raw_printer.h | 2 +-
- src/stacktrace_powerpc-linux-inl.h | 4 +-
- src/stacktrace_x86-inl.h | 6 +-
- src/tcmalloc.cc | 10 ++--
- src/tests/atomicops_unittest.cc | 36 ++++++------
- src/tests/malloc_hook_test.cc | 4 +-
- src/tests/raw_printer_test.cc | 8 +-
- src/tests/tcmalloc_unittest.cc | 28 ++++----
- 40 files changed, 240 insertions(+), 240 deletions(-)
+ Substitute namespaces
+
+ perl -p -i -e 's,base::,tcmalloc::,g' $(find . -name *.h -o -name \*.cc)
+ perl -p -i -e 's,namespace base,namespace tcmalloc,g' $(find . -name *.h -o -name \*.cc)
diff --git a/src/base/atomicops-internals-arm-generic.h b/src/base/atomicops-internals-arm-generic.h
index d0f9413..c81f1e6 100644
@@ -1299,7 +1258,7 @@ index 64c2165..1b0dcd0 100644
// These are available as C bindings as well as C++, hence their
// definition outside the MallocHook class.
diff --git a/src/page_heap.cc b/src/page_heap.cc
-index 712732a..c1c7ed3 100644
+index 7dd5646..673cfa6 100644
--- a/src/page_heap.cc
+++ b/src/page_heap.cc
@@ -582,7 +582,7 @@ void PageHeap::GetLargeSpanStats(LargeSpanStats* result) {
@@ -1338,7 +1297,7 @@ index 712732a..c1c7ed3 100644
}
return true;
diff --git a/src/page_heap.h b/src/page_heap.h
-index 34f18bb..c040b2d 100644
+index bf50394..ec487bc 100644
--- a/src/page_heap.h
+++ b/src/page_heap.h
@@ -65,7 +65,7 @@
@@ -1360,10 +1319,10 @@ index 34f18bb..c040b2d 100644
// Page heap statistics
struct Stats {
diff --git a/src/page_heap_allocator.h b/src/page_heap_allocator.h
-index 440b8ad..bc60a21 100644
+index 3fecabd..9a2f791 100644
--- a/src/page_heap_allocator.h
+++ b/src/page_heap_allocator.h
-@@ -163,7 +163,7 @@ class STLPageHeapAllocator {
+@@ -164,7 +164,7 @@ class STLPageHeapAllocator {
private:
struct Storage {
@@ -1372,7 +1331,7 @@ index 440b8ad..bc60a21 100644
PageHeapAllocator<T> allocator;
bool initialized;
};
-@@ -171,7 +171,7 @@ class STLPageHeapAllocator {
+@@ -172,7 +172,7 @@ class STLPageHeapAllocator {
};
template<typename T, class LockingTag>
@@ -1441,10 +1400,10 @@ index 46eb5d8..ae3287b 100644
STT_FUNC, &rt_sigreturn_symbol_info) ||
!vdso.LookupSymbol("__kernel_vsyscall", "LINUX_2.5",
diff --git a/src/tcmalloc.cc b/src/tcmalloc.cc
-index be0a6ee..e506bd3 100644
+index 1f22dfb..a8fc5ee 100644
--- a/src/tcmalloc.cc
+++ b/src/tcmalloc.cc
-@@ -589,7 +589,7 @@ static void IterateOverRanges(void* arg, MallocExtension::RangeFunction func) {
+@@ -590,7 +590,7 @@ static void IterateOverRanges(void* arg, MallocExtension::RangeFunction func) {
while (!done) {
// Accumulate a small number of ranges in a local buffer
static const int kNumRanges = 16;
@@ -1453,7 +1412,7 @@ index be0a6ee..e506bd3 100644
int n = 0;
{
SpinLockHolder h(Static::pageheap_lock());
-@@ -1823,7 +1823,7 @@ void* memalign_pages(size_t align, size_t size,
+@@ -1824,7 +1824,7 @@ void* memalign_pages(size_t align, size_t size,
template <void* OOMHandler(size_t)>
ATTRIBUTE_ALWAYS_INLINE inline
static void * malloc_fast_path(size_t size) {
@@ -1462,7 +1421,7 @@ index be0a6ee..e506bd3 100644
return tcmalloc::dispatch_allocate_full<OOMHandler>(size);
}
-@@ -1874,7 +1874,7 @@ void* tc_malloc(size_t size) PERFTOOLS_NOTHROW {
+@@ -1875,7 +1875,7 @@ void* tc_malloc(size_t size) PERFTOOLS_NOTHROW {
static ATTRIBUTE_ALWAYS_INLINE inline
void free_fast_path(void *ptr) {
@@ -1471,7 +1430,7 @@ index be0a6ee..e506bd3 100644
tcmalloc::invoke_hooks_and_free(ptr);
return;
}
-@@ -1888,7 +1888,7 @@ void tc_free(void* ptr) PERFTOOLS_NOTHROW {
+@@ -1889,7 +1889,7 @@ void tc_free(void* ptr) PERFTOOLS_NOTHROW {
extern "C" PERFTOOLS_DLL_DECL CACHELINE_ALIGNED_FN
void tc_free_sized(void *ptr, size_t size) PERFTOOLS_NOTHROW {
@@ -1480,7 +1439,7 @@ index be0a6ee..e506bd3 100644
tcmalloc::invoke_hooks_and_free(ptr);
return;
}
-@@ -1990,7 +1990,7 @@ extern "C" PERFTOOLS_DLL_DECL void tc_delete_nothrow(void* p, const std::nothrow
+@@ -1991,7 +1991,7 @@ extern "C" PERFTOOLS_DLL_DECL void tc_delete_nothrow(void* p, const std::nothrow
TC_ALIAS(tc_free);
#else
{
@@ -1690,6 +1649,3 @@ index a9c6429..9149fde 100644
}
#ifndef DEBUGALLOCATION
---
-1.7.1
-
http://git-wip-us.apache.org/repos/asf/kudu/blob/250375f9/thirdparty/patches/gperftools-sized-alloc-build-fix.patch
----------------------------------------------------------------------
diff --git a/thirdparty/patches/gperftools-sized-alloc-build-fix.patch b/thirdparty/patches/gperftools-sized-alloc-build-fix.patch
deleted file mode 100644
index 2566070..0000000
--- a/thirdparty/patches/gperftools-sized-alloc-build-fix.patch
+++ /dev/null
@@ -1,36 +0,0 @@
---- ./configure.ac.orig 2017-12-09 09:43:20.000000000 -0800
-+++ ./configure.ac 2018-01-29 17:31:48.000000000 -0800
-@@ -370,8 +370,8 @@
- [perftools_cv_sized_deallocation_result],
- [AC_LANG_PUSH(C++)
- OLD_CXXFLAGS="$CXXFLAGS"
-- CXXFLAGS="$CXXFLAGS -fsized-deallocation"
-- AC_COMPILE_IFELSE([AC_LANG_PROGRAM(
-+ CXXFLAGS="$CXXFLAGS -fsized-deallocation -O0 -fno-builtin"
-+ AC_LINK_IFELSE([AC_LANG_PROGRAM(
- [[#include <new>]],
- [[(::operator delete)(0, 256)]])],
- perftools_cv_sized_deallocation_result=yes,
-@@ -385,18 +385,21 @@
- AC_CACHE_CHECK([if C++ compiler supports std::align_val_t without options],
- [perftools_cv_have_align_val_t],
- [AC_LANG_PUSH(C++)
-+ OLD_CXXFLAGS="$CXXFLAGS"
-+ CXXFLAGS="$CXXFLAGS -O0 -fno-builtin"
- AC_LINK_IFELSE([AC_LANG_PROGRAM(
- [[#include <new>]],
- [[(::operator delete)((::operator new)(256, std::align_val_t(16)), std::align_val_t(16))]])],
- perftools_cv_have_align_val_t=yes,
- perftools_cv_have_align_val_t=no)
-+ CXXFLAGS="$OLD_CXXFLAGS"
- AC_LANG_POP(C++)])
-
- AC_CACHE_CHECK([if C++ compiler supports -faligned-new],
- [perftools_cv_have_f_aligned_new],
- [AC_LANG_PUSH(C++)
- OLD_CXXFLAGS="$CXXFLAGS"
-- CXXFLAGS="$CXXFLAGS -faligned-new"
-+ CXXFLAGS="$CXXFLAGS -faligned-new -O0 -fno-builtin"
- AC_LINK_IFELSE([AC_LANG_PROGRAM(
- [[#include <new>]],
- [[(::operator delete)((::operator new)(256, std::align_val_t(16)), std::align_val_t(16))]])],
http://git-wip-us.apache.org/repos/asf/kudu/blob/250375f9/thirdparty/vars.sh
----------------------------------------------------------------------
diff --git a/thirdparty/vars.sh b/thirdparty/vars.sh
index 6937d06..c5ee56e 100644
--- a/thirdparty/vars.sh
+++ b/thirdparty/vars.sh
@@ -46,7 +46,7 @@ GMOCK_VERSION=1.8.0
GMOCK_NAME=googletest-release-$GMOCK_VERSION
GMOCK_SOURCE=$TP_SOURCE_DIR/$GMOCK_NAME
-GPERFTOOLS_VERSION=2.6.3
+GPERFTOOLS_VERSION=2.6.90
GPERFTOOLS_NAME=gperftools-$GPERFTOOLS_VERSION
GPERFTOOLS_SOURCE=$TP_SOURCE_DIR/$GPERFTOOLS_NAME
[4/5] kudu git commit: [master-test] TestDumpStacksOnRpcQueueOverflow
is not relevant on macOS
Posted by al...@apache.org.
[master-test] TestDumpStacksOnRpcQueueOverflow is not relevant on macOS
As Will noticed, the TestDumpStacksOnRpcQueueOverflow scenario always
fails on macOS since the stack trace dumping is not supported there.
This patch simply moves the TestDumpStacksOnRpcQueueOverflow scenario
under '#ifndef __APPLE__ ... #endif' macro.
This is a follow-up for 96b3b63a48c270a77819e6fc80d1dfdb381a0df7.
Change-Id: Id6e1424a37628537f45b6cda7fd9b7e06101e53e
Reviewed-on: http://gerrit.cloudera.org:8080/9720
Tested-by: Kudu Jenkins
Reviewed-by: Will Berkeley <wd...@gmail.com>
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/e391e8a9
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/e391e8a9
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/e391e8a9
Branch: refs/heads/master
Commit: e391e8a9c2c92daab4124ab9d2c9803a8df6afd7
Parents: bc8a502
Author: Alexey Serbin <as...@cloudera.com>
Authored: Mon Mar 19 16:16:26 2018 -0700
Committer: Alexey Serbin <as...@cloudera.com>
Committed: Tue Mar 20 17:42:30 2018 +0000
----------------------------------------------------------------------
src/kudu/master/master-test.cc | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/kudu/blob/e391e8a9/src/kudu/master/master-test.cc
----------------------------------------------------------------------
diff --git a/src/kudu/master/master-test.cc b/src/kudu/master/master-test.cc
index bc5a0d1..8b48c32 100644
--- a/src/kudu/master/master-test.cc
+++ b/src/kudu/master/master-test.cc
@@ -798,8 +798,11 @@ TEST_F(MasterTest, TestInvalidGetTableLocations) {
}
}
+#ifndef __APPLE__
// Test that, if the master's RPC service queue overflows, thread stack traces
// are dumped to the diagnostics log.
+//
+// This test is not relevant on OS X where stack trace dumping is not supported.
TEST_F(MasterTest, TestDumpStacksOnRpcQueueOverflow) {
mini_master_->mutable_options()->rpc_opts.num_service_threads = 1;
mini_master_->mutable_options()->rpc_opts.service_queue_length = 1;
@@ -842,7 +845,7 @@ TEST_F(MasterTest, TestDumpStacksOnRpcQueueOverflow) {
});
ASSERT_EQ(dump_count, 1) << log_contents.ToString();
}
-
+#endif // #ifndef __APPLE__
// Tests that if the master is shutdown while a table visitor is active, the
// shutdown waits for the visitor to finish, avoiding racing and crashing.
[5/5] kudu git commit: [delete_table-itest] fix flake in
TestUnknownTabletsAreNotDeleted
Posted by al...@apache.org.
[delete_table-itest] fix flake in TestUnknownTabletsAreNotDeleted
Fixed flake in the DeleteTableITest.TestUnknownTabletsAreNotDeleted.
The flake was easily reproducible under macOS.
The scenario involves removing master's data directory along with the
IPKI information. Once the master re-generates its IPKI system records
and starts using the new TLS server certificate signed by the newly
generated CA private key, the tserver fails to verify the new master's
server certificate using the old CA certificate.
With the RPC authentication set to "optional" and 10ms tserver->master
heartbeat interval, the tserver in most cases was able to establish
a connection to the restarted master before it re-generates its IPKI
records, so no TLS certificate was used for authentication.
Change-Id: Ib6fd439c0ef5fb66b752f7f49175e4c2d818412e
Reviewed-on: http://gerrit.cloudera.org:8080/9722
Tested-by: Kudu Jenkins
Reviewed-by: Will Berkeley <wd...@gmail.com>
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/5d7b68d5
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/5d7b68d5
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/5d7b68d5
Branch: refs/heads/master
Commit: 5d7b68d59ae78560d0997168a27a73c4e5280ecc
Parents: e391e8a
Author: Alexey Serbin <as...@cloudera.com>
Authored: Mon Mar 19 20:04:38 2018 -0700
Committer: Alexey Serbin <as...@cloudera.com>
Committed: Tue Mar 20 17:43:34 2018 +0000
----------------------------------------------------------------------
.../integration-tests/delete_table-itest.cc | 48 ++++++++++++--------
1 file changed, 30 insertions(+), 18 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/kudu/blob/5d7b68d5/src/kudu/integration-tests/delete_table-itest.cc
----------------------------------------------------------------------
diff --git a/src/kudu/integration-tests/delete_table-itest.cc b/src/kudu/integration-tests/delete_table-itest.cc
index 43ed3b2..e611a01 100644
--- a/src/kudu/integration-tests/delete_table-itest.cc
+++ b/src/kudu/integration-tests/delete_table-itest.cc
@@ -1093,10 +1093,32 @@ TEST_F(DeleteTableITest, TestWebPageForTombstonedTablet) {
}
TEST_F(DeleteTableITest, TestUnknownTabletsAreNotDeleted) {
- // Speed up heartbeating so that the unknown tablet is detected faster.
- vector<string> extra_ts_flags = { "--heartbeat_interval_ms=10" };
+ //
+ // NOTE on disabled RPC authentication and encryption:
+ // This test scenario would be flaky if the master/tserver authentication
+ // were done via TLS certificates. That's because the scenario involves
+ // removing master's data directory along with the IPKI information. Once
+ // the master re-generates its IPKI system records and starts using the new
+ // TLS server certificate signed by the newly generated CA private key,
+ // the tserver fails to verify the new master's certificate using the old CA
+ // certificate.
+ //
+ constexpr int kNumTabletServers = 1;
+ const vector<string> extra_ts_flags = {
+ // Speed up heartbeating so that the unknown tablet is detected faster.
+ "--heartbeat_interval_ms=10",
+
+ // See the note above on disabled RPC authentication and encryption.
+ "--rpc_authentication=disabled",
+ "--rpc_encryption=disabled",
+ };
+ const vector<string> extra_master_flags = {
+ // See the note above on disabled RPC authentication and encryption.
+ "--rpc_authentication=disabled",
+ "--rpc_encryption=disabled",
+ };
- NO_FATALS(StartCluster(extra_ts_flags, {}, 1));
+ NO_FATALS(StartCluster(extra_ts_flags, extra_master_flags, kNumTabletServers));
Schema schema(GetSimpleTestSchema());
client::KuduSchema client_schema(client::KuduSchemaFromSchema(schema));
@@ -1113,22 +1135,12 @@ TEST_F(DeleteTableITest, TestUnknownTabletsAreNotDeleted) {
ASSERT_OK(cluster_->master()->DeleteFromDisk());
ASSERT_OK(cluster_->master()->Restart());
- // Give the master a chance to finish writing the new master tablet to disk
- // so that it can be found after the subsequent restart below.
- ASSERT_OK(cluster_->master()->WaitForCatalogManager());
-
- // The master should not delete the tablet. Let's wait for at least one
- // heartbeat to pass.
+ // Let's wait for tablet server registration with the master: it guarantees
+ // at least one heartbeat is processed by the master.
+ ASSERT_OK(cluster_->WaitForTabletServerCount(kNumTabletServers,
+ MonoDelta::FromSeconds(30)));
+ // The master should not delete the tablet.
int64_t num_delete_attempts;
- ASSERT_EVENTUALLY([&]() {
- int64_t num_heartbeats;
- ASSERT_OK(GetInt64Metric(
- cluster_->master()->bound_http_hostport(),
- &METRIC_ENTITY_server, "kudu.master",
- &METRIC_handler_latency_kudu_master_MasterService_TSHeartbeat, "total_count",
- &num_heartbeats));
- ASSERT_GE(num_heartbeats, 1);
- });
ASSERT_OK(GetInt64Metric(
cluster_->tablet_server(0)->bound_http_hostport(),
&METRIC_ENTITY_server, "kudu.tabletserver",