You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kudu.apache.org by to...@apache.org on 2016/12/12 04:55:11 UTC
[1/5] kudu git commit: Remove gcc/libstdcxx from
thirdparty/LICENSE.txt
Repository: kudu
Updated Branches:
refs/heads/master 2d74f7c06 -> 2eba7b884
Remove gcc/libstdcxx from thirdparty/LICENSE.txt
We no longer download and build libstdcxx, but forgot to remove this
from the LICENSE.txt file.
Change-Id: I6c040520f701d4b8b4f04176aa03640b38451fea
Reviewed-on: http://gerrit.cloudera.org:8080/5466
Tested-by: Todd Lipcon <to...@apache.org>
Reviewed-by: Adar Dembo <ad...@cloudera.com>
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/47905c7e
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/47905c7e
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/47905c7e
Branch: refs/heads/master
Commit: 47905c7e985d0895af3eb16d41247bc1ed81e18f
Parents: 2d74f7c
Author: Todd Lipcon <to...@apache.org>
Authored: Sun Dec 11 22:35:42 2016 +0800
Committer: Todd Lipcon <to...@apache.org>
Committed: Mon Dec 12 01:53:36 2016 +0000
----------------------------------------------------------------------
thirdparty/LICENSE.txt | 5 -----
1 file changed, 5 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/kudu/blob/47905c7e/thirdparty/LICENSE.txt
----------------------------------------------------------------------
diff --git a/thirdparty/LICENSE.txt b/thirdparty/LICENSE.txt
index ba5c2d5..558daee 100644
--- a/thirdparty/LICENSE.txt
+++ b/thirdparty/LICENSE.txt
@@ -637,11 +637,6 @@ source: https://github.com/gcovr/gcovr
NOTE: optional build-time dependency
--------------------------------------------------------------------------------
-thirdparty/gcc-*/: GPL version 2
-Source: https://gcc.gnu.org
-NOTE: build-time dependency, not linked or bundled.
-
---------------------------------------------------------------------------------
thirdparty/gmock-*/: BSD 3-clause
Source: https://github.com/google/googletest
NOTE: build-time dependency
[3/5] kudu git commit: Make tablet_copy-itest less flaky
Posted by to...@apache.org.
Make tablet_copy-itest less flaky
The test TestDisableTabletCopy_NoTightLoopWhenTabletDeleted sometimes
sleeps longer than 1 second when we call SleepFor() due to system load,
so we average our "per second" calculation over the time actually
elapsed.
Also bump slightly the number of log messages we tolerate due to
spurious log messages for unrelated components.
Change-Id: I5132d3ea99a8ecd05191a576edca83ffb210a90e
Reviewed-on: http://gerrit.cloudera.org:8080/5441
Tested-by: Kudu Jenkins
Reviewed-by: Todd Lipcon <to...@apache.org>
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/88745f32
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/88745f32
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/88745f32
Branch: refs/heads/master
Commit: 88745f32ca5ba69041170173ecede34c2ed85a39
Parents: 28a07bf
Author: Mike Percy <mp...@apache.org>
Authored: Wed Dec 7 15:42:36 2016 +0000
Committer: Todd Lipcon <to...@apache.org>
Committed: Mon Dec 12 04:39:21 2016 +0000
----------------------------------------------------------------------
src/kudu/integration-tests/tablet_copy-itest.cc | 11 ++++++++---
1 file changed, 8 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/kudu/blob/88745f32/src/kudu/integration-tests/tablet_copy-itest.cc
----------------------------------------------------------------------
diff --git a/src/kudu/integration-tests/tablet_copy-itest.cc b/src/kudu/integration-tests/tablet_copy-itest.cc
index 4d2482d..41980db 100644
--- a/src/kudu/integration-tests/tablet_copy-itest.cc
+++ b/src/kudu/integration-tests/tablet_copy-itest.cc
@@ -711,18 +711,23 @@ TEST_F(TabletCopyITest, TestDisableTabletCopy_NoTightLoopWhenTabletDeleted) {
// Ensure that, if we sleep for a second while still doing writes to the leader:
// a) we don't spew logs on the leader side
// b) we don't get hit with a lot of UpdateConsensus calls on the replica.
+ MonoTime start = MonoTime::Now();
int64_t num_update_rpcs_initial = CountUpdateConsensusCalls(replica_ets, tablet_id);
int64_t num_logs_initial = CountLogMessages(leader_ts);
SleepFor(MonoDelta::FromSeconds(1));
int64_t num_update_rpcs_after_sleep = CountUpdateConsensusCalls(replica_ets, tablet_id);
int64_t num_logs_after_sleep = CountLogMessages(leader_ts);
+ MonoTime end = MonoTime::Now();
+ MonoDelta elapsed = end - start;
// Calculate rate per second of RPCs and log messages
- int64_t update_rpcs_per_second = num_update_rpcs_after_sleep - num_update_rpcs_initial;
+ int64_t update_rpcs_per_second =
+ (num_update_rpcs_after_sleep - num_update_rpcs_initial) / elapsed.ToSeconds();
EXPECT_LT(update_rpcs_per_second, 20);
- int64_t num_logs_per_second = num_logs_after_sleep - num_logs_initial;
- EXPECT_LT(num_logs_per_second, 20);
+ int64_t num_logs_per_second =
+ (num_logs_after_sleep - num_logs_initial) / elapsed.ToSeconds();
+ EXPECT_LT(num_logs_per_second, 30); // We might occasionally get unrelated log messages.
}
// Test that if a Tablet Copy is taking a long time but the client peer is still responsive,
[2/5] kudu git commit: Make TabletCopyITest subclass
ExternalMiniClusterITestBase
Posted by to...@apache.org.
Make TabletCopyITest subclass ExternalMiniClusterITestBase
No functional changes, just deleting duplicated code.
Change-Id: I8d8928406ba15c5d69aab8850abed30b48826415
Reviewed-on: http://gerrit.cloudera.org:8080/5440
Tested-by: Kudu Jenkins
Reviewed-by: Todd Lipcon <to...@apache.org>
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/28a07bf7
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/28a07bf7
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/28a07bf7
Branch: refs/heads/master
Commit: 28a07bf7a3055ebdc526abb176458ba9145959dc
Parents: 47905c7
Author: Mike Percy <mp...@apache.org>
Authored: Fri Dec 9 16:13:10 2016 +0000
Committer: Todd Lipcon <to...@apache.org>
Committed: Mon Dec 12 04:39:12 2016 +0000
----------------------------------------------------------------------
src/kudu/integration-tests/tablet_copy-itest.cc | 60 +-------------------
1 file changed, 3 insertions(+), 57 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/kudu/blob/28a07bf7/src/kudu/integration-tests/tablet_copy-itest.cc
----------------------------------------------------------------------
diff --git a/src/kudu/integration-tests/tablet_copy-itest.cc b/src/kudu/integration-tests/tablet_copy-itest.cc
index 497fa63..4d2482d 100644
--- a/src/kudu/integration-tests/tablet_copy-itest.cc
+++ b/src/kudu/integration-tests/tablet_copy-itest.cc
@@ -18,10 +18,10 @@
#include <atomic>
#include <boost/optional.hpp>
#include <gflags/gflags.h>
-#include <gtest/gtest.h>
#include <string>
#include <thread>
#include <unordered_map>
+#include <vector>
#include "kudu/client/client-test-util.h"
#include "kudu/client/client.h"
@@ -31,16 +31,13 @@
#include "kudu/gutil/strings/substitute.h"
#include "kudu/integration-tests/cluster_itest_util.h"
#include "kudu/integration-tests/cluster_verifier.h"
-#include "kudu/integration-tests/external_mini_cluster.h"
-#include "kudu/integration-tests/external_mini_cluster_fs_inspector.h"
+#include "kudu/integration-tests/external_mini_cluster-itest-base.h"
#include "kudu/integration-tests/test_workload.h"
#include "kudu/tablet/tablet_bootstrap.h"
#include "kudu/tablet/tablet_metadata.h"
#include "kudu/tserver/tablet_copy_client.h"
#include "kudu/util/barrier.h"
#include "kudu/util/metrics.h"
-#include "kudu/util/pstack_watcher.h"
-#include "kudu/util/test_util.h"
DEFINE_int32(test_num_threads, 16,
"Number of test threads to launch");
@@ -53,12 +50,9 @@ DEFINE_int32(test_delete_leader_payload_bytes, 16 * 1024,
DEFINE_int32(test_delete_leader_num_writer_threads, 1,
"Number of writer threads in TestDeleteLeaderDuringTabletCopyStressTest.");
-using kudu::client::KuduClient;
-using kudu::client::KuduClientBuilder;
using kudu::client::KuduSchema;
using kudu::client::KuduSchemaFromSchema;
using kudu::client::KuduTableCreator;
-using kudu::client::sp::shared_ptr;
using kudu::consensus::CONSENSUS_CONFIG_COMMITTED;
using kudu::itest::TServerDetails;
using kudu::tablet::TABLET_DATA_DELETED;
@@ -68,7 +62,6 @@ using kudu::tserver::TabletCopyClient;
using std::string;
using std::unordered_map;
using std::vector;
-using strings::Substitute;
METRIC_DECLARE_entity(server);
METRIC_DECLARE_histogram(handler_latency_kudu_consensus_ConsensusService_UpdateConsensus);
@@ -78,56 +71,9 @@ METRIC_DECLARE_counter(glog_error_messages);
namespace kudu {
-class TabletCopyITest : public KuduTest {
- public:
- virtual void TearDown() OVERRIDE {
- if (HasFatalFailure()) {
- LOG(INFO) << "Found fatal failure";
- for (int i = 0; i < 3; i++) {
- if (!cluster_->tablet_server(i)->IsProcessAlive()) {
- LOG(INFO) << "Tablet server " << i << " is not running. Cannot dump its stacks.";
- continue;
- }
- LOG(INFO) << "Attempting to dump stacks of TS " << i
- << " with UUID " << cluster_->tablet_server(i)->uuid()
- << " and pid " << cluster_->tablet_server(i)->pid();
- WARN_NOT_OK(PstackWatcher::DumpPidStacks(cluster_->tablet_server(i)->pid()),
- "Couldn't dump stacks");
- }
- }
- if (cluster_) cluster_->Shutdown();
- KuduTest::TearDown();
- STLDeleteValues(&ts_map_);
- }
-
- protected:
- void StartCluster(const vector<string>& extra_tserver_flags = vector<string>(),
- const vector<string>& extra_master_flags = vector<string>(),
- int num_tablet_servers = 3);
-
- gscoped_ptr<ExternalMiniCluster> cluster_;
- gscoped_ptr<itest::ExternalMiniClusterFsInspector> inspect_;
- shared_ptr<KuduClient> client_;
- unordered_map<string, TServerDetails*> ts_map_;
+class TabletCopyITest : public ExternalMiniClusterITestBase {
};
-void TabletCopyITest::StartCluster(const vector<string>& extra_tserver_flags,
- const vector<string>& extra_master_flags,
- int num_tablet_servers) {
- ExternalMiniClusterOptions opts;
- opts.num_tablet_servers = num_tablet_servers;
- opts.extra_tserver_flags = extra_tserver_flags;
- opts.extra_tserver_flags.push_back("--never_fsync"); // fsync causes flakiness on EC2.
- opts.extra_master_flags = extra_master_flags;
- cluster_.reset(new ExternalMiniCluster(opts));
- ASSERT_OK(cluster_->Start());
- inspect_.reset(new itest::ExternalMiniClusterFsInspector(cluster_.get()));
- ASSERT_OK(itest::CreateTabletServerMap(cluster_->master_proxy().get(),
- cluster_->messenger(),
- &ts_map_));
- ASSERT_OK(cluster_->CreateClient(nullptr, &client_));
-}
-
// If a rogue (a.k.a. zombie) leader tries to replace a tombstoned
// tablet via Tablet Copy, make sure its term isn't older than the latest term
// we observed. If it is older, make sure we reject the request, to avoid allowing old
[4/5] kudu git commit: Attempt to make
tablet_copy_client_session-itest less flaky
Posted by to...@apache.org.
Attempt to make tablet_copy_client_session-itest less flaky
I'm having trouble reproducing the errors seen on EC2 but I'm hoping
this will make the test less flaky on the test dashboard. Let's see.
Change-Id: If076e7997e95335d25346492938764ccb283f813
Reviewed-on: http://gerrit.cloudera.org:8080/5468
Tested-by: Kudu Jenkins
Reviewed-by: Todd Lipcon <to...@apache.org>
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/770f8e13
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/770f8e13
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/770f8e13
Branch: refs/heads/master
Commit: 770f8e134a2d4d38619ee15505474a742607247f
Parents: 88745f3
Author: Mike Percy <mp...@apache.org>
Authored: Sun Dec 11 08:17:04 2016 -0800
Committer: Todd Lipcon <to...@apache.org>
Committed: Mon Dec 12 04:49:48 2016 +0000
----------------------------------------------------------------------
.../tablet_copy_client_session-itest.cc | 19 +++++++++++++------
1 file changed, 13 insertions(+), 6 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/kudu/blob/770f8e13/src/kudu/integration-tests/tablet_copy_client_session-itest.cc
----------------------------------------------------------------------
diff --git a/src/kudu/integration-tests/tablet_copy_client_session-itest.cc b/src/kudu/integration-tests/tablet_copy_client_session-itest.cc
index 4a2935f..304c9b5 100644
--- a/src/kudu/integration-tests/tablet_copy_client_session-itest.cc
+++ b/src/kudu/integration-tests/tablet_copy_client_session-itest.cc
@@ -15,6 +15,8 @@
// specific language governing permissions and limitations
// under the License.
+#include <stdlib.h>
+
#include <boost/optional.hpp>
#include "kudu/common/wire_protocol.h"
@@ -90,7 +92,11 @@ void TabletCopyClientSessionITest::PrepareClusterForTabletCopy(
// Regression test for KUDU-1785. Ensure that starting a tablet copy session
// while a tablet is bootstrapping will result in a simple failure, not a crash.
TEST_F(TabletCopyClientSessionITest, TestStartTabletCopyWhileSourceBootstrapping) {
- const MonoDelta kTimeout = MonoDelta::FromSeconds(60);
+ if (!AllowSlowTests()) {
+ LOG(INFO) << "Test only runs in slow test mode";
+ return;
+ }
+ const MonoDelta kTimeout = MonoDelta::FromSeconds(90); // Can be very slow on TSAN.
NO_FATALS(PrepareClusterForTabletCopy());
TServerDetails* ts0 = ts_map_[cluster_->tablet_server(0)->uuid()];
@@ -119,7 +125,8 @@ TEST_F(TabletCopyClientSessionITest, TestStartTabletCopyWhileSourceBootstrapping
Status s;
while (true) {
if (MonoTime::Now() > deadline) {
- FAIL() << "Timed out waiting for bootstrap: " << s.ToString();
+ LOG(WARNING) << "Test thread timed out waiting for bootstrap: " << s.ToString();
+ return;
}
s = StartTabletCopy(ts1, tablet_id, ts0->uuid(), src_addr, 0,
deadline - MonoTime::Now());
@@ -130,11 +137,11 @@ TEST_F(TabletCopyClientSessionITest, TestStartTabletCopyWhileSourceBootstrapping
if (s.ok()) {
break;
}
- SleepFor(MonoDelta::FromMilliseconds(10));
+ SleepFor(MonoDelta::FromMilliseconds(rand() % 50));
continue;
}
- LOG(INFO) << "Waiting until tablet running...";
- EXPECT_OK(WaitUntilTabletRunning(ts1, tablet_id, deadline - MonoTime::Now()));
+ // If we got here, we either successfully started a tablet copy or we
+ // observed the tablet running.
}, &t));
threads.push_back(t);
}
@@ -144,7 +151,7 @@ TEST_F(TabletCopyClientSessionITest, TestStartTabletCopyWhileSourceBootstrapping
// Wait for one of the threads to succeed with its tablet copy and for the
// tablet to be running on TS 1.
- EXPECT_OK(WaitUntilTabletRunning(ts1, tablet_id, kTimeout));
+ ASSERT_OK(WaitUntilTabletRunning(ts1, tablet_id, kTimeout));
for (auto& t : threads) {
t->Join();
[5/5] kudu git commit: docs: fix gflag name
Posted by to...@apache.org.
docs: fix gflag name
Change-Id: Ie818b2e641569581ac2936a947311aaed9b1c5e0
Reviewed-on: http://gerrit.cloudera.org:8080/5456
Tested-by: Kudu Jenkins
Reviewed-by: Alexey Serbin <as...@cloudera.com>
Reviewed-by: Todd Lipcon <to...@apache.org>
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/2eba7b88
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/2eba7b88
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/2eba7b88
Branch: refs/heads/master
Commit: 2eba7b8846dba70577483f2df1baf0813564383a
Parents: 770f8e1
Author: Adar Dembo <ad...@cloudera.com>
Authored: Fri Dec 9 18:03:54 2016 -0800
Committer: Todd Lipcon <to...@apache.org>
Committed: Mon Dec 12 04:50:32 2016 +0000
----------------------------------------------------------------------
docs/prior_release_notes.adoc | 2 +-
docs/release_notes.adoc | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/kudu/blob/2eba7b88/docs/prior_release_notes.adoc
----------------------------------------------------------------------
diff --git a/docs/prior_release_notes.adoc b/docs/prior_release_notes.adoc
index b18d26b..228592a 100644
--- a/docs/prior_release_notes.adoc
+++ b/docs/prior_release_notes.adoc
@@ -936,7 +936,7 @@ operations will become available in future betas.
The following are known bugs and issues with the current release of Kudu. They will
be addressed in later beta releases.
-* If the Kudu master is configured with the `-log_fsync_all` option, tablet servers
+* If the Kudu master is configured with the `-log_force_fsync_all` option, tablet servers
and clients will experience frequent timeouts, and the cluster may become unusable.
* If a tablet server has a very large number of tablets, it may take several minutes
http://git-wip-us.apache.org/repos/asf/kudu/blob/2eba7b88/docs/release_notes.adoc
----------------------------------------------------------------------
diff --git a/docs/release_notes.adoc b/docs/release_notes.adoc
index ccc5207..a9aa587 100644
--- a/docs/release_notes.adoc
+++ b/docs/release_notes.adoc
@@ -213,7 +213,7 @@ The following are known bugs and issues with the current release of Kudu. They w
be addressed in later releases. Note that this list is not exhaustive, and is meant
to communicate only the most important known issues.
-* If the Kudu master is configured with the `-log_fsync_all` option, tablet servers
+* If the Kudu master is configured with the `-log_force_fsync_all` option, tablet servers
and clients will experience frequent timeouts, and the cluster may become unusable.
* If a tablet server has a very large number of tablets, it may take several minutes