You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kudu.apache.org by ad...@apache.org on 2016/08/16 01:38:18 UTC
[1/3] kudu git commit: Another pass on 0.10.0 release notes
Repository: kudu
Updated Branches:
refs/heads/master 2c3fc7c27 -> 98fe55f5c
Another pass on 0.10.0 release notes
Change-Id: I4414bdebb15d976c025dfce5a3f2bda5768bd5a9
Reviewed-on: http://gerrit.cloudera.org:8080/3979
Tested-by: Kudu Jenkins
Reviewed-by: Adar Dembo <ad...@cloudera.com>
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/aa941858
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/aa941858
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/aa941858
Branch: refs/heads/master
Commit: aa9418585864588abc5cadcc753bbde0cb327798
Parents: 2c3fc7c
Author: Todd Lipcon <to...@apache.org>
Authored: Mon Aug 15 01:02:09 2016 -0700
Committer: Todd Lipcon <to...@apache.org>
Committed: Tue Aug 16 00:43:15 2016 +0000
----------------------------------------------------------------------
docs/release_notes.adoc | 59 ++++++++++++++++++++++++++++++++++++--------
1 file changed, 49 insertions(+), 10 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/kudu/blob/aa941858/docs/release_notes.adoc
----------------------------------------------------------------------
diff --git a/docs/release_notes.adoc b/docs/release_notes.adoc
index fdb5719..53164c4 100644
--- a/docs/release_notes.adoc
+++ b/docs/release_notes.adoc
@@ -56,7 +56,17 @@ Hadoop storage technologies.
[[rn_0.10.0]]
=== Release notes specific to 0.10.0
-Kudu 0.10.0 delivers TODO what it delivers and how compatible it is.
+Kudu 0.10.0 delivers a number of new features, bug fixes, and optimizations,
+detailed below.
+
+Kudu 0.10.0 maintains wire-compatibility with previous releases, meaning
+that applications using the Kudu client libraries may be upgraded either
+before, at the same time, or after the Kudu servers. However, if you begin
+using new features of Kudu 0.10.0 such as manually range-partitioned tables,
+you must first upgrade all clients to this release.
+
+This release does not maintain full Java API or ABI compatibility with
+Kudu 0.9.0 due to a package rename and some other small changes. See below for details.
See also +++<a href="https://issues.apache.org/jira/issues/?jql=project%20%3D%20KUDU%20AND%20status%20%3D%20Resolved
%20AND%20fixVersion%20%3D%200.10.0">JIRAs resolved
@@ -94,9 +104,16 @@ To upgrade to Kudu 0.10.0, see link:installation.html#upgrade[Upgrade from 0.9.x
deprecated and replaced by a new method `TableCreator::add_range_split` which allows
easier use of smart pointers for safe memory management.
-- TODO(dan): should we list the change in buffer size performance due to
- aa3d4473d229807fa8ee27c6ffb5b4b6fb2893a4 here? users may need to tweak something
- to keep their existing performance
+- The Java client's internal buffering has been reworked. Previously, the number of
+ buffered write operations was constrained on a per-tablet-server basis. Now, the configured
+ maximum buffer size constrains the total number of buffered operations across all
+ tablet servers in the cluster. This provides a more consistent bound on the memory
+ usage of the client regardless of the size of the cluster to which it is writing.
+
+ This change can negatively affect the write performance of Java clients which rely on
+ buffered writes. Consider using the `setMutationBufferSpace` API to increase a
+ session's maximum buffer size if write performance seems to be degraded after upgrading
+ to Kudu 0.10.0.
- The "remote bootstrap" process used to copy a tablet replica from one host to
another has been renamed to "Tablet Copy". This resulted in the renaming of
@@ -111,11 +128,33 @@ To upgrade to Kudu 0.10.0, see link:installation.html#upgrade[Upgrade from 0.9.x
[[rn_0.10.0_new_features]]
==== New features
-- TODO(dan): add a release note for non-covering range partitions
+- Users may now manually manage the partitioning of a range-partitioned table.
+ When a table is created, the user may specify a set of range partitions that
+ do not cover the entire available key space. A user may add or drop range
+ partitions to existing tables.
+
+ This feature can be particularly helpful with time series workloads in which
+ new partitions can be created on an hourly or daily basis. Old partitions
+ may be efficiently dropped if the application does not need to retain historical
+ data past a certain point.
+
+// TODO(dan) can we link to the docs once they're committed?
+
+- Support for running Kudu clusters with multiple masters has been stabilized.
+ Users may start a cluster with three or five masters to provide fault tolerance
+ despite a failure of one or two masters, respectively.
+
+ Note that certain tools (e.g. `ksck`) are still lacking complete support for
+ multiple masters. These deficiencies will be addressed in a following release.
-- TODO(adar): add something about multi-master support
+- Kudu now supports the ability to reserve a certain amount of free disk space
+ in each of its configured data directories. If a directory's free disk space
+ drops to less than the configured minimum, Kudu will stop writing to that
+ directory until space becomes available. If no space is available in any
+ configured directory, Kudu will abort.
-- TODO(mpercy): add something about reserved space configuration
+ This feature may be configured using the `fs_data_dirs_reserved_bytes` and
+ `fs_wal_dir_reserved_bytes` flags.
- The Spark integration's `KuduContext` now supports four new methods for writing to
Kudu tables: `insertRows`, `upsertRows`, `updateRows`, and `deleteRows`. These are
@@ -186,8 +225,9 @@ To upgrade to Kudu 0.10.0, see link:installation.html#upgrade[Upgrade from 0.9.x
improve the performance of random-write workloads at the cost of an
incremental increase in disk space usage.
-- The C++ API now has a published Doxygen-based online documentation, which can be found at
- TODO ADD LINK TO DOXYGEN DOC.
+- The Kudu C++ client library now has Doxygen-based
+ link:http://kudu.apache.org/apidocs/cpp-client-api/[API documentation]
+ available online.
- Kudu now
link:http://kudu.apache.org/2016/06/17/raft-consensus-single-node.html[
@@ -195,7 +235,6 @@ To upgrade to Kudu 0.10.0, see link:installation.html#upgrade[Upgrade from 0.9.x
This change simplifies code and will also allow administrators to enable
replication on a previously-unreplicated table. This change is internal and
should not be visible to users.
- TODO(mpercy): does this prevent downgrade? does everything upgrade OK on its own?
[[rn_0.9.1]]
=== Release notes specific to 0.9.1
[3/3] kudu git commit: KUDU-1474: single to multi-master deployment
migration
Posted by ad...@apache.org.
KUDU-1474: single to multi-master deployment migration
This patch introduces the machinery needed to migrate from a single-node
master deployment to multi-master:
1. Inclusion of the tablet copy service in the master. This was easy; the
master module already depends on 'tserver', so registering the service is
just a few lines of code.
2. The new "kudu" command line tool, whose operations are used in migration.
It'll also serve as the basis for KUDU-619.
3. An end-to-end integration test to showcase the migration.
A couple notes about the new tool. I began with a gflags-based approach, but
found it unwieldy after a while, because it's tough to provide a good help
experience, and the lack of positional argument support makes for long
command lines.
I briefly looked at boost but didn't want to reintroduce a library dependency,
so I ended up writing my own (simple) parser. It maps command line arguments to
"actions" and allows for arbitrary levels of nesting, so you can do things like
"kudu tablet copy blah" as well as "kudu fs dump". At each level, if
insufficient arguments are provided, an informative help message is printed
though there's room for improvement here.
Change-Id: I89c741381ced3731736228cd07fe85106ae72541
Reviewed-on: http://gerrit.cloudera.org:8080/3880
Reviewed-by: Todd Lipcon <to...@apache.org>
Tested-by: Kudu Jenkins
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/98fe55f5
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/98fe55f5
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/98fe55f5
Branch: refs/heads/master
Commit: 98fe55f5c492fbf1a77021d160336e4d8164795f
Parents: 1bce731
Author: Adar Dembo <ad...@cloudera.com>
Authored: Wed Jul 27 22:15:06 2016 -0700
Committer: Adar Dembo <ad...@cloudera.com>
Committed: Tue Aug 16 01:33:07 2016 +0000
----------------------------------------------------------------------
build-support/dist_test.py | 1 +
src/kudu/gutil/strings/join.h | 18 ++
src/kudu/integration-tests/CMakeLists.txt | 3 +
.../integration-tests/external_mini_cluster.cc | 2 +-
.../integration-tests/external_mini_cluster.h | 13 +-
.../integration-tests/master_migration-itest.cc | 220 +++++++++++++++++++
src/kudu/master/master.cc | 10 +-
src/kudu/master/sys_catalog.cc | 7 +-
src/kudu/master/sys_catalog.h | 5 +-
src/kudu/tools/CMakeLists.txt | 17 ++
src/kudu/tools/tool_action.cc | 78 +++++++
src/kudu/tools/tool_action.h | 116 ++++++++++
src/kudu/tools/tool_action_fs.cc | 83 +++++++
src/kudu/tools/tool_action_tablet.cc | 200 +++++++++++++++++
src/kudu/tools/tool_main.cc | 164 ++++++++++++++
15 files changed, 926 insertions(+), 11 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/kudu/blob/98fe55f5/build-support/dist_test.py
----------------------------------------------------------------------
diff --git a/build-support/dist_test.py b/build-support/dist_test.py
index bb5e3f2..f56513f 100755
--- a/build-support/dist_test.py
+++ b/build-support/dist_test.py
@@ -78,6 +78,7 @@ DEPS_FOR_ALL = \
#".../example-tweets.txt",
# Tests that require tooling require these.
+ "build/latest/bin/kudu",
"build/latest/bin/kudu-admin",
]
http://git-wip-us.apache.org/repos/asf/kudu/blob/98fe55f5/src/kudu/gutil/strings/join.h
----------------------------------------------------------------------
diff --git a/src/kudu/gutil/strings/join.h b/src/kudu/gutil/strings/join.h
index 1e8e626..4369c4c 100644
--- a/src/kudu/gutil/strings/join.h
+++ b/src/kudu/gutil/strings/join.h
@@ -200,6 +200,24 @@ inline string JoinStrings(const CONTAINER& components,
return result;
}
+// Join the strings produced by calling 'functor' on each element of
+// 'components'.
+template<class CONTAINER, typename FUNC>
+string JoinMapped(const CONTAINER& components,
+ const FUNC& functor,
+ const StringPiece& delim) {
+ string result;
+ for (typename CONTAINER::const_iterator iter = components.begin();
+ iter != components.end();
+ iter++) {
+ if (iter != components.begin()) {
+ result.append(delim.data(), delim.size());
+ }
+ result.append(functor(*iter));
+ }
+ return result;
+}
+
template <class ITERATOR>
void JoinStringsIterator(const ITERATOR& start,
const ITERATOR& end,
http://git-wip-us.apache.org/repos/asf/kudu/blob/98fe55f5/src/kudu/integration-tests/CMakeLists.txt
----------------------------------------------------------------------
diff --git a/src/kudu/integration-tests/CMakeLists.txt b/src/kudu/integration-tests/CMakeLists.txt
index 5b99b7f..89b5fb8 100644
--- a/src/kudu/integration-tests/CMakeLists.txt
+++ b/src/kudu/integration-tests/CMakeLists.txt
@@ -55,6 +55,9 @@ ADD_KUDU_TEST(external_mini_cluster-test RESOURCE_LOCK "master-rpc-ports")
ADD_KUDU_TEST(fuzz-itest)
ADD_KUDU_TEST(linked_list-test RESOURCE_LOCK "master-rpc-ports")
ADD_KUDU_TEST(master_failover-itest RESOURCE_LOCK "master-rpc-ports")
+ADD_KUDU_TEST(master_migration-itest RESOURCE_LOCK "master-rpc-ports")
+ADD_KUDU_TEST_DEPENDENCIES(master_migration-itest
+ kudu)
ADD_KUDU_TEST(master_replication-itest RESOURCE_LOCK "master-rpc-ports")
ADD_KUDU_TEST(master-stress-test RESOURCE_LOCK "master-rpc-ports")
ADD_KUDU_TEST(raft_consensus-itest RUN_SERIAL true)
http://git-wip-us.apache.org/repos/asf/kudu/blob/98fe55f5/src/kudu/integration-tests/external_mini_cluster.cc
----------------------------------------------------------------------
diff --git a/src/kudu/integration-tests/external_mini_cluster.cc b/src/kudu/integration-tests/external_mini_cluster.cc
index 0d3f7c6..4ac0c29 100644
--- a/src/kudu/integration-tests/external_mini_cluster.cc
+++ b/src/kudu/integration-tests/external_mini_cluster.cc
@@ -434,7 +434,7 @@ Status ExternalMiniCluster::GetLeaderMasterIndex(int* idx) {
}
}
if (!found) {
- // There is never a situation where shis should happen, so it's
+ // There is never a situation where this should happen, so it's
// better to exit with a FATAL log message right away vs. return a
// Status::IllegalState().
LOG(FATAL) << "Leader master is not in masters_";
http://git-wip-us.apache.org/repos/asf/kudu/blob/98fe55f5/src/kudu/integration-tests/external_mini_cluster.h
----------------------------------------------------------------------
diff --git a/src/kudu/integration-tests/external_mini_cluster.h b/src/kudu/integration-tests/external_mini_cluster.h
index 4573b7e..abe3f39 100644
--- a/src/kudu/integration-tests/external_mini_cluster.h
+++ b/src/kudu/integration-tests/external_mini_cluster.h
@@ -256,6 +256,16 @@ class ExternalMiniCluster {
const std::string& flag,
const std::string& value);
+ // Returns the path where 'binary' is expected to live, based on
+ // ExternalMiniClusterOptions.daemon_bin_path if it was provided, or on the
+ // path of the currently running executable otherwise.
+ std::string GetBinaryPath(const std::string& binary) const;
+
+ // Returns the path where 'daemon_id' is expected to store its data, based on
+ // ExternalMiniClusterOptions.data_root if it was provided, or on the
+ // standard Kudu test directory otherwise.
+ std::string GetDataPath(const std::string& daemon_id) const;
+
private:
FRIEND_TEST(MasterFailoverTest, TestKillAnyMaster);
@@ -263,9 +273,6 @@ class ExternalMiniCluster {
Status StartDistributedMasters();
- std::string GetBinaryPath(const std::string& binary) const;
- std::string GetDataPath(const std::string& daemon_id) const;
-
Status DeduceBinRoot(std::string* ret);
Status HandleOptions();
http://git-wip-us.apache.org/repos/asf/kudu/blob/98fe55f5/src/kudu/integration-tests/master_migration-itest.cc
----------------------------------------------------------------------
diff --git a/src/kudu/integration-tests/master_migration-itest.cc b/src/kudu/integration-tests/master_migration-itest.cc
new file mode 100644
index 0000000..15ebac1
--- /dev/null
+++ b/src/kudu/integration-tests/master_migration-itest.cc
@@ -0,0 +1,220 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <glog/logging.h>
+#include <gtest/gtest.h>
+#include <memory>
+#include <string>
+#include <vector>
+
+#include "kudu/client/client.h"
+#include "kudu/client/client-test-util.h"
+#include "kudu/gutil/strings/strip.h"
+#include "kudu/gutil/strings/substitute.h"
+#include "kudu/integration-tests/external_mini_cluster.h"
+#include "kudu/master/sys_catalog.h"
+#include "kudu/util/env.h"
+#include "kudu/util/path_util.h"
+#include "kudu/util/subprocess.h"
+#include "kudu/util/test_util.h"
+
+using kudu::client::KuduClient;
+using kudu::client::KuduClientBuilder;
+using kudu::client::KuduColumnSchema;
+using kudu::client::KuduScanner;
+using kudu::client::KuduSchema;
+using kudu::client::KuduSchemaBuilder;
+using kudu::client::KuduTable;
+using kudu::client::KuduTableCreator;
+using kudu::client::sp::shared_ptr;
+using kudu::master::SysCatalogTable;
+using std::string;
+using std::vector;
+using std::unique_ptr;
+using strings::Substitute;
+
+namespace kudu {
+
+class MasterMigrationTest : public KuduTest {
+ public:
+
+ virtual void SetUp() OVERRIDE {
+ KuduTest::SetUp();
+ ASSERT_NO_FATAL_FAILURE(RestartCluster());
+ }
+
+ virtual void TearDown() OVERRIDE {
+ if (cluster_) {
+ cluster_->Shutdown();
+ }
+ KuduTest::TearDown();
+ }
+
+ void RestartCluster() {
+ if (cluster_) {
+ cluster_->Shutdown();
+ }
+ cluster_.reset(new ExternalMiniCluster(ExternalMiniClusterOptions()));
+ ASSERT_OK(cluster_->Start());
+ }
+
+ protected:
+ unique_ptr<ExternalMiniCluster> cluster_;
+};
+
+static Status CreateTable(ExternalMiniCluster* cluster,
+ const std::string& table_name) {
+ shared_ptr<KuduClient> client;
+ KuduClientBuilder builder;
+ RETURN_NOT_OK(cluster->CreateClient(builder, &client));
+ KuduSchema schema;
+ KuduSchemaBuilder b;
+ b.AddColumn("key")->Type(KuduColumnSchema::INT32)->NotNull()->PrimaryKey();
+ RETURN_NOT_OK(b.Build(&schema));
+ unique_ptr<KuduTableCreator> table_creator(client->NewTableCreator());
+ return table_creator->table_name(table_name)
+ .schema(&schema)
+ .set_range_partition_columns({ "key" })
+ .num_replicas(1)
+ .Create();
+}
+
+// Tests migration of a deployment from one master to multiple masters.
+TEST_F(MasterMigrationTest, TestEndToEndMigration) {
+ const vector<uint16_t> kMasterRpcPorts = { 11010, 11011, 11012 };
+ const string kTableName = "test";
+ const string kBinPath = cluster_->GetBinaryPath("kudu");
+
+ // Initial state: single-master cluster with one table.
+ ASSERT_OK(CreateTable(cluster_.get(), kTableName));
+ cluster_->Shutdown();
+
+ // List of every master's UUID and port. Used when rewriting the single
+ // master's cmeta.
+ vector<pair<string, uint64_t>> master_uuids_and_ports;
+ master_uuids_and_ports.emplace_back(cluster_->master()->uuid(), kMasterRpcPorts[0]);
+
+ // Format a filesystem tree for each of the new masters and get the uuids.
+ for (int i = 1; i < kMasterRpcPorts.size(); i++) {
+ string data_root = cluster_->GetDataPath(Substitute("master-$0", i));
+ {
+ vector<string> args = {
+ kBinPath,
+ "fs",
+ "format",
+ "--fs_wal_dir=" + data_root,
+ "--fs_data_dirs=" + data_root
+ };
+ ASSERT_OK(Subprocess::Call(args));
+ }
+ {
+ vector<string> args = {
+ kBinPath,
+ "fs",
+ "print_uuid",
+ "--fs_wal_dir=" + data_root,
+ "--fs_data_dirs=" + data_root
+ };
+ string uuid;
+ ASSERT_OK(Subprocess::Call(args, &uuid));
+ StripWhiteSpace(&uuid);
+ master_uuids_and_ports.emplace_back(uuid, kMasterRpcPorts[i]);
+ }
+ }
+
+ // Rewrite the single master's cmeta to reflect the new Raft configuration.
+ {
+ string data_root = cluster_->GetDataPath("master-0");
+ vector<string> args = {
+ kBinPath,
+ "tablet",
+ "rewrite_raft_config",
+ "--fs_wal_dir=" + data_root,
+ "--fs_data_dirs=" + data_root,
+ SysCatalogTable::kSysCatalogTabletId
+ };
+ for (const auto& m : master_uuids_and_ports) {
+ args.push_back(Substitute("$0:127.0.0.1:$1", m.first, m.second));
+ }
+ ASSERT_OK(Subprocess::Call(args));
+ }
+
+ // Temporarily bring up the cluster (in its old configuration) to remote
+ // bootstrap the new masters.
+ //
+ // The single-node master is running in an odd state. The cmeta changes have
+ // made it aware that it should replicate to the new masters, but they're not
+ // actually running. Thus, it cannot become leader or do any real work. But,
+ // it can still service remote bootstrap requests.
+ NO_FATALS(RestartCluster());
+
+ // Use remote bootstrap to copy the master tablet to each of the new masters'
+ // filesystems.
+ for (int i = 1; i < kMasterRpcPorts.size(); i++) {
+ string data_root = cluster_->GetDataPath(Substitute("master-$0", i));
+ vector<string> args = {
+ kBinPath,
+ "tablet",
+ "copy",
+ "--fs_wal_dir=" + data_root,
+ "--fs_data_dirs=" + data_root,
+ SysCatalogTable::kSysCatalogTabletId,
+ cluster_->master()->bound_rpc_hostport().ToString()
+ };
+ ASSERT_OK(Subprocess::Call(args));
+ }
+
+ // Bring down the old cluster configuration and bring up the new one.
+ cluster_->Shutdown();
+ ExternalMiniClusterOptions opts;
+ opts.master_rpc_ports = kMasterRpcPorts;
+ opts.num_masters = kMasterRpcPorts.size();
+ ExternalMiniCluster migrated_cluster(opts);
+ ASSERT_OK(migrated_cluster.Start());
+
+
+ // Perform an operation that requires an elected leader.
+ shared_ptr<KuduClient> client;
+ KuduClientBuilder builder;
+ ASSERT_OK(migrated_cluster.CreateClient(builder, &client));
+
+ shared_ptr<KuduTable> table;
+ ASSERT_OK(client->OpenTable(kTableName, &table));
+ ASSERT_EQ(0, CountTableRows(table.get()));
+
+ // Perform an operation that requires replication.
+ ASSERT_OK(CreateTable(&migrated_cluster, "second_table"));
+
+ // Repeat these operations with each of the masters paused.
+ //
+ // Only in slow mode.
+ if (AllowSlowTests()) {
+ for (int i = 0; i < migrated_cluster.num_masters(); i++) {
+ migrated_cluster.master(i)->Pause();
+ ScopedResumeExternalDaemon resume_daemon(migrated_cluster.master(i));
+ ASSERT_OK(client->OpenTable(kTableName, &table));
+ ASSERT_EQ(0, CountTableRows(table.get()));
+
+ // See MasterFailoverTest.TestCreateTableSync to understand why we must
+ // check for IsAlreadyPresent as well.
+ Status s = CreateTable(&migrated_cluster, Substitute("table-$0", i));
+ ASSERT_TRUE(s.ok() || s.IsAlreadyPresent());
+ }
+ }
+}
+
+} // namespace kudu
http://git-wip-us.apache.org/repos/asf/kudu/blob/98fe55f5/src/kudu/master/master.cc
----------------------------------------------------------------------
diff --git a/src/kudu/master/master.cc b/src/kudu/master/master.cc
index c8a4ab8..efbf8cf 100644
--- a/src/kudu/master/master.cc
+++ b/src/kudu/master/master.cc
@@ -36,6 +36,7 @@
#include "kudu/rpc/service_if.h"
#include "kudu/rpc/service_pool.h"
#include "kudu/server/rpc_server.h"
+#include "kudu/tserver/tablet_copy_service.h"
#include "kudu/tserver/tablet_service.h"
#include "kudu/util/flag_tags.h"
#include "kudu/util/maintenance_manager.h"
@@ -55,6 +56,7 @@ using std::vector;
using kudu::consensus::RaftPeerPB;
using kudu::rpc::ServiceIf;
using kudu::tserver::ConsensusServiceImpl;
+using kudu::tserver::TabletCopyServiceImpl;
using strings::Substitute;
namespace kudu {
@@ -110,12 +112,14 @@ Status Master::StartAsync() {
RETURN_NOT_OK(maintenance_manager_->Init());
gscoped_ptr<ServiceIf> impl(new MasterServiceImpl(this));
- gscoped_ptr<ServiceIf> consensus_service(new ConsensusServiceImpl(metric_entity(),
- result_tracker(),
- catalog_manager_.get()));
+ gscoped_ptr<ServiceIf> consensus_service(new ConsensusServiceImpl(
+ metric_entity(), result_tracker(), catalog_manager_.get()));
+ gscoped_ptr<ServiceIf> tablet_copy_service(new TabletCopyServiceImpl(
+ fs_manager_.get(), catalog_manager_.get(), metric_entity(), result_tracker()));
RETURN_NOT_OK(ServerBase::RegisterService(std::move(impl)));
RETURN_NOT_OK(ServerBase::RegisterService(std::move(consensus_service)));
+ RETURN_NOT_OK(ServerBase::RegisterService(std::move(tablet_copy_service)));
RETURN_NOT_OK(ServerBase::Start());
// Now that we've bound, construct our ServerRegistrationPB.
http://git-wip-us.apache.org/repos/asf/kudu/blob/98fe55f5/src/kudu/master/sys_catalog.cc
----------------------------------------------------------------------
diff --git a/src/kudu/master/sys_catalog.cc b/src/kudu/master/sys_catalog.cc
index 0658cb3..02af8c0 100644
--- a/src/kudu/master/sys_catalog.cc
+++ b/src/kudu/master/sys_catalog.cc
@@ -75,13 +75,14 @@ using strings::Substitute;
namespace kudu {
namespace master {
-static const char* const kSysCatalogTabletId = "00000000000000000000000000000000";
-
static const char* const kSysCatalogTableColType = "entry_type";
static const char* const kSysCatalogTableColId = "entry_id";
static const char* const kSysCatalogTableColMetadata = "metadata";
-const char* SysCatalogTable::kInjectedFailureStatusMsg = "INJECTED FAILURE";
+const char* const SysCatalogTable::kSysCatalogTabletId =
+ "00000000000000000000000000000000";
+const char* const SysCatalogTable::kInjectedFailureStatusMsg =
+ "INJECTED FAILURE";
SysCatalogTable::SysCatalogTable(Master* master, MetricRegistry* metrics,
ElectedLeaderCallback leader_cb)
http://git-wip-us.apache.org/repos/asf/kudu/blob/98fe55f5/src/kudu/master/sys_catalog.h
----------------------------------------------------------------------
diff --git a/src/kudu/master/sys_catalog.h b/src/kudu/master/sys_catalog.h
index 9d22ef9..ff11ca3 100644
--- a/src/kudu/master/sys_catalog.h
+++ b/src/kudu/master/sys_catalog.h
@@ -63,6 +63,9 @@ class TabletVisitor {
// as a "normal table", instead we have Master APIs to query the table.
class SysCatalogTable {
public:
+ // Magic ID of the system tablet.
+ static const char* const kSysCatalogTabletId;
+
typedef Callback<Status()> ElectedLeaderCallback;
enum CatalogEntryType {
@@ -186,7 +189,7 @@ class SysCatalogTable {
// Special string injected into SyncWrite() random failures (if enabled).
//
// Only useful for tests.
- static const char* kInjectedFailureStatusMsg;
+ static const char* const kInjectedFailureStatusMsg;
// Table schema, without IDs, used to send messages to the TabletPeer
Schema schema_;
http://git-wip-us.apache.org/repos/asf/kudu/blob/98fe55f5/src/kudu/tools/CMakeLists.txt
----------------------------------------------------------------------
diff --git a/src/kudu/tools/CMakeLists.txt b/src/kudu/tools/CMakeLists.txt
index ab7c0c5..b455924 100644
--- a/src/kudu/tools/CMakeLists.txt
+++ b/src/kudu/tools/CMakeLists.txt
@@ -92,6 +92,23 @@ target_link_libraries(kudu-pbc-dump
${LINK_LIBS}
)
+add_executable(kudu
+ tool_action.cc
+ tool_action_fs.cc
+ tool_action_tablet.cc
+ tool_main.cc
+)
+target_link_libraries(kudu
+ consensus
+ gutil
+ kudu_common
+ kudu_fs
+ kudu_util
+ master
+ tserver
+ ${KUDU_BASE_LIBS}
+)
+
set(KUDU_TEST_LINK_LIBS
ksck
kudu_tools_util
http://git-wip-us.apache.org/repos/asf/kudu/blob/98fe55f5/src/kudu/tools/tool_action.cc
----------------------------------------------------------------------
diff --git a/src/kudu/tools/tool_action.cc b/src/kudu/tools/tool_action.cc
new file mode 100644
index 0000000..d9a4d73
--- /dev/null
+++ b/src/kudu/tools/tool_action.cc
@@ -0,0 +1,78 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "kudu/tools/tool_action.h"
+
+#include <deque>
+#include <string>
+#include <vector>
+
+#include "kudu/gutil/strings/join.h"
+#include "kudu/gutil/strings/substitute.h"
+
+using std::deque;
+using std::string;
+using std::vector;
+using strings::Substitute;
+
+namespace kudu {
+namespace tools {
+
+string BuildActionChainString(const vector<Action>& chain) {
+ return JoinMapped(chain, [](const Action& a){ return a.name; }, " ");
+}
+
+string BuildUsageString(const vector<Action>& chain) {
+ return Substitute("Usage: $0", BuildActionChainString(chain));
+}
+
+string BuildHelpString(const vector<Action>& sub_actions, string usage_str) {
+ string msg = Substitute("$0 <action>\n", usage_str);
+ msg += "Action can be one of the following:\n";
+ for (const auto& a : sub_actions) {
+ msg += Substitute(" $0 : $1\n", a.name, a.description);
+ }
+ return msg;
+}
+
+string BuildLeafActionHelpString(const vector<Action>& chain) {
+ DCHECK(!chain.empty());
+ Action action = chain.back();
+ string msg = Substitute("$0\n", BuildUsageString(chain));
+ msg += Substitute("$0\n", action.description);
+ return msg;
+}
+
+string BuildNonLeafActionHelpString(const vector<Action>& chain) {
+ string usage = BuildUsageString(chain);
+ DCHECK(!chain.empty());
+ return BuildHelpString(chain.back().sub_actions, usage);
+}
+
+Status ParseAndRemoveArg(const char* arg_name,
+ deque<string>* remaining_args,
+ string* parsed_arg) {
+ if (remaining_args->empty()) {
+ return Status::InvalidArgument(Substitute("must provide $0", arg_name));
+ }
+ *parsed_arg = remaining_args->front();
+ remaining_args->pop_front();
+ return Status::OK();
+}
+
+} // namespace tools
+} // namespace kudu
http://git-wip-us.apache.org/repos/asf/kudu/blob/98fe55f5/src/kudu/tools/tool_action.h
----------------------------------------------------------------------
diff --git a/src/kudu/tools/tool_action.h b/src/kudu/tools/tool_action.h
new file mode 100644
index 0000000..93e0cc4
--- /dev/null
+++ b/src/kudu/tools/tool_action.h
@@ -0,0 +1,116 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <deque>
+#include <glog/logging.h>
+#include <string>
+#include <vector>
+
+#include "kudu/gutil/strings/substitute.h"
+#include "kudu/util/status.h"
+
+namespace kudu {
+namespace tools {
+
+// Encapsulates all knowledge for a particular tool action.
+//
+// All actions are arranged in a tree. Leaf actions are invokable: they will
+// do something meaningful when run() is called. Non-leaf actions do not have
+// a run(); an attempt to invoke them will yield help() instead.
+//
+// Sample action tree:
+//
+// fs
+// | |
+// +--+ +--+
+// | |
+// format print_uuid
+//
+// Given this tree:
+// - "<program> fs" will print some text explaining all of fs's actions.
+// - "<program> fs format" will format a filesystem.
+// - "<program> fs print_uuid" will print a filesystem's UUID.
+struct Action {
+ // The name of the action (e.g. "fs").
+ std::string name;
+
+ // The description of the action (e.g. "Operate on a local Kudu filesystem").
+ std::string description;
+
+ // Invokes an action, passing in the complete action chain and all remaining
+ // command line arguments. The arguments are passed by value so that the
+ // function can modify them if need be.
+ std::function<Status(const std::vector<Action>&,
+ std::deque<std::string>)> run;
+
+ // Get help for an action, passing in the complete action chain.
+ std::function<std::string(const std::vector<Action>&)> help;
+
+ // This action's children.
+ std::vector<Action> sub_actions;
+};
+
+// Constructs a string with the names of all actions in the chain
+// (e.g. "<program> fs format").
+std::string BuildActionChainString(const std::vector<Action>& chain);
+
+// Constructs a usage string (e.g. "Usage: <program> fs format").
+std::string BuildUsageString(const std::vector<Action>& chain);
+
+// Constructs a help string suitable for leaf actions.
+std::string BuildLeafActionHelpString(const std::vector<Action>& chain);
+
+// Constructs a help string suitable for non-leaf actions.
+std::string BuildNonLeafActionHelpString(const std::vector<Action>& chain);
+
+// Constructs a string appropriate for displaying program help, using
+// 'sub_actions' as a list of actions to include and 'usage_str' as a string
+// to prepend.
+std::string BuildHelpString(const std::vector<Action>& sub_actions,
+ std::string usage_str);
+
+// Removes one argument from 'remaining_args' and stores it in 'parsed_arg'.
+//
+// If 'remaining_args' is empty, returns InvalidArgument with 'arg_name' in the
+// message.
+Status ParseAndRemoveArg(const char* arg_name,
+ std::deque<std::string>* remaining_args,
+ std::string* parsed_arg);
+
+// Checks that 'args' is empty. If not, returns a bad status.
+template <typename CONTAINER>
+Status CheckNoMoreArgs(const std::vector<Action>& chain,
+ const CONTAINER& args) {
+ if (args.empty()) {
+ return Status::OK();
+ }
+ DCHECK(!chain.empty());
+ Action action = chain.back();
+ return Status::InvalidArgument(strings::Substitute(
+ "too many arguments\n$0", action.help(chain)));
+}
+
+// Returns the "fs" action node.
+Action BuildFsAction();
+
+// Returns the "tablet" action node.
+Action BuildTabletAction();
+
+} // namespace tools
+} // namespace kudu
http://git-wip-us.apache.org/repos/asf/kudu/blob/98fe55f5/src/kudu/tools/tool_action_fs.cc
----------------------------------------------------------------------
diff --git a/src/kudu/tools/tool_action_fs.cc b/src/kudu/tools/tool_action_fs.cc
new file mode 100644
index 0000000..08ec6ff
--- /dev/null
+++ b/src/kudu/tools/tool_action_fs.cc
@@ -0,0 +1,83 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "kudu/tools/tool_action.h"
+
+#include <deque>
+#include <iostream>
+#include <string>
+
+#include "kudu/fs/fs_manager.h"
+#include "kudu/util/status.h"
+
+using std::cout;
+using std::deque;
+using std::endl;
+using std::string;
+using std::vector;
+
+namespace kudu {
+namespace tools {
+
+namespace {
+
+Status Format(const vector<Action>& chain, deque<string> args) {
+ RETURN_NOT_OK(CheckNoMoreArgs(chain, args));
+
+ FsManager fs_manager(Env::Default(), FsManagerOpts());
+ return fs_manager.CreateInitialFileSystemLayout();
+}
+
+Status PrintUuid(const vector<Action>& chain, deque<string> args) {
+ RETURN_NOT_OK(CheckNoMoreArgs(chain, args));
+
+ FsManagerOpts opts;
+ opts.read_only = true;
+ FsManager fs_manager(Env::Default(), opts);
+ RETURN_NOT_OK(fs_manager.Open());
+ cout << fs_manager.uuid() << endl;
+ return Status::OK();
+}
+
+} // anonymous namespace
+
+Action BuildFsAction() {
+ Action fs_format;
+ fs_format.name = "format";
+ fs_format.description = "Format a new Kudu filesystem";
+ fs_format.help = &BuildLeafActionHelpString;
+ fs_format.run = &Format;
+
+ Action fs_print_uuid;
+ fs_print_uuid.name = "print_uuid";
+ fs_print_uuid.description = "Print the UUID of a Kudu filesystem";
+ fs_print_uuid.help = &BuildLeafActionHelpString;
+ fs_print_uuid.run = &PrintUuid;
+
+ Action fs;
+ fs.name = "fs";
+ fs.description = "Operate on a local Kudu filesystem";
+ fs.help = &BuildNonLeafActionHelpString;
+ fs.sub_actions = {
+ fs_format,
+ fs_print_uuid
+ };
+ return fs;
+}
+
+} // namespace tools
+} // namespace kudu
http://git-wip-us.apache.org/repos/asf/kudu/blob/98fe55f5/src/kudu/tools/tool_action_tablet.cc
----------------------------------------------------------------------
diff --git a/src/kudu/tools/tool_action_tablet.cc b/src/kudu/tools/tool_action_tablet.cc
new file mode 100644
index 0000000..9d02942
--- /dev/null
+++ b/src/kudu/tools/tool_action_tablet.cc
@@ -0,0 +1,200 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "kudu/tools/tool_action.h"
+
+#include <deque>
+#include <memory>
+#include <string>
+#include <utility>
+
+#include "kudu/common/wire_protocol.h"
+#include "kudu/consensus/consensus_meta.h"
+#include "kudu/fs/fs_manager.h"
+#include "kudu/gutil/strings/substitute.h"
+#include "kudu/master/sys_catalog.h"
+#include "kudu/rpc/messenger.h"
+#include "kudu/tserver/tablet_copy_client.h"
+#include "kudu/util/env.h"
+#include "kudu/util/env_util.h"
+#include "kudu/util/net/net_util.h"
+#include "kudu/util/status.h"
+
+using kudu::consensus::ConsensusMetadata;
+using kudu::consensus::RaftConfigPB;
+using kudu::consensus::RaftPeerPB;
+using kudu::rpc::Messenger;
+using kudu::rpc::MessengerBuilder;
+using kudu::tserver::TabletCopyClient;
+using std::deque;
+using std::shared_ptr;
+using std::string;
+using std::unique_ptr;
+using std::vector;
+using strings::Substitute;
+
+namespace kudu {
+namespace tools {
+
+namespace {
+
+// Parses a colon-delimited string containing a hostname or IP address and port
+// into its respective parts. For example, "localhost:12345" parses into
+// hostname=localhost, and port=12345.
+//
+// Does not allow a port with value 0.
+Status ParseHostPortString(const string& hostport_str, HostPort* hostport) {
+ HostPort hp;
+ Status s = hp.ParseString(hostport_str, 0);
+ if (!s.ok()) {
+ return s.CloneAndPrepend(Substitute(
+ "error while parsing peer '$0'", hostport_str));
+ }
+ if (hp.port() == 0) {
+ return Status::InvalidArgument(
+ Substitute("peer '$0' has port of 0", hostport_str));
+ }
+ *hostport = hp;
+ return Status::OK();
+}
+
+// Parses a colon-delimited string containing a uuid, hostname or IP address,
+// and port into its respective parts. For example,
+// "1c7f19e7ecad4f918c0d3d23180fdb18:localhost:12345" parses into
+// uuid=1c7f19e7ecad4f918c0d3d23180fdb18, hostname=localhost, and port=12345.
+Status ParsePeerString(const string& peer_str,
+ string* uuid,
+ HostPort* hostport) {
+ int first_colon_idx = peer_str.find(":");
+ if (first_colon_idx == string::npos) {
+ return Status::InvalidArgument(Substitute("bad peer '$0'", peer_str));
+ }
+ string hostport_str = peer_str.substr(first_colon_idx + 1);
+ RETURN_NOT_OK(ParseHostPortString(hostport_str, hostport));
+ *uuid = peer_str.substr(0, first_colon_idx);
+ return Status::OK();
+}
+
+Status RewriteRaftConfig(const vector<Action>& chain, deque<string> args) {
+ // Parse tablet ID argument.
+ string tablet_id;
+ RETURN_NOT_OK(ParseAndRemoveArg("tablet ID", &args, &tablet_id));
+ if (tablet_id != master::SysCatalogTable::kSysCatalogTabletId) {
+ LOG(WARNING) << "Master will not notice rewritten Raft config of regular "
+ << "tablets. A regular Raft config change must occur.";
+ }
+
+ // Parse peer arguments.
+ vector<pair<string, HostPort>> peers;
+ for (const auto& arg : args) {
+ pair<string, HostPort> parsed_peer;
+ RETURN_NOT_OK(ParsePeerString(arg,
+ &parsed_peer.first, &parsed_peer.second));
+ peers.push_back(parsed_peer);
+ }
+ if (peers.empty()) {
+ return Status::InvalidArgument(Substitute(
+ "must provide at least one peer of form uuid:hostname:port"));
+ }
+
+ // Make a copy of the old file before rewriting it.
+ Env* env = Env::Default();
+ FsManager fs_manager(env, FsManagerOpts());
+ RETURN_NOT_OK(fs_manager.Open());
+ string cmeta_filename = fs_manager.GetConsensusMetadataPath(tablet_id);
+ string backup_filename = Substitute("$0.pre_rewrite.$1",
+ cmeta_filename, env->NowMicros());
+ WritableFileOptions opts;
+ opts.mode = Env::CREATE_NON_EXISTING;
+ opts.sync_on_close = true;
+ RETURN_NOT_OK(env_util::CopyFile(env, cmeta_filename, backup_filename, opts));
+ LOG(INFO) << "Backed up current config to " << backup_filename;
+
+ // Load the cmeta file and rewrite the raft config.
+ unique_ptr<ConsensusMetadata> cmeta;
+ RETURN_NOT_OK(ConsensusMetadata::Load(&fs_manager, tablet_id,
+ fs_manager.uuid(), &cmeta));
+ RaftConfigPB current_config = cmeta->committed_config();
+ RaftConfigPB new_config = current_config;
+ new_config.clear_peers();
+ for (const auto& p : peers) {
+ RaftPeerPB new_peer;
+ new_peer.set_member_type(RaftPeerPB::VOTER);
+ new_peer.set_permanent_uuid(p.first);
+ HostPortPB new_peer_host_port_pb;
+ RETURN_NOT_OK(HostPortToPB(p.second, &new_peer_host_port_pb));
+ new_peer.mutable_last_known_addr()->CopyFrom(new_peer_host_port_pb);
+ new_config.add_peers()->CopyFrom(new_peer);
+ }
+ cmeta->set_committed_config(new_config);
+ return cmeta->Flush();
+}
+
+Status Copy(const vector<Action>& chain, deque<string> args) {
+ // Parse the tablet ID and source arguments.
+ string tablet_id;
+ RETURN_NOT_OK(ParseAndRemoveArg("tablet ID", &args, &tablet_id));
+ string rpc_address;
+ RETURN_NOT_OK(ParseAndRemoveArg("source RPC address of form hostname:port",
+ &args, &rpc_address));
+ RETURN_NOT_OK(CheckNoMoreArgs(chain, args));
+
+ HostPort hp;
+ RETURN_NOT_OK(ParseHostPortString(rpc_address, &hp));
+
+ // Copy the tablet over.
+ FsManager fs_manager(Env::Default(), FsManagerOpts());
+ RETURN_NOT_OK(fs_manager.Open());
+ MessengerBuilder builder("tablet_copy_client");
+ shared_ptr<Messenger> messenger;
+ builder.Build(&messenger);
+ TabletCopyClient client(tablet_id, &fs_manager, messenger);
+ RETURN_NOT_OK(client.Start(hp, nullptr));
+ RETURN_NOT_OK(client.FetchAll(nullptr));
+ return client.Finish();
+}
+
+} // anonymous namespace
+
+Action BuildTabletAction() {
+ Action tablet_rewrite_raft_config;
+ tablet_rewrite_raft_config.name = "rewrite_raft_config";
+ tablet_rewrite_raft_config.description = "Rewrite a replica's Raft configuration";
+ tablet_rewrite_raft_config.help = &BuildLeafActionHelpString;
+ tablet_rewrite_raft_config.run = &RewriteRaftConfig;
+
+ // TODO: Need to include required arguments in the help for these actions.
+ Action tablet_copy;
+ tablet_copy.name = "copy";
+ tablet_copy.description = "Copy a replica from a remote server";
+ tablet_copy.help = &BuildLeafActionHelpString;
+ tablet_copy.run = &Copy;
+
+ Action tablet;
+ tablet.name = "tablet";
+ tablet.description = "Operate on a local Kudu replica";
+ tablet.help = &BuildNonLeafActionHelpString;
+ tablet.sub_actions = {
+ tablet_rewrite_raft_config,
+ tablet_copy
+ };
+ return tablet;
+}
+
+} // namespace tools
+} // namespace kudu
+
http://git-wip-us.apache.org/repos/asf/kudu/blob/98fe55f5/src/kudu/tools/tool_main.cc
----------------------------------------------------------------------
diff --git a/src/kudu/tools/tool_main.cc b/src/kudu/tools/tool_main.cc
new file mode 100644
index 0000000..82c9e0c
--- /dev/null
+++ b/src/kudu/tools/tool_main.cc
@@ -0,0 +1,164 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <deque>
+#include <gflags/gflags.h>
+#include <glog/logging.h>
+#include <iostream>
+#include <string>
+#include <vector>
+
+#include "kudu/gutil/strings/substitute.h"
+#include "kudu/tools/tool_action.h"
+#include "kudu/util/flags.h"
+#include "kudu/util/logging.h"
+#include "kudu/util/status.h"
+
+DECLARE_bool(help);
+DECLARE_bool(helpshort);
+DECLARE_string(helpon);
+DECLARE_string(helpmatch);
+DECLARE_bool(helppackage);
+DECLARE_bool(helpxml);
+
+using std::cerr;
+using std::cout;
+using std::deque;
+using std::endl;
+using std::string;
+using std::vector;
+using strings::Substitute;
+
+namespace kudu {
+namespace tools {
+
+int DispatchCommand(const vector<Action>& chain, const deque<string>& args) {
+ DCHECK(!chain.empty());
+ Action action = chain.back();
+ Status s = action.run(chain, args);
+ if (s.ok()) {
+ return 0;
+ } else {
+ cerr << s.ToString() << endl;
+ return 1;
+ }
+}
+
+int RunTool(const Action& root, int argc, char** argv, bool show_help) {
+ // Initialize arg parsing state.
+ vector<Action> chain = { root };
+
+ // Parse the arguments, matching them up with actions.
+ for (int i = 1; i < argc; i++) {
+ const Action* cur = &chain.back();
+ const auto& sub_actions = cur->sub_actions;
+ if (sub_actions.empty()) {
+ // We've reached an invokable action.
+ if (show_help) {
+ cerr << cur->help(chain) << endl;
+ return 1;
+ } else {
+ // Invoke it with whatever arguments remain.
+ deque<string> remaining_args;
+ for (int j = i; j < argc; j++) {
+ remaining_args.push_back(argv[j]);
+ }
+ return DispatchCommand(chain, remaining_args);
+ }
+ }
+
+ // This action is not invokable. Interpret the next command line argument
+ // as a subaction and continue parsing.
+ const Action* next = nullptr;
+ for (const auto& a : sub_actions) {
+ if (a.name == argv[i]) {
+ next = &a;
+ break;
+ }
+ }
+
+ if (next == nullptr) {
+ // We couldn't find a subaction for the next argument. Raise an error.
+ string msg = Substitute("$0 $1\n",
+ BuildActionChainString(chain), argv[i]);
+ msg += BuildHelpString(sub_actions, BuildUsageString(chain));
+ Status s = Status::InvalidArgument(msg);
+ cerr << s.ToString() << endl;
+ return 1;
+ }
+
+ // We're done parsing this argument. Loop and continue.
+ chain.emplace_back(*next);
+ }
+
+ // We made it to a subaction with no arguments left. Run the subaction if
+ // possible, otherwise print its help.
+ const Action* last = &chain.back();
+ if (show_help || !last->run) {
+ cerr << last->help(chain) << endl;
+ return 1;
+ } else {
+ DCHECK(last->run);
+ return DispatchCommand(chain, {});
+ }
+}
+
+} // namespace tools
+} // namespace kudu
+
+static bool ParseCommandLineFlags(int* argc, char*** argv) {
+ // Hide the regular gflags help unless --helpfull is used.
+ //
+ // Inspired by https://github.com/gflags/gflags/issues/43#issuecomment-168280647.
+ bool show_help = false;
+ gflags::ParseCommandLineNonHelpFlags(argc, argv, true);
+ if (FLAGS_help ||
+ FLAGS_helpshort ||
+ !FLAGS_helpon.empty() ||
+ !FLAGS_helpmatch.empty() ||
+ FLAGS_helppackage ||
+ FLAGS_helpxml) {
+ FLAGS_help = false;
+ FLAGS_helpshort = false;
+ FLAGS_helpon = "";
+ FLAGS_helpmatch = "";
+ FLAGS_helppackage = false;
+ FLAGS_helpxml = false;
+ show_help = true;
+ }
+ gflags::HandleCommandLineHelpFlags();
+ return show_help;
+}
+
+int main(int argc, char** argv) {
+ kudu::tools::Action root = {
+ argv[0],
+ "The root action", // doesn't matter, won't get printed
+ nullptr,
+ &kudu::tools::BuildNonLeafActionHelpString,
+ {
+ kudu::tools::BuildFsAction(),
+ kudu::tools::BuildTabletAction()
+ }
+ };
+ string usage = root.help({ root });
+ google::SetUsageMessage(usage);
+ bool show_help = ParseCommandLineFlags(&argc, &argv);
+ FLAGS_logtostderr = true;
+ kudu::InitGoogleLoggingSafe(argv[0]);
+ return kudu::tools::RunTool(root, argc, argv, show_help);
+}
[2/3] kudu git commit: fs: allow format with user-specified uuid
Posted by ad...@apache.org.
fs: allow format with user-specified uuid
The CLI tool is going to use this for the "handling permanent failure"
workflow.
Change-Id: Ib4189d6150a263b7dde304dd19a449d3c0ab6c8c
Reviewed-on: http://gerrit.cloudera.org:8080/3968
Reviewed-by: Todd Lipcon <to...@apache.org>
Tested-by: Adar Dembo <ad...@cloudera.com>
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/1bce7312
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/1bce7312
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/1bce7312
Branch: refs/heads/master
Commit: 1bce73126799cb046755c0555bbe5be7622c9ace
Parents: aa94185
Author: Adar Dembo <ad...@cloudera.com>
Authored: Thu Aug 11 21:14:36 2016 -0700
Committer: Adar Dembo <ad...@cloudera.com>
Committed: Tue Aug 16 01:32:58 2016 +0000
----------------------------------------------------------------------
src/kudu/fs/fs_manager-test.cc | 25 ++++++++++++++--
src/kudu/fs/fs_manager.cc | 17 ++++++++---
src/kudu/fs/fs_manager.h | 11 +++++--
src/kudu/util/CMakeLists.txt | 1 +
src/kudu/util/oid_generator-test.cc | 50 ++++++++++++++++++++++++++++++++
src/kudu/util/oid_generator.cc | 37 +++++++++++++++++++----
src/kudu/util/oid_generator.h | 12 ++++++++
7 files changed, 139 insertions(+), 14 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/kudu/blob/1bce7312/src/kudu/fs/fs_manager-test.cc
----------------------------------------------------------------------
diff --git a/src/kudu/fs/fs_manager-test.cc b/src/kudu/fs/fs_manager-test.cc
index 922e617..21bb460 100644
--- a/src/kudu/fs/fs_manager-test.cc
+++ b/src/kudu/fs/fs_manager-test.cc
@@ -21,12 +21,15 @@
#include "kudu/fs/block_manager.h"
#include "kudu/fs/fs_manager.h"
+#include "kudu/gutil/strings/substitute.h"
#include "kudu/gutil/strings/util.h"
#include "kudu/util/metrics.h"
+#include "kudu/util/oid_generator.h"
#include "kudu/util/test_macros.h"
#include "kudu/util/test_util.h"
using std::shared_ptr;
+using strings::Substitute;
namespace kudu {
@@ -155,7 +158,7 @@ TEST_F(FsManagerTestBase, TestCannotUseNonEmptyFsRoot) {
}
TEST_F(FsManagerTestBase, TestEmptyWALPath) {
- ReinitFsManager("", vector<string>());
+ ReinitFsManager("", {});
Status s = fs_manager()->CreateInitialFileSystemLayout();
ASSERT_TRUE(s.IsIOError());
ASSERT_STR_CONTAINS(s.ToString(), "directory (fs_wal_dir) not provided");
@@ -165,7 +168,7 @@ TEST_F(FsManagerTestBase, TestOnlyWALPath) {
string path = GetTestPath("new_fs_root");
ASSERT_OK(env_->CreateDir(path));
- ReinitFsManager(path, vector<string>());
+ ReinitFsManager(path, {});
ASSERT_OK(fs_manager()->CreateInitialFileSystemLayout());
ASSERT_TRUE(HasPrefixString(fs_manager()->GetWalsRootDir(), path));
ASSERT_TRUE(HasPrefixString(fs_manager()->GetConsensusMetadataDir(), path));
@@ -175,4 +178,22 @@ TEST_F(FsManagerTestBase, TestOnlyWALPath) {
ASSERT_TRUE(HasPrefixString(data_dirs[0], path));
}
+TEST_F(FsManagerTestBase, TestFormatWithSpecificUUID) {
+ string path = GetTestPath("new_fs_root");
+ ReinitFsManager(path, {});
+
+ // Use an invalid uuid at first.
+ string uuid = "not_a_valid_uuid";
+ Status s = fs_manager()->CreateInitialFileSystemLayout(uuid);
+ ASSERT_TRUE(s.IsInvalidArgument());
+ ASSERT_STR_CONTAINS(s.ToString(), Substitute("invalid uuid $0", uuid));
+
+ // Now use a valid one.
+ ObjectIdGenerator oid_generator;
+ uuid = oid_generator.Next();
+ ASSERT_OK(fs_manager()->CreateInitialFileSystemLayout(uuid));
+ ASSERT_OK(fs_manager()->Open());
+ ASSERT_EQ(uuid, fs_manager()->uuid());
+}
+
} // namespace kudu
http://git-wip-us.apache.org/repos/asf/kudu/blob/1bce7312/src/kudu/fs/fs_manager.cc
----------------------------------------------------------------------
diff --git a/src/kudu/fs/fs_manager.cc b/src/kudu/fs/fs_manager.cc
index 117389f..86e5d4f 100644
--- a/src/kudu/fs/fs_manager.cc
+++ b/src/kudu/fs/fs_manager.cc
@@ -22,6 +22,7 @@
#include <map>
#include <unordered_set>
+#include <boost/optional.hpp>
#include <glog/logging.h>
#include <glog/stl_logging.h>
#include <google/protobuf/message.h>
@@ -244,7 +245,7 @@ Status FsManager::Open() {
return Status::OK();
}
-Status FsManager::CreateInitialFileSystemLayout() {
+Status FsManager::CreateInitialFileSystemLayout(boost::optional<string> uuid) {
CHECK(!read_only_);
RETURN_NOT_OK(Init());
@@ -271,7 +272,7 @@ Status FsManager::CreateInitialFileSystemLayout() {
ElementDeleter d(&delete_on_failure);
InstanceMetadataPB metadata;
- CreateInstanceMetadata(&metadata);
+ RETURN_NOT_OK(CreateInstanceMetadata(std::move(uuid), &metadata));
unordered_set<string> to_sync;
for (const string& root : canonicalized_all_fs_roots_) {
bool created;
@@ -319,9 +320,16 @@ Status FsManager::CreateInitialFileSystemLayout() {
return Status::OK();
}
-void FsManager::CreateInstanceMetadata(InstanceMetadataPB* metadata) {
+Status FsManager::CreateInstanceMetadata(boost::optional<string> uuid,
+ InstanceMetadataPB* metadata) {
ObjectIdGenerator oid_generator;
- metadata->set_uuid(oid_generator.Next());
+ if (uuid) {
+ string canonicalized_uuid;
+ RETURN_NOT_OK(oid_generator.Canonicalize(uuid.get(), &canonicalized_uuid));
+ metadata->set_uuid(canonicalized_uuid);
+ } else {
+ metadata->set_uuid(oid_generator.Next());
+ }
string time_str;
StringAppendStrftime(&time_str, "%Y-%m-%d %H:%M:%S", time(nullptr), false);
@@ -330,6 +338,7 @@ void FsManager::CreateInstanceMetadata(InstanceMetadataPB* metadata) {
hostname = "<unknown host>";
}
metadata->set_format_stamp(Substitute("Formatted at $0 on $1", time_str, hostname));
+ return Status::OK();
}
Status FsManager::WriteInstanceMetadata(const InstanceMetadataPB& metadata,
http://git-wip-us.apache.org/repos/asf/kudu/blob/1bce7312/src/kudu/fs/fs_manager.h
----------------------------------------------------------------------
diff --git a/src/kudu/fs/fs_manager.h b/src/kudu/fs/fs_manager.h
index fab0436..3c4420a 100644
--- a/src/kudu/fs/fs_manager.h
+++ b/src/kudu/fs/fs_manager.h
@@ -18,6 +18,8 @@
#ifndef KUDU_FS_FS_MANAGER_H
#define KUDU_FS_FS_MANAGER_H
+#include <boost/none.hpp>
+#include <boost/optional/optional.hpp>
#include <gtest/gtest_prod.h>
#include <iosfwd>
#include <memory>
@@ -107,10 +109,12 @@ class FsManager {
// the on-disk structures.
Status Open();
- // Create the initial filesystem layout.
+ // Create the initial filesystem layout. If 'uuid' is provided, uses it as
+ // uuid of the filesystem. Otherwise generates one at random.
//
// Returns an error if the file system is already initialized.
- Status CreateInitialFileSystemLayout();
+ Status CreateInitialFileSystemLayout(
+ boost::optional<std::string> uuid = boost::none);
void DumpFileSystemTree(std::ostream& out);
@@ -212,7 +216,8 @@ class FsManager {
void InitBlockManager();
// Create a new InstanceMetadataPB.
- void CreateInstanceMetadata(InstanceMetadataPB* metadata);
+ Status CreateInstanceMetadata(boost::optional<std::string> uuid,
+ InstanceMetadataPB* metadata);
// Save a InstanceMetadataPB to the filesystem.
// Does not mutate the current state of the fsmanager.
http://git-wip-us.apache.org/repos/asf/kudu/blob/1bce7312/src/kudu/util/CMakeLists.txt
----------------------------------------------------------------------
diff --git a/src/kudu/util/CMakeLists.txt b/src/kudu/util/CMakeLists.txt
index 4a000b6..a4527b9 100644
--- a/src/kudu/util/CMakeLists.txt
+++ b/src/kudu/util/CMakeLists.txt
@@ -309,6 +309,7 @@ ADD_KUDU_TEST(mt-threadlocal-test RUN_SERIAL true)
ADD_KUDU_TEST(net/dns_resolver-test)
ADD_KUDU_TEST(net/net_util-test)
ADD_KUDU_TEST(object_pool-test)
+ADD_KUDU_TEST(oid_generator-test)
ADD_KUDU_TEST(once-test)
ADD_KUDU_TEST(os-util-test)
ADD_KUDU_TEST(path_util-test)
http://git-wip-us.apache.org/repos/asf/kudu/blob/1bce7312/src/kudu/util/oid_generator-test.cc
----------------------------------------------------------------------
diff --git a/src/kudu/util/oid_generator-test.cc b/src/kudu/util/oid_generator-test.cc
new file mode 100644
index 0000000..a38b496
--- /dev/null
+++ b/src/kudu/util/oid_generator-test.cc
@@ -0,0 +1,50 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "kudu/util/oid_generator.h"
+
+#include <gtest/gtest.h>
+#include <string>
+
+#include "kudu/util/test_util.h"
+
+using std::string;
+
+namespace kudu {
+
+TEST(ObjectIdGeneratorTest, TestCanoicalizeUuid) {
+ ObjectIdGenerator gen;
+ const string kExpectedCanonicalized = "0123456789abcdef0123456789abcdef";
+ string canonicalized;
+ Status s = gen.Canonicalize("not_a_uuid", &canonicalized);
+ {
+ SCOPED_TRACE(s.ToString());
+ ASSERT_TRUE(s.IsInvalidArgument());
+ ASSERT_STR_CONTAINS(s.ToString(), "invalid uuid");
+ }
+ ASSERT_OK(gen.Canonicalize(
+ "01234567-89ab-cdef-0123-456789abcdef", &canonicalized));
+ ASSERT_EQ(kExpectedCanonicalized, canonicalized);
+ ASSERT_OK(gen.Canonicalize(
+ "0123456789abcdef0123456789abcdef", &canonicalized));
+ ASSERT_EQ(kExpectedCanonicalized, canonicalized);
+ ASSERT_OK(gen.Canonicalize(
+ "0123456789AbCdEf0123456789aBcDeF", &canonicalized));
+ ASSERT_EQ(kExpectedCanonicalized, canonicalized);
+}
+
+} // namespace kudu
http://git-wip-us.apache.org/repos/asf/kudu/blob/1bce7312/src/kudu/util/oid_generator.cc
----------------------------------------------------------------------
diff --git a/src/kudu/util/oid_generator.cc b/src/kudu/util/oid_generator.cc
index da7acd8..580463c 100644
--- a/src/kudu/util/oid_generator.cc
+++ b/src/kudu/util/oid_generator.cc
@@ -15,21 +15,48 @@
// specific language governing permissions and limitations
// under the License.
+#include "kudu/util/oid_generator.h"
+
+#include <boost/uuid/uuid_generators.hpp>
+#include <exception>
#include <mutex>
#include <string>
#include "kudu/gutil/stringprintf.h"
-#include "kudu/util/oid_generator.h"
+#include "kudu/gutil/strings/substitute.h"
+#include "kudu/util/status.h"
+
+using strings::Substitute;
namespace kudu {
-string ObjectIdGenerator::Next() {
- std::lock_guard<LockType> l(oid_lock_);
- boost::uuids::uuid oid = oid_generator_();
- const uint8_t *uuid = oid.data;
+namespace {
+
+string ConvertUuidToString(const boost::uuids::uuid& to_convert) {
+ const uint8_t* uuid = to_convert.data;
return StringPrintf("%02x%02x%02x%02x%02x%02x%02x%02x%02x%02x%02x%02x%02x%02x%02x%02x",
uuid[0], uuid[1], uuid[2], uuid[3], uuid[4], uuid[5], uuid[6], uuid[7],
uuid[8], uuid[9], uuid[10], uuid[11], uuid[12], uuid[13], uuid[14], uuid[15]);
}
+} // anonymous namespace
+
+string ObjectIdGenerator::Next() {
+ std::lock_guard<LockType> l(oid_lock_);
+ boost::uuids::uuid uuid = oid_generator_();
+ return ConvertUuidToString(uuid);
+}
+
+Status ObjectIdGenerator::Canonicalize(const string& input,
+ string* output) const {
+ try {
+ boost::uuids::uuid uuid = oid_validator_(input);
+ *output = ConvertUuidToString(uuid);
+ return Status::OK();
+ } catch (std::exception& e) {
+ return Status::InvalidArgument(Substitute("invalid uuid $0: $1",
+ input, e.what()));
+ }
+}
+
} // namespace kudu
http://git-wip-us.apache.org/repos/asf/kudu/blob/1bce7312/src/kudu/util/oid_generator.h
----------------------------------------------------------------------
diff --git a/src/kudu/util/oid_generator.h b/src/kudu/util/oid_generator.h
index 85a7412..7acccc9 100644
--- a/src/kudu/util/oid_generator.h
+++ b/src/kudu/util/oid_generator.h
@@ -23,6 +23,7 @@
#include "kudu/gutil/macros.h"
#include "kudu/util/locks.h"
+#include "kudu/util/status.h"
namespace kudu {
@@ -33,15 +34,26 @@ class ObjectIdGenerator {
ObjectIdGenerator() {}
~ObjectIdGenerator() {}
+ // Generates and returns a new UUID.
std::string Next();
+ // Validates an existing UUID and converts it into the format used by Kudu
+ // (that is, 16 hexadecimal bytes without any dashes).
+ Status Canonicalize(const std::string& input, std::string* output) const;
+
private:
DISALLOW_COPY_AND_ASSIGN(ObjectIdGenerator);
typedef simple_spinlock LockType;
+ // Protects 'oid_generator_'.
LockType oid_lock_;
+
+ // Generates new UUIDs.
boost::uuids::random_generator oid_generator_;
+
+ // Validates provided UUIDs.
+ boost::uuids::string_generator oid_validator_;
};
} // namespace kudu