You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kudu.apache.org by mp...@apache.org on 2019/01/17 19:06:51 UTC
[kudu] branch master updated (389f997 -> 3277fe3)
This is an automated email from the ASF dual-hosted git repository.
mpercy pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git.
from 389f997 Do not fallthrough to next case upon parsing "\n" in JSON
new 6b9719c [docs] Add docs for rack/location-awareness
new 3450c78 client: avoid accessing state after scheduling retry
new 3277fe3 docs: Add Gerrit HTTP endpoint instructions
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
docs/administration.adoc | 100 ++++++++++++++++++++++++++++++++++++
docs/contributing.adoc | 45 +++++++++++++---
src/kudu/client/master_proxy_rpc.cc | 12 +++--
3 files changed, 145 insertions(+), 12 deletions(-)
[kudu] 01/03: [docs] Add docs for rack/location-awareness
Posted by mp...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
mpercy pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit 6b9719cb48b4c8b932fadd0f37299179b03b9f32
Author: Will Berkeley <wd...@gmail.org>
AuthorDate: Fri Jan 11 12:01:06 2019 -0800
[docs] Add docs for rack/location-awareness
I stuck to the name "rack-awareness" throughout, preferring it to the
term we have used internally, "location-awareness", because even though
the latter term is more technically correct, the former is the more
typical and colloquial name for this sort of thing.
These docs borrow heavily from the WIP blog post written by Alexey.
Change-Id: I505b7fd545a4253170a5b7cac63c33e7d9669490
Reviewed-on: http://gerrit.cloudera.org:8080/12219
Tested-by: Kudu Jenkins
Reviewed-by: Andrew Wong <aw...@cloudera.com>
---
docs/administration.adoc | 100 +++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 100 insertions(+)
diff --git a/docs/administration.adoc b/docs/administration.adoc
index 40690b2..ce3afad 100644
--- a/docs/administration.adoc
+++ b/docs/administration.adoc
@@ -202,6 +202,82 @@ format exposed by the HTTP endpoint above.
The frequency with which metrics are dumped to the diagnostics log is configured using the
`--metrics_log_interval_ms` flag. By default, Kudu logs metrics every 60 seconds.
+[[rack_awareness]]
+== Rack Awareness
+
+As of version 1.9, Kudu supports a rack awareness feature. Kudu's ordinary
+re-replication methods ensure the availability of the cluster in the event of a
+single node failure. However, clusters can be vulnerable to correlated failures
+of multiple nodes. For example, all of the physical hosts on the same rack in
+a datacenter may become unavailable simultaneously if the top-of-rack switch
+fails. Kudu's rack awareness feature provides protection from some kinds of
+correlated failures, like the failure of a single rack in a datacenter.
+
+The first element of Kudu's rack awareness feature is location assignment. When
+a tablet server or client registers with a master, the master assigns it a
+location. A location is a `/`-separated string that begins with a `/` and where
+each `/`-separated component consists of characters from the set
+`[a-zA-Z0-9_-.]`. For example, `/dc-0/rack-09` is a valid location, while
+`rack-04` and `/rack=1` are not valid locations. Thus location strings resemble
+absolute UNIX file paths where characters in directory and file names are
+restricted to the set `[a-zA-Z0-9_-.]`. Presently, Kudu does not use the
+hierarchical structure of locations, but it may in the future. Location
+assignment is done by a user-provided command, whose path should be specified
+using the `--location_mapping_cmd` master flag. The command should take a single
+argument, the IP address or hostname of a tablet server or client, and return
+the location for the tablet server or client. Make sure that all Kudu masters
+are using the same location mapping command.
+
+The second element of Kudu's rack awareness feature is the placement policy,
+which is
+
+ Do not place a majority of replicas of a tablet on tablet servers in the same location.
+
+The leader master, when placing newly created replicas on tablet servers and
+when re-replicating existing tablets, will attempt to place the replicas in a
+way that complies with the placement policy. For example, in a cluster with five
+tablet servers `A`, `B`, `C`, `D`, and `E`, with respective locations `/L0`,
+`/L0`, `/L1`, `/L1`, `/L2`, to comply with the placement policy a new 3x
+replicated tablet could have its replicas placed on `A`, `C`, and `E`, but not
+on `A`, `B`, and `C`, because then the tablet would have 2/3 replicas in
+location `/L0`. As another example, if a tablet has replicas on tablet servers
+`A`, `C`, and `E`, and then `C` fails, the replacement replica must be placed on
+`D` in order to comply with the placement policy.
+
+In the case where it is impossible to place replicas in a way that complies with
+the placement policy, Kudu will violate the policy and place a replica anyway.
+For example, using the setup described in the previous paragraph, if a tablet
+has replicas on tablet servers `A`, `C`, and `E`, and then `E` fails, Kudu will
+re-replicate the tablet onto one of `B` or `D`, violating the placement policy,
+rather than leaving the tablet under-replicated indefinitely. The
+`kudu cluster rebalance` tool can reestablish the placement policy if it is
+possible to do so. The `kudu cluster rebalance` tool can also be used to
+establish the placement policy on a cluster if the cluster has just been
+configured to use the rack awareness feature and existing replicas need to be
+moved to comply with the placement policy. See
+<<rebalancer_tool_with_rack_awareness,running the tablet rebalancing tool on a rack-aware cluster>>
+for more information.
+
+The third and final element of Kudu's rack awareness feature is the use of
+client locations to find "nearby" servers. As mentioned, the masters also
+assign a location to clients when they connect to the cluster. The client
+(whether Java, {cpp}, or Python) uses its own location and the locations of
+tablet servers in the cluster to prefer "nearby" replicas when scanning in
+`CLOSEST_REPLICA` mode. Clients choose replicas to scan in the following order:
+
+. Scan a replica on a tablet server on the same host, if there is one.
+. Scan a replica on a tablet server in the same location, if there is one.
+. Scan some replica.
+
+For example, using the cluster setup described above, if a client on the same
+host as tablet server `A` scans a tablet with replicas on tablet servers
+`A`, `C`, and `E` in `CLOSEST_REPLICA` mode, it will choose to scan from the
+replica on `A`, since the client and the replica on `A` are on the same host.
+If the client scans a tablet with replicas on tablet servers `B`, `C`, and `E`,
+it will choose to scan from the replica on `B`, since it is in the same
+location as the client, `/L0`. If there are multiple replicas meeting a
+criterion, one is chosen arbitrarily.
+
== Common Kudu workflows
[[migrate_to_multi_master]]
@@ -1261,6 +1337,30 @@ If the rebalancer is running against a cluster where rebalancing replication
factor one tables is not supported, it will rebalance all the other tables
and the cluster as if those singly-replicated tables did not exist.
+[[rebalancer_tool_with_rack_awareness]]
+=== Running the tablet rebalancing tool on a rack-aware cluster
+
+As detailed in the <<rack_awareness, rack awareness>> section, it's possible
+to use the `kudu cluster rebalance` tool to establish the placement policy on a
+cluster. This might be necessary when the rack awareness feature is first
+configured or when re-replication violated the placement policy. The rebalancing
+tool breaks its work into three phases:
+
+. The rack-aware rebalancer tries to establish the placement policy. Use the
+ `--disable_policy_fixer` flag to skip this phase.
+. The rebalancer tries to balance load by location, moving tablet replicas
+ between locations in an attempt to spread tablet replicas among locations
+ evenly. The load of a location is measured as the total number of replicas in
+ the location divided by the number of tablet servers in the location. Use the
+ `--disable_cross_location_rebalancing` flag to skip this phase.
+. The rebalancer tries to balance the tablet replica distribution within each
+ location, as if the location were a cluster on its own. Use the
+ `--disable_intra_location_rebalancing` flag to skip this phase.
+
+By using the `--report_only` flag, it's also possible to check if all tablets in
+the cluster conform to the placement policy without attempting any replica
+movement.
+
[[tablet_server_decommissioning]]
=== Decommissioning or Permanently Removing a Tablet Server From a Cluster
[kudu] 03/03: docs: Add Gerrit HTTP endpoint instructions
Posted by mp...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
mpercy pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit 3277fe3f9990a10faae08a218db7ec6d6577a8dc
Author: Mike Percy <mp...@apache.org>
AuthorDate: Fri Jan 11 08:44:40 2019 -0800
docs: Add Gerrit HTTP endpoint instructions
Some users cannot use SSH to connect to Gerrit. We should provide
explicit instructions for these users on how to submit a patch via
Gerrit so they don't have to figure it out themselves from the Gerrit
docs.
Change-Id: I70eb2abf6d4c11500d50be27e398bdf0cb3f85c2
Reviewed-on: http://gerrit.cloudera.org:8080/12218
Tested-by: Kudu Jenkins
Reviewed-by: Adar Dembo <ad...@cloudera.com>
---
docs/contributing.adoc | 45 +++++++++++++++++++++++++++++++++++++--------
1 file changed, 37 insertions(+), 8 deletions(-)
diff --git a/docs/contributing.adoc b/docs/contributing.adoc
index 68aece1..75d13fb 100644
--- a/docs/contributing.adoc
+++ b/docs/contributing.adoc
@@ -38,11 +38,16 @@ for an overview of using Gerrit for code review.
=== Initial Setup for Gerrit
-. Sign in to link:http://gerrit.cloudera.org:8080[Gerrit] using your Github username.
+. Sign in to link:https://gerrit.cloudera.org/[Gerrit] using your Github username.
-. Go to link:http://gerrit.cloudera.org:8080/#/settings/[Settings]. Update your name
-and email address on the *Contact Information* page, and upload a SSH public key.
-If you do not update your name, it will show up as "Anonymous Coward" in Gerrit reviews.
+. Go to link:https://gerrit.cloudera.org/settings/[Settings]. Update your name
+and email address on the *Contact Information* page, and upload a SSH public
+key under *SSH Public Keys* if you would like to use SSH to connect to Gerrit.
+Generate an HTTP password under *HTTP Password* if you would like to use HTTP
+or HTTPS to connect to Gerrit. (Most Kudu developers use the SSH option.)
++
+NOTE: If you do not update your name, it will appear as "Anonymous Coward" in
+Gerrit reviews.
. If you have not done so, clone the main Kudu repository. By default, the main remote
is called `origin`. When you fetch or pull, you will do so from `origin`.
@@ -54,16 +59,40 @@ git clone https://github.com/apache/kudu
. Change to the new `kudu` directory.
-. Add a `gerrit` remote. In the following command, substitute <username> with your
-Github username.
+. Add a `gerrit` remote.
++
+If using SSH to connect to Gerrit, use the following command to add the Gerrit
+remote (substitute <username> with your Github username):
+
[source,bash]
----
git remote add gerrit ssh://<username>@gerrit.cloudera.org:29418/kudu
----
++
+If using HTTP or HTTPS to connect to Gerrit, use the following command to add
+the Gerrit remote (http:// also works):
++
+[source,bash]
+----
+git remote add gerrit https://gerrit.cloudera.org/a/kudu
+----
++
+[TIP]
+====
+If you are using Gerrit's HTTP or HTTPS endpoints and prefer not to type a
+username and password each time you submit a patch, you can put your login and
+password into a `.netrc` file located at `$HOME/.netrc` and Git will use it.
+The password is stored as plaintext and the file format is as follows:
+
+[source]
+----
+machine <hostname>
+ login <username>
+ password <password>
+----
+====
-. Run the following command to install the
-Gerrit `commit-msg` hook. Use the following command:
+. Run the following command to install the Gerrit `commit-msg` hook:
+
----
cd kudu
[kudu] 02/03: client: avoid accessing state after scheduling retry
Posted by mp...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
mpercy pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit 3450c78d2990dfc65c38e47661901fc68fdf8ced
Author: Andrew Wong <aw...@cloudera.com>
AuthorDate: Sat Jan 5 22:22:19 2019 -0800
client: avoid accessing state after scheduling retry
I saw a failure of MasterMigrationTest.TestEndToEndMigration in which
the stack watchdog logged the following stacktrace:
User stack:
@ 0x7fa3e575c330 (unknown) at ??:0
@ 0x52dc09 __sanitizer::internal_read() at /data/8/awong/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/sanitizer_common/sanitizer_syscall_linux_x86_64.inc:46 (discriminator 7)
@ 0x52f79f __sanitizer::ReadFromFile() at /data/8/awong/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/sanitizer_common/sanitizer_posix.cc:176
@ 0x53acb9 __sanitizer::SymbolizerProcess::ReadFromSymbolizer() at /data/8/awong/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_libcdep.cc:468
@ 0x53ba76 __sanitizer::SymbolizerProcess::SendCommand() at /data/8/awong/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_libcdep.cc:445
@ 0x53c3c5 __sanitizer::Symbolizer::SymbolizePC() at /data/8/awong/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_libcdep.cc:356
@ 0x539673 __sanitizer::StackTrace::Print() at /data/8/awong/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/sanitizer_common/sanitizer_stacktrace_libcdep.cc:36
@ 0x541c44 MaybePrintStackTrace() at /data/8/awong/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/ubsan/ubsan_diag.cc:48
@ 0x5422ee __ubsan::ScopedReport::~ScopedReport() at /data/8/awong/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/ubsan/ubsan_diag.cc:73
@ 0x549608 HandleDynamicTypeCacheMiss() at /data/8/awong/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/ubsan/ubsan_handlers_cxx.cc:81
@ 0x549a43 __ubsan_handle_dynamic_type_cache_miss_abort at /data/8/awong/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/ubsan/ubsan_handlers_cxx.cc:93
@ 0x7fa3f0086643 _ZZN4kudu6client8internal20AsyncLeaderMasterRpcINS_6master23GetTableSchemaRequestPBENS3_24GetTableSchemaResponsePBEE27RetryOrReconnectIfNecessaryEPNS_6StatusEENKUlvE_clEv at ??:0
@ 0x7fa3f0059b8c _ZN4kudu13ScopedCleanupIZNS_6client8internal20AsyncLeaderMasterRpcINS_6master23GetTableSchemaRequestPBENS4_24GetTableSchemaResponsePBEE27RetryOrReconnectIfNecessaryEPNS_6StatusEEUlvE_ED2Ev at ??:0
@ 0x7fa3f005945c kudu::client::internal::AsyncLeaderMasterRpc<>::RetryOrReconnectIfNecessary() at ??:0
@ 0x7fa3f0057d0a kudu::client::internal::AsyncLeaderMasterRpc<>::SendRpcCb() at ??:0
@ 0x7fa3f0085a85 _ZNSt5_BindIFSt7_Mem_fnIMN4kudu6client8internal20AsyncLeaderMasterRpcINS1_6master23GetTableSchemaRequestPBENS5_24GetTableSchemaResponsePBEEEFvRKNS1_6StatusEEEPS8_S9_EE6__callIvJEJLm0ELm1EEEET_OSt5tupleIJDpT0_EESt12_Index_tupleIJXspT1_EEE at ??:0
#19 0x7fa3ddc84ffc in clone sysdeps/unix/sysv/linux/x86_64/clone.S:111
Upon looking into this and similar failures, it seems like the following
series of events is possible:
T1: allocate an AsyncLeaderMasterRpc on the stack with
Synchronizer.Wait() as the user-specified callback
T2: the RPC call results in an error that makes it retry via
RetryOrReconnectIfNecessary(); a retry is scheduled
T3: the retry completes and calls the user-specified callback to begin
execution on T1
T1: with the RPC call completed, destroys RPC object
T2: the KLOG messages attempt to access state from the destroyed RPC
object and hit a crash, race, undefined behavior, etc.
The fix is to make a copy of the RPC's state that was to be logged so
there's no chance that T1's destruction of the RPC will affect T2. I
tested this by looping a test[1] that repeatedly called
Client::IsCreateTableInProgress() on a multi-master cluster configured
to have election storms, and observing 5/1000 that yielded TSAN errors
around the logging calls in RetryOrReconnectIfNecessary. With this
patch, 1000/1000 pass.
[1] https://gist.github.com/andrwng/5d552f75a2e0d671b7ed54dd01892c66
Change-Id: I8cefd9613018247a1a25d17adedc021e8be166f6
Reviewed-on: http://gerrit.cloudera.org:8080/12170
Tested-by: Kudu Jenkins
Reviewed-by: Adar Dembo <ad...@cloudera.com>
Reviewed-by: Alexey Serbin <as...@cloudera.com>
---
src/kudu/client/master_proxy_rpc.cc | 12 ++++++++----
1 file changed, 8 insertions(+), 4 deletions(-)
diff --git a/src/kudu/client/master_proxy_rpc.cc b/src/kudu/client/master_proxy_rpc.cc
index 393a681..326df12 100644
--- a/src/kudu/client/master_proxy_rpc.cc
+++ b/src/kudu/client/master_proxy_rpc.cc
@@ -182,10 +182,14 @@ void AsyncLeaderMasterRpc<ReqClass, RespClass>::ResetMasterLeaderAndRetry(
template <class ReqClass, class RespClass>
bool AsyncLeaderMasterRpc<ReqClass, RespClass>::RetryOrReconnectIfNecessary(
Status* status) {
- auto warn_on_retry = MakeScopedCleanup([&] {
- KLOG_EVERY_N_SECS(WARNING, 1) <<
- Substitute("Re-attempting $0 request to leader Master ($1)",
- rpc_name_, client_->data_->leader_master_hostport().ToString());
+ const string retry_warning = Substitute("Re-attempting $0 request to leader Master ($1)",
+ rpc_name_, client_->data_->leader_master_hostport().ToString());
+ auto warn_on_retry = MakeScopedCleanup([&retry_warning] {
+ // NOTE: we pass in a ref to 'retry_warning' rather than evaluating it here
+ // because any of the below retries may end up completing and calling
+ // 'user_cb_', which may very well destroy this Rpc object and render
+ // 'rpc_name_' and 'client_' inaccessible.
+ KLOG_EVERY_N_SECS(WARNING, 1) << retry_warning;
});
// Pull out the RPC status.
const bool is_multi_master = client_->IsMultiMaster();