[kudu] branch master updated: [docs] add notes on running chronyd in local reference mode

[al...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

alexey pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kudu.git


The following commit(s) were added to refs/heads/master by this push:
     new 0fc1d8b  [docs] add notes on running chronyd in local reference mode
0fc1d8b is described below

commit 0fc1d8be8ba95a4e3b10e2479d3d73f0b2e10fd7
Author: Alexey Serbin <al...@apache.org>
AuthorDate: Tue Mar 22 21:59:50 2022 -0700

    [docs] add notes on running chronyd in local reference mode
    
    When running a Kudu cluster in an isolated network, use a dedicated
    server to run the reference chronyd in local reference mode [1].
    Avoid placing kudu-master and kudu-tserver on such a server.
    
    [1] https://chrony.tuxfamily.org/doc/3.4/chrony.conf.html#local
    
    Change-Id: Ie346ed02f1faa52bce1c919cf89584d2cb398c7b
    Reviewed-on: http://gerrit.cloudera.org:8080/18344
    Tested-by: Kudu Jenkins
    Reviewed-by: Andrew Wong <aw...@cloudera.com>
---
 docs/troubleshooting.adoc | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/docs/troubleshooting.adoc b/docs/troubleshooting.adoc
index df42be7..9e2ae61 100644
--- a/docs/troubleshooting.adoc
+++ b/docs/troubleshooting.adoc
@@ -515,6 +515,36 @@ For troubleshooting tips on clock synchronisation with chronyd see
 link:https://chrony.tuxfamily.org/faq.html#_computer_is_not_synchronising[this
 useful guide].
 
+===== Notes on running `chronyd` in local reference mode
+When running Kudu cluster in an isolated network, it's possible to synchronize
+the clocks on all nodes with one another, but not necessarily to true time.
+The local clock of one dedicated server can be kept vaguely in line with true
+time by manual input, and other nodes will synchronize with that one. The
+`chronyd` at the dedicated server runs in
+link:https://chrony.tuxfamily.org/doc/3.4/chrony.conf.html#local[local reference mode],
+and `chronyd` at other nodes synchronize with it as with a regular true time
+reference.
+
+If using such a setup, it's not recommended to run any Kudu servers at the
+node where `chronyd` is running with local reference mode (at least for
+`chronyd` of version 3.4 and below). Otherwise, the following might happen:
+
+ - `chronyc tracking` reports that all is well, `Leap Status` is `Normal`
+ - `ntptime` reports that the local clock is out of sync
+ - `ntp_gettime() returns code 5 (ERROR)`
+ - `ntp_adjtime() returns code 5 (ERROR)` with `status 0x40 (UNSYNC)`
+
+With that, `kudu-master` and `kudu-tserver` fail to start, reporting an
+errors like below:
+
+----
+E0321 09:47:04.076095  2558 master_main.cc:40] Service unavailable: RunMasterServer() failed: Cannot initialize clock: timed out waiting for clock synchronisation: Error reading clock. Clock considered unsynchronized
+----
+
+----
+E0321 09:50:03.023491  2853 tablet_server_main.cc:40] Service unavailable: RunTabletServer() failed: Cannot initialize clock: timed out waiting for clock synchronisation: Error reading clock. Clock considered unsynchronized
+----
+
 ==== NTP Configuration Best Practices
 In order to provide stable time synchronization with low maximum error, follow
 these best NTP configuration best practices.