You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "helifu (Code Review)" <ge...@cloudera.org> on 2019/07/09 06:50:32 UTC
[kudu-CR] [docs] update the upgrade documentation
helifu has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13820
Change subject: [docs] update the upgrade documentation
......................................................................
[docs] update the upgrade documentation
The process of upgrading the cluster has been added to
the installation.adoc.
Change-Id: I6b3e5c549dc05c3388c0b0dd628d205a356da344
---
M docs/installation.adoc
1 file changed, 16 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/20/13820/1
--
To view, visit http://gerrit.cloudera.org:8080/13820
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I6b3e5c549dc05c3388c0b0dd628d205a356da344
Gerrit-Change-Number: 13820
Gerrit-PatchSet: 1
Gerrit-Owner: helifu <hz...@corp.netease.com>
[kudu-CR] [docs] update the upgrade documentation
Posted by "helifu (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins, Adar Dembo, Priyanka Chheda,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/13820
to look at the new patch set (#2).
Change subject: [docs] update the upgrade documentation
......................................................................
[docs] update the upgrade documentation
The process of upgrading the cluster has been added to
the installation.adoc.
Change-Id: I6b3e5c549dc05c3388c0b0dd628d205a356da344
---
M docs/installation.adoc
1 file changed, 26 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/20/13820/2
--
To view, visit http://gerrit.cloudera.org:8080/13820
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6b3e5c549dc05c3388c0b0dd628d205a356da344
Gerrit-Change-Number: 13820
Gerrit-PatchSet: 2
Gerrit-Owner: helifu <hz...@corp.netease.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Priyanka Chheda <pc...@cloudera.com>
Gerrit-Reviewer: helifu <hz...@corp.netease.com>
[kudu-CR] [docs] update the upgrade documentation
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13820 )
Change subject: [docs] update the upgrade documentation
......................................................................
Patch Set 6: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/13820
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6b3e5c549dc05c3388c0b0dd628d205a356da344
Gerrit-Change-Number: 13820
Gerrit-PatchSet: 6
Gerrit-Owner: helifu <hz...@corp.netease.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Priyanka Chheda <pc...@cloudera.com>
Gerrit-Reviewer: helifu <hz...@corp.netease.com>
Gerrit-Comment-Date: Tue, 16 Jul 2019 08:00:00 +0000
Gerrit-HasComments: No
[kudu-CR] [docs] update the upgrade documentation
Posted by "helifu (Code Review)" <ge...@cloudera.org>.
helifu has posted comments on this change. ( http://gerrit.cloudera.org:8080/13820 )
Change subject: [docs] update the upgrade documentation
......................................................................
Patch Set 2:
(9 comments)
http://gerrit.cloudera.org:8080/#/c/13820/2/docs/installation.adoc
File docs/installation.adoc:
http://gerrit.cloudera.org:8080/#/c/13820/2/docs/installation.adoc@635
PS2, Line 635: for the scenario of compiling from source code.
> "relevant when building from source code".
Done
http://gerrit.cloudera.org:8080/#/c/13820/2/docs/installation.adoc@640
PS2, Line 640: 2hours
> "2 hours"
Done
http://gerrit.cloudera.org:8080/#/c/13820/2/docs/installation.adoc@640
PS2, Line 640: kudu-tserver
> "tablet server"
Done
http://gerrit.cloudera.org:8080/#/c/13820/2/docs/installation.adoc@645
PS2, Line 645: --force
> Shouldn't need this; the flag is tagged as 'runtime'.
Ah, Grant has tagged the follower_unavailable_considered_failed_sec flag as runtime.
http://gerrit.cloudera.org:8080/#/c/13820/2/docs/installation.adoc@647
PS2, Line 647: kudu-tserver
> "the tablet servers"
Done
http://gerrit.cloudera.org:8080/#/c/13820/2/docs/installation.adoc@647
PS2, Line 647: and the gflag above should be reset after every reboot.
> Since step 4 restores the gflag value, do we really need this step?
The 'reset' here means setting to 7200.
========================================
Let me describe it more clearly. If there are 3 tablet servers A, B, C:
1.Set the unavailable time for every kudu-tserver to a large value:
./kudu tserver set_flag A follower_unavailable_considered_failed_sec 7200
./kudu tserver set_flag B follower_unavailable_considered_failed_sec 7200
./kudu tserver set_flag C follower_unavailable_considered_failed_sec 7200
2.Rolling restart kudu-tserver and the gflag above should be reset after every reboot:
restart A
./kudu tserver set_flag A follower_unavailable_considered_failed_sec 7200
restart B
./kudu tserver set_flag B follower_unavailable_considered_failed_sec 7200
restart C
./kudu tserver set_flag C follower_unavailable_considered_failed_sec 7200
3.Restore the default gflag value (5 minutes) for every kudu-tserver:
./kudu tserver set_flag A follower_unavailable_considered_failed_sec 300
./kudu tserver set_flag B follower_unavailable_considered_failed_sec 300
./kudu tserver set_flag C follower_unavailable_considered_failed_sec 300
http://gerrit.cloudera.org:8080/#/c/13820/2/docs/installation.adoc@649
PS2, Line 649: [NOTE]
> Shouldn't this be just after "Set the unavailable time..." so users underst
The note is used to emphasize the 'reset' is very important, not the first time set.
http://gerrit.cloudera.org:8080/#/c/13820/2/docs/installation.adoc@651
PS2, Line 651: If the gflag is not reset, kudu-tserver which is restarting would be possibly evicted from the cluster.
> "If the unavailable time is not extended, restarted tablet servers could be
Done
http://gerrit.cloudera.org:8080/#/c/13820/2/docs/installation.adoc@654
PS2, Line 654: kudu-master
> "the masters"
Done
--
To view, visit http://gerrit.cloudera.org:8080/13820
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6b3e5c549dc05c3388c0b0dd628d205a356da344
Gerrit-Change-Number: 13820
Gerrit-PatchSet: 2
Gerrit-Owner: helifu <hz...@corp.netease.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Priyanka Chheda <pc...@cloudera.com>
Gerrit-Reviewer: helifu <hz...@corp.netease.com>
Gerrit-Comment-Date: Wed, 10 Jul 2019 22:48:20 +0000
Gerrit-HasComments: Yes
[kudu-CR] [docs] update the upgrade documentation
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/13820 )
Change subject: [docs] update the upgrade documentation
......................................................................
Patch Set 1:
(7 comments)
Could you push this change to a github fork so we can also review the rendered form of the documentation?
http://gerrit.cloudera.org:8080/#/c/13820/1/docs/installation.adoc
File docs/installation.adoc:
http://gerrit.cloudera.org:8080/#/c/13820/1/docs/installation.adoc@635
PS1, Line 635: . Prepare the software.
This recommendation is only necessary if you're actually building from source. If not, it doesn't make sense. Could you condition it behind that?
http://gerrit.cloudera.org:8080/#/c/13820/1/docs/installation.adoc@636
PS1, Line 636: building
build
http://gerrit.cloudera.org:8080/#/c/13820/1/docs/installation.adoc@638
PS1, Line 638: big
large
http://gerrit.cloudera.org:8080/#/c/13820/1/docs/installation.adoc@645
PS1, Line 645: - Rolling restart `kudu-tserver` and the gflag above should be reset after every rebooting.
Could you provide specific instructions for restoring the default gflag value?
Also, when referring to kudu-master and kudu-tserver _processes_ (as opposed to binary names), no need to surround with backticks. Same with tserver below.
http://gerrit.cloudera.org:8080/#/c/13820/1/docs/installation.adoc@645
PS1, Line 645: rebooting
reboot
http://gerrit.cloudera.org:8080/#/c/13820/1/docs/installation.adoc@648
PS1, Line 648: . Restore the original gflags.
Likewise, this duplicates the suggestion on L645.
http://gerrit.cloudera.org:8080/#/c/13820/1/docs/installation.adoc@650
PS1, Line 650: WARNING: To prevent the restarted `tserver` from being evicted from the cluster, the gflag should be reset.
Could you roll this into the rolling restart instructions? No need to duplicate the same suggestion.
--
To view, visit http://gerrit.cloudera.org:8080/13820
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6b3e5c549dc05c3388c0b0dd628d205a356da344
Gerrit-Change-Number: 13820
Gerrit-PatchSet: 1
Gerrit-Owner: helifu <hz...@corp.netease.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Wed, 10 Jul 2019 03:36:52 +0000
Gerrit-HasComments: Yes
[kudu-CR] [docs] update the upgrade documentation
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13820 )
Change subject: [docs] update the upgrade documentation
......................................................................
Patch Set 5: Code-Review+1
(2 comments)
Pretty much LGTM! Thanks for writing this up :)
http://gerrit.cloudera.org:8080/#/c/13820/5/docs/installation.adoc
File docs/installation.adoc:
http://gerrit.cloudera.org:8080/#/c/13820/5/docs/installation.adoc@666
PS5, Line 666: #Step1:
nit: add spaces, e.g.
# Step 1: Set...
http://gerrit.cloudera.org:8080/#/c/13820/5/docs/installation.adoc@672
PS5, Line 672: restart A and wait until A is online
nit: maybe to make it more obvious this itself is an action and not a comment or something, wrap it in <>s? e.g.
<restart A and wait until A is online>
--
To view, visit http://gerrit.cloudera.org:8080/13820
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6b3e5c549dc05c3388c0b0dd628d205a356da344
Gerrit-Change-Number: 13820
Gerrit-PatchSet: 5
Gerrit-Owner: helifu <hz...@corp.netease.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Priyanka Chheda <pc...@cloudera.com>
Gerrit-Reviewer: helifu <hz...@corp.netease.com>
Gerrit-Comment-Date: Tue, 16 Jul 2019 05:39:22 +0000
Gerrit-HasComments: Yes
[kudu-CR] [docs] update the upgrade documentation
Posted by "helifu (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins, Andrew Wong, Adar Dembo, Priyanka Chheda, Grant Henke,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/13820
to look at the new patch set (#5).
Change subject: [docs] update the upgrade documentation
......................................................................
[docs] update the upgrade documentation
The process of upgrading the cluster has been added to
the installation.adoc.
Change-Id: I6b3e5c549dc05c3388c0b0dd628d205a356da344
---
M docs/installation.adoc
1 file changed, 52 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/20/13820/5
--
To view, visit http://gerrit.cloudera.org:8080/13820
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6b3e5c549dc05c3388c0b0dd628d205a356da344
Gerrit-Change-Number: 13820
Gerrit-PatchSet: 5
Gerrit-Owner: helifu <hz...@corp.netease.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Priyanka Chheda <pc...@cloudera.com>
Gerrit-Reviewer: helifu <hz...@corp.netease.com>
[kudu-CR] [docs] update the upgrade documentation
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13820 )
Change subject: [docs] update the upgrade documentation
......................................................................
[docs] update the upgrade documentation
The process of upgrading the cluster has been added to
the installation.adoc.
Change-Id: I6b3e5c549dc05c3388c0b0dd628d205a356da344
Reviewed-on: http://gerrit.cloudera.org:8080/13820
Tested-by: Kudu Jenkins
Reviewed-by: Andrew Wong <aw...@cloudera.com>
---
M docs/installation.adoc
1 file changed, 52 insertions(+), 2 deletions(-)
Approvals:
Kudu Jenkins: Verified
Andrew Wong: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/13820
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I6b3e5c549dc05c3388c0b0dd628d205a356da344
Gerrit-Change-Number: 13820
Gerrit-PatchSet: 7
Gerrit-Owner: helifu <hz...@corp.netease.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Priyanka Chheda <pc...@cloudera.com>
Gerrit-Reviewer: helifu <hz...@corp.netease.com>
[kudu-CR] [docs] update the upgrade documentation
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13820 )
Change subject: [docs] update the upgrade documentation
......................................................................
Patch Set 4:
(6 comments)
http://gerrit.cloudera.org:8080/#/c/13820/4/docs/installation.adoc
File docs/installation.adoc:
http://gerrit.cloudera.org:8080/#/c/13820/4/docs/installation.adoc@638
PS4, Line 638: r`
nit: add a comma here
http://gerrit.cloudera.org:8080/#/c/13820/4/docs/installation.adoc@641
PS4, Line 641: Set the unavailable time for every tablet server to a large value (2x your expected
: restart time) by gflag
: link:configuration_reference.html#kudu-tserver_follower_unavailable_considered_failed_sec[follower_unavailable_considered_failed_sec].
nit: maybe reword this part to include more info so we can get rid of the NOTE below, maybe something like:
"Set the `follower_unavailable_considered_failed_sec` configuration to a high value (conservatively, twice the expected restart time) to prevent tablet replicas hosted on restarting tablet servers from being evicted and re-replicated."
http://gerrit.cloudera.org:8080/#/c/13820/4/docs/installation.adoc@651
PS4, Line 651: /tablets
nit: add backticks here?
http://gerrit.cloudera.org:8080/#/c/13820/4/docs/installation.adoc@652
PS4, Line 652: - Again raise the `--follower_unavailable_considered_failed_sec` runtime gflag.
nit: maybe clarify this a bit:
"Restarting the tablet server will have reset the `follower_unavailable_considered_failed_sec` configuration. Raise it again as needed."
http://gerrit.cloudera.org:8080/#/c/13820/4/docs/installation.adoc@656
PS4, Line 656: Re
nit: lowercase "replicas"
http://gerrit.cloudera.org:8080/#/c/13820/4/docs/installation.adoc@660
PS4, Line 660: - Restore the default gflag value (5 minutes) for every tablet server.
nit: reword, "Restore the original gflag value of every tablet server (the default is 5 minutes)."
--
To view, visit http://gerrit.cloudera.org:8080/13820
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6b3e5c549dc05c3388c0b0dd628d205a356da344
Gerrit-Change-Number: 13820
Gerrit-PatchSet: 4
Gerrit-Owner: helifu <hz...@corp.netease.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Priyanka Chheda <pc...@cloudera.com>
Gerrit-Reviewer: helifu <hz...@corp.netease.com>
Gerrit-Comment-Date: Tue, 16 Jul 2019 03:49:22 +0000
Gerrit-HasComments: Yes
[kudu-CR] [docs] update the upgrade documentation
Posted by "helifu (Code Review)" <ge...@cloudera.org>.
helifu has posted comments on this change. ( http://gerrit.cloudera.org:8080/13820 )
Change subject: [docs] update the upgrade documentation
......................................................................
Patch Set 4:
(6 comments)
http://gerrit.cloudera.org:8080/#/c/13820/4/docs/installation.adoc
File docs/installation.adoc:
http://gerrit.cloudera.org:8080/#/c/13820/4/docs/installation.adoc@638
PS4, Line 638: r`
> nit: add a comma here
Done
http://gerrit.cloudera.org:8080/#/c/13820/4/docs/installation.adoc@641
PS4, Line 641: Set the unavailable time for every tablet server to a large value (2x your expected
: restart time) by gflag
: link:configuration_reference.html#kudu-tserver_follower_unavailable_considered_failed_sec[follower_unavailable_considered_failed_sec].
> nit: maybe reword this part to include more info so we can get rid of the N
Done
http://gerrit.cloudera.org:8080/#/c/13820/4/docs/installation.adoc@651
PS4, Line 651: /tablets
> nit: add backticks here?
Done
http://gerrit.cloudera.org:8080/#/c/13820/4/docs/installation.adoc@652
PS4, Line 652: - Again raise the `--follower_unavailable_considered_failed_sec` runtime gflag.
> nit: maybe clarify this a bit:
Done
http://gerrit.cloudera.org:8080/#/c/13820/4/docs/installation.adoc@656
PS4, Line 656: Re
> nit: lowercase "replicas"
Done
http://gerrit.cloudera.org:8080/#/c/13820/4/docs/installation.adoc@660
PS4, Line 660: - Restore the default gflag value (5 minutes) for every tablet server.
> nit: reword, "Restore the original gflag value of every tablet server (the
Done
--
To view, visit http://gerrit.cloudera.org:8080/13820
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6b3e5c549dc05c3388c0b0dd628d205a356da344
Gerrit-Change-Number: 13820
Gerrit-PatchSet: 4
Gerrit-Owner: helifu <hz...@corp.netease.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Priyanka Chheda <pc...@cloudera.com>
Gerrit-Reviewer: helifu <hz...@corp.netease.com>
Gerrit-Comment-Date: Tue, 16 Jul 2019 04:52:40 +0000
Gerrit-HasComments: Yes
[kudu-CR] [docs] update the upgrade documentation
Posted by "helifu (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins, Adar Dembo, Priyanka Chheda,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/13820
to look at the new patch set (#3).
Change subject: [docs] update the upgrade documentation
......................................................................
[docs] update the upgrade documentation
The process of upgrading the cluster has been added to
the installation.adoc.
Change-Id: I6b3e5c549dc05c3388c0b0dd628d205a356da344
---
M docs/installation.adoc
1 file changed, 52 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/20/13820/3
--
To view, visit http://gerrit.cloudera.org:8080/13820
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6b3e5c549dc05c3388c0b0dd628d205a356da344
Gerrit-Change-Number: 13820
Gerrit-PatchSet: 3
Gerrit-Owner: helifu <hz...@corp.netease.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Priyanka Chheda <pc...@cloudera.com>
Gerrit-Reviewer: helifu <hz...@corp.netease.com>
[kudu-CR] [docs] update the upgrade documentation
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/13820 )
Change subject: [docs] update the upgrade documentation
......................................................................
Patch Set 2:
(11 comments)
http://gerrit.cloudera.org:8080/#/c/13820/2/docs/installation.adoc
File docs/installation.adoc:
http://gerrit.cloudera.org:8080/#/c/13820/2/docs/installation.adoc@635
PS2, Line 635: for the scenario of compiling from source code.
"relevant when building from source code".
http://gerrit.cloudera.org:8080/#/c/13820/2/docs/installation.adoc@640
PS2, Line 640: kudu-tserver
"tablet server"
http://gerrit.cloudera.org:8080/#/c/13820/2/docs/installation.adoc@640
PS2, Line 640: 2hours
"2 hours"
http://gerrit.cloudera.org:8080/#/c/13820/2/docs/installation.adoc@645
PS2, Line 645: --force
Shouldn't need this; the flag is tagged as 'runtime'.
http://gerrit.cloudera.org:8080/#/c/13820/2/docs/installation.adoc@647
PS2, Line 647: and the gflag above should be reset after every reboot.
Since step 4 restores the gflag value, do we really need this step?
http://gerrit.cloudera.org:8080/#/c/13820/2/docs/installation.adoc@647
PS2, Line 647: kudu-tserver
"the tablet servers"
http://gerrit.cloudera.org:8080/#/c/13820/2/docs/installation.adoc@649
PS2, Line 649: [NOTE]
Shouldn't this be just after "Set the unavailable time..." so users understand why it's important to change the gflag value before the rolling restart?
http://gerrit.cloudera.org:8080/#/c/13820/2/docs/installation.adoc@651
PS2, Line 651: If the gflag is not reset, kudu-tserver which is restarting would be possibly evicted from the cluster.
"If the unavailable time is not extended, restarted tablet servers could be evicted from the cluster."
http://gerrit.cloudera.org:8080/#/c/13820/2/docs/installation.adoc@654
PS2, Line 654: Rolling restart
We should define what "rolling restart" means too.
http://gerrit.cloudera.org:8080/#/c/13820/2/docs/installation.adoc@654
PS2, Line 654: kudu-master
"the masters"
http://gerrit.cloudera.org:8080/#/c/13820/2/docs/installation.adoc@655
PS2, Line 655: . Restore the default gflag value (5 minutes) for every kudu-tserver.
: +
: [source,bash]
: ----
: ./kudu tserver set_flag <tserver> follower_unavailable_considered_failed_sec 300 --force
: ----
Is this actually necessary? Won't restarting the process restore the default value of all of its gflags?
Come to think of it, how does the gflag value overriding work if restarting a tserver restores the original value? Seems like we'd need to restart the tserver AND THEN override follower_unavailable_considered_failed_sec.
--
To view, visit http://gerrit.cloudera.org:8080/13820
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6b3e5c549dc05c3388c0b0dd628d205a356da344
Gerrit-Change-Number: 13820
Gerrit-PatchSet: 2
Gerrit-Owner: helifu <hz...@corp.netease.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Priyanka Chheda <pc...@cloudera.com>
Gerrit-Reviewer: helifu <hz...@corp.netease.com>
Gerrit-Comment-Date: Wed, 10 Jul 2019 14:19:13 +0000
Gerrit-HasComments: Yes
[kudu-CR] [docs] update the upgrade documentation
Posted by "helifu (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins, Andrew Wong, Adar Dembo, Priyanka Chheda, Grant Henke,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/13820
to look at the new patch set (#6).
Change subject: [docs] update the upgrade documentation
......................................................................
[docs] update the upgrade documentation
The process of upgrading the cluster has been added to
the installation.adoc.
Change-Id: I6b3e5c549dc05c3388c0b0dd628d205a356da344
---
M docs/installation.adoc
1 file changed, 52 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/20/13820/6
--
To view, visit http://gerrit.cloudera.org:8080/13820
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6b3e5c549dc05c3388c0b0dd628d205a356da344
Gerrit-Change-Number: 13820
Gerrit-PatchSet: 6
Gerrit-Owner: helifu <hz...@corp.netease.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Priyanka Chheda <pc...@cloudera.com>
Gerrit-Reviewer: helifu <hz...@corp.netease.com>
[kudu-CR] [docs] update the upgrade documentation
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/13820 )
Change subject: [docs] update the upgrade documentation
......................................................................
Patch Set 3: Code-Review+1
(1 comment)
Would be good for someone else to review too.
http://gerrit.cloudera.org:8080/#/c/13820/3/docs/installation.adoc
File docs/installation.adoc:
http://gerrit.cloudera.org:8080/#/c/13820/3/docs/installation.adoc@652
PS3, Line 652: restarted tablet servers
"Replicas hosted on restarted tablet servers"
--
To view, visit http://gerrit.cloudera.org:8080/13820
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6b3e5c549dc05c3388c0b0dd628d205a356da344
Gerrit-Change-Number: 13820
Gerrit-PatchSet: 3
Gerrit-Owner: helifu <hz...@corp.netease.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Priyanka Chheda <pc...@cloudera.com>
Gerrit-Reviewer: helifu <hz...@corp.netease.com>
Gerrit-Comment-Date: Thu, 11 Jul 2019 05:55:09 +0000
Gerrit-HasComments: Yes
[kudu-CR] [docs] update the upgrade documentation
Posted by "helifu (Code Review)" <ge...@cloudera.org>.
helifu has posted comments on this change. ( http://gerrit.cloudera.org:8080/13820 )
Change subject: [docs] update the upgrade documentation
......................................................................
Patch Set 5:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/13820/5/docs/installation.adoc
File docs/installation.adoc:
http://gerrit.cloudera.org:8080/#/c/13820/5/docs/installation.adoc@666
PS5, Line 666: #Step1:
> nit: add spaces, e.g.
Done
http://gerrit.cloudera.org:8080/#/c/13820/5/docs/installation.adoc@672
PS5, Line 672: restart A and wait until A is online
> nit: maybe to make it more obvious this itself is an action and not a comme
Done
--
To view, visit http://gerrit.cloudera.org:8080/13820
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6b3e5c549dc05c3388c0b0dd628d205a356da344
Gerrit-Change-Number: 13820
Gerrit-PatchSet: 5
Gerrit-Owner: helifu <hz...@corp.netease.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Priyanka Chheda <pc...@cloudera.com>
Gerrit-Reviewer: helifu <hz...@corp.netease.com>
Gerrit-Comment-Date: Tue, 16 Jul 2019 05:53:09 +0000
Gerrit-HasComments: Yes
[kudu-CR] [docs] update the upgrade documentation
Posted by "Grant Henke (Code Review)" <ge...@cloudera.org>.
Grant Henke has posted comments on this change. ( http://gerrit.cloudera.org:8080/13820 )
Change subject: [docs] update the upgrade documentation
......................................................................
Patch Set 3:
(5 comments)
I added a few comments. I would also be interested in Andrews feedback. He recently started looking at improved rolling restarts and upgrades which includes things like clean shutdowns to reduce startup time, faster startups, etc.
http://gerrit.cloudera.org:8080/#/c/13820/3/docs/installation.adoc
File docs/installation.adoc:
http://gerrit.cloudera.org:8080/#/c/13820/3/docs/installation.adoc@a635
PS3, Line 635:
I like the idea of documenting how to do a rolling upgrade because we know users are already doing it. However, I think we should prioritize writing upgrade tests that can be run for each release if we are going to encourage it and remove the disclaimer about it being supported.
http://gerrit.cloudera.org:8080/#/c/13820/3/docs/installation.adoc@638
PS3, Line 638: - Copy the `kudu-tserver`, `kudu-master` and `kudu` binaries from your build directory.
Maybe say something like "Replace the `kudu-server`..."
Otherwise it's not clear what you mean by "copy".
http://gerrit.cloudera.org:8080/#/c/13820/3/docs/installation.adoc@640
PS3, Line 640: - Set the unavailable time for every tablet server to a large value (2 hours or more) by gflag
The 2 hours or more feels a bit arbitrary. Maybe say something like 2x your expected restart time? The example command will still have 2 hours as the sample value.
http://gerrit.cloudera.org:8080/#/c/13820/3/docs/installation.adoc@647
PS3, Line 647: wait until it is online
Maybe note what a user should a user look at to know it's online?
http://gerrit.cloudera.org:8080/#/c/13820/3/docs/installation.adoc@652
PS3, Line 652: restarted tablet servers
> "Replicas hosted on restarted tablet servers"
This seems like a race against time given the default is 5 minutes.
In general an upgrade doesn't take longer than the average restart, so I am not sure setting this config is needed. 3-4-3 replication will prevent to much re-replication, and if this flag is needed for restarts it might need to be set permanently anyway.
--
To view, visit http://gerrit.cloudera.org:8080/13820
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6b3e5c549dc05c3388c0b0dd628d205a356da344
Gerrit-Change-Number: 13820
Gerrit-PatchSet: 3
Gerrit-Owner: helifu <hz...@corp.netease.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Priyanka Chheda <pc...@cloudera.com>
Gerrit-Reviewer: helifu <hz...@corp.netease.com>
Gerrit-Comment-Date: Thu, 11 Jul 2019 14:22:19 +0000
Gerrit-HasComments: Yes
[kudu-CR] [docs] update the upgrade documentation
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13820 )
Change subject: [docs] update the upgrade documentation
......................................................................
Patch Set 3:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/13820/3/docs/installation.adoc
File docs/installation.adoc:
http://gerrit.cloudera.org:8080/#/c/13820/3/docs/installation.adoc@635
PS3, Line 635: WARNING: The following upgrade process is only relevant when building from source code.
Why is this the case?
I guess if you are using thirdparty software to manage Kudu (e.g. Cloudera Manager) it wouldn't be quite right. Bui the steps seem to be fine as long as you have binaries available, regardless of whether or not you built it from source.
http://gerrit.cloudera.org:8080/#/c/13820/3/docs/installation.adoc@638
PS3, Line 638: - Copy the `kudu-tserver`, `kudu-master` and `kudu` binaries from your build directory.
> Maybe say something like "Replace the `kudu-server`..."
Maybe "Place the new `kudu-tserver`, `kudu-master`, and `kudu` binaries into the appropriate Kudu binary directory."
http://gerrit.cloudera.org:8080/#/c/13820/3/docs/installation.adoc@647
PS3, Line 647: - Restart a tablet server and wait until it is online, then reset the gflag above to be 7200 again.
I think this should be broken down further:
- Restart the tablet server.
- Again raise the `--follower_unavailable_considered_failed_sec` runtime gflag.
- Wait for all tablet replicas on the tablet server to finish bootstrapping by viewing /tablets page in the tablet server web UI.
This would help clarify exactly what a user needs to wait for: the user should confirm the health of the tablets on the tablet server before bringing down the next server.
http://gerrit.cloudera.org:8080/#/c/13820/3/docs/installation.adoc@651
PS3, Line 651: Make sure the restarted tablet server is already online before resetting the gflag.
This is good information to have, but I'd be concerned that it would confuse operators going through these steps since "Make sure the restarted tablet server is already online before resetting the gflag" is already a part of the listed steps, so maybe remove this note?
--
To view, visit http://gerrit.cloudera.org:8080/13820
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6b3e5c549dc05c3388c0b0dd628d205a356da344
Gerrit-Change-Number: 13820
Gerrit-PatchSet: 3
Gerrit-Owner: helifu <hz...@corp.netease.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Priyanka Chheda <pc...@cloudera.com>
Gerrit-Reviewer: helifu <hz...@corp.netease.com>
Gerrit-Comment-Date: Mon, 15 Jul 2019 06:01:18 +0000
Gerrit-HasComments: Yes
[kudu-CR] [docs] update the upgrade documentation
Posted by "helifu (Code Review)" <ge...@cloudera.org>.
helifu has posted comments on this change. ( http://gerrit.cloudera.org:8080/13820 )
Change subject: [docs] update the upgrade documentation
......................................................................
Patch Set 3:
(10 comments)
http://gerrit.cloudera.org:8080/#/c/13820/3/docs/installation.adoc
File docs/installation.adoc:
http://gerrit.cloudera.org:8080/#/c/13820/3/docs/installation.adoc@a635
PS3, Line 635:
> I like the idea of documenting how to do a rolling upgrade because we know
We have been using this rolling upgrade approach for almost 2 years and I'm sure others are also using it too. I want to hear what other people think about upgrade tests :)
http://gerrit.cloudera.org:8080/#/c/13820/3/docs/installation.adoc@635
PS3, Line 635: WARNING: The following upgrade process is only relevant when building from source code.
> Why is this the case?
Done
http://gerrit.cloudera.org:8080/#/c/13820/3/docs/installation.adoc@638
PS3, Line 638: - Copy the `kudu-tserver`, `kudu-master` and `kudu` binaries from your build directory.
> Maybe say something like "Replace the `kudu-server`..."
Done
http://gerrit.cloudera.org:8080/#/c/13820/3/docs/installation.adoc@638
PS3, Line 638: - Copy the `kudu-tserver`, `kudu-master` and `kudu` binaries from your build directory.
> Maybe "Place the new `kudu-tserver`, `kudu-master`, and `kudu` binaries int
Done
http://gerrit.cloudera.org:8080/#/c/13820/3/docs/installation.adoc@640
PS3, Line 640: - Set the unavailable time for every tablet server to a large value (2 hours or more) by gflag
> The 2 hours or more feels a bit arbitrary. Maybe say something like 2x your
Done
http://gerrit.cloudera.org:8080/#/c/13820/3/docs/installation.adoc@647
PS3, Line 647: - Restart a tablet server and wait until it is online, then reset the gflag above to be 7200 again.
> I think this should be broken down further:
Done
http://gerrit.cloudera.org:8080/#/c/13820/3/docs/installation.adoc@647
PS3, Line 647: wait until it is online
> Maybe note what a user should a user look at to know it's online?
Done
http://gerrit.cloudera.org:8080/#/c/13820/3/docs/installation.adoc@651
PS3, Line 651: Make sure the restarted tablet server is already online before resetting the gflag.
> This is good information to have, but I'd be concerned that it would confus
Done
http://gerrit.cloudera.org:8080/#/c/13820/3/docs/installation.adoc@652
PS3, Line 652: restarted tablet servers
> "Replicas hosted on restarted tablet servers"
Done
http://gerrit.cloudera.org:8080/#/c/13820/3/docs/installation.adoc@652
PS3, Line 652: restarted tablet servers
> This seems like a race against time given the default is 5 minutes.
In our production cluster, it takes more than half an hour to restart a tserver on average. So, it's important to raise the gflag.
--
To view, visit http://gerrit.cloudera.org:8080/13820
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6b3e5c549dc05c3388c0b0dd628d205a356da344
Gerrit-Change-Number: 13820
Gerrit-PatchSet: 3
Gerrit-Owner: helifu <hz...@corp.netease.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Priyanka Chheda <pc...@cloudera.com>
Gerrit-Reviewer: helifu <hz...@corp.netease.com>
Gerrit-Comment-Date: Mon, 15 Jul 2019 11:37:16 +0000
Gerrit-HasComments: Yes
[kudu-CR] [docs] update the upgrade documentation
Posted by "helifu (Code Review)" <ge...@cloudera.org>.
helifu has posted comments on this change. ( http://gerrit.cloudera.org:8080/13820 )
Change subject: [docs] update the upgrade documentation
......................................................................
Patch Set 1:
(7 comments)
Here is a link to display the rendered form:
https://github.com/helifu/kudu/blob/master/docs/installation.adoc#upgrade
And I use "asciidoctor installation.adoc -b html" to verify it locally.
http://gerrit.cloudera.org:8080/#/c/13820/1/docs/installation.adoc
File docs/installation.adoc:
http://gerrit.cloudera.org:8080/#/c/13820/1/docs/installation.adoc@635
PS1, Line 635: . Prepare the software.
> This recommendation is only necessary if you're actually building from sour
Done
http://gerrit.cloudera.org:8080/#/c/13820/1/docs/installation.adoc@636
PS1, Line 636: building
> build
Done
http://gerrit.cloudera.org:8080/#/c/13820/1/docs/installation.adoc@638
PS1, Line 638: big
> large
Done
http://gerrit.cloudera.org:8080/#/c/13820/1/docs/installation.adoc@645
PS1, Line 645: - Rolling restart `kudu-tserver` and the gflag above should be reset after every rebooting.
> Could you provide specific instructions for restoring the default gflag val
Done
http://gerrit.cloudera.org:8080/#/c/13820/1/docs/installation.adoc@645
PS1, Line 645: rebooting
> reboot
Done
http://gerrit.cloudera.org:8080/#/c/13820/1/docs/installation.adoc@648
PS1, Line 648: . Restore the original gflags.
> Likewise, this duplicates the suggestion on L645.
Done
http://gerrit.cloudera.org:8080/#/c/13820/1/docs/installation.adoc@650
PS1, Line 650: WARNING: To prevent the restarted `tserver` from being evicted from the cluster, the gflag should be reset.
> Could you roll this into the rolling restart instructions? No need to dupli
Done
--
To view, visit http://gerrit.cloudera.org:8080/13820
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6b3e5c549dc05c3388c0b0dd628d205a356da344
Gerrit-Change-Number: 13820
Gerrit-PatchSet: 1
Gerrit-Owner: helifu <hz...@corp.netease.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Priyanka Chheda <pc...@cloudera.com>
Gerrit-Reviewer: helifu <hz...@corp.netease.com>
Gerrit-Comment-Date: Wed, 10 Jul 2019 08:26:27 +0000
Gerrit-HasComments: Yes
[kudu-CR] [docs] update the upgrade documentation
Posted by "helifu (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins, Andrew Wong, Adar Dembo, Priyanka Chheda, Grant Henke,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/13820
to look at the new patch set (#4).
Change subject: [docs] update the upgrade documentation
......................................................................
[docs] update the upgrade documentation
The process of upgrading the cluster has been added to
the installation.adoc.
Change-Id: I6b3e5c549dc05c3388c0b0dd628d205a356da344
---
M docs/installation.adoc
1 file changed, 57 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/20/13820/4
--
To view, visit http://gerrit.cloudera.org:8080/13820
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6b3e5c549dc05c3388c0b0dd628d205a356da344
Gerrit-Change-Number: 13820
Gerrit-PatchSet: 4
Gerrit-Owner: helifu <hz...@corp.netease.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Priyanka Chheda <pc...@cloudera.com>
Gerrit-Reviewer: helifu <hz...@corp.netease.com>
[kudu-CR] [docs] update the upgrade documentation
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/13820 )
Change subject: [docs] update the upgrade documentation
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/13820/2/docs/installation.adoc
File docs/installation.adoc:
http://gerrit.cloudera.org:8080/#/c/13820/2/docs/installation.adoc@647
PS2, Line 647: and the gflag above should be reset after every reboot.
> The 'reset' here means setting to 7200.
Gotcha. Can you add this to the documentation?
There's another catch: depending on the size of the deployment, it could take some time until the tserver starts responding to RPCs. For example, I think the block manager has to fully load first, and the tserver has to finish waiting for NTP to synchronize (if it's not already synchronized).
--
To view, visit http://gerrit.cloudera.org:8080/13820
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6b3e5c549dc05c3388c0b0dd628d205a356da344
Gerrit-Change-Number: 13820
Gerrit-PatchSet: 2
Gerrit-Owner: helifu <hz...@corp.netease.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Priyanka Chheda <pc...@cloudera.com>
Gerrit-Reviewer: helifu <hz...@corp.netease.com>
Gerrit-Comment-Date: Wed, 10 Jul 2019 23:00:39 +0000
Gerrit-HasComments: Yes