You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Ádám Bakai (Code Review)" <ge...@cloudera.org> on 2024/02/06 14:44:16 UTC
[kudu-CR] [catalog manager] Tighten leader UUID fallback
Ádám Bakai has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21004
Change subject: [catalog_manager] Tighten leader UUID fallback
......................................................................
[catalog_manager] Tighten leader UUID fallback
It is safe to assume that if the term is the same in the current cstate
as in the previous cstate then even if the leader is not set, it will
be the same. But it is possible that cmeta file is deleted then
recreated with "local_replica cmeta unsafe_recreate" command. In this
case the leader_uuid is empty in the new cmeta file. This means that the
peer doesn't consider itself a leader, so no health report is generated
in tablet report and it has no leader_uuid set either. When a master
receives tablet report like this and there isn't a new term, then the
catalog master will treat this peer as a leader, but it will fail on a
check because the leader has to be in healthy status. This happened in
ToolTest::TestRecreateCMeta.
The solution is that catalog manager only assumes the previous leader
for the peer if the previous leader is not the peer itself. This gives
time for the peers to form a consensus about the leader.
Change-Id: I06a80a4a0a9fd422b50860e8cd8bf0e12973cd43
---
M src/kudu/master/catalog_manager.cc
1 file changed, 2 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/04/21004/1
--
To view, visit http://gerrit.cloudera.org:8080/21004
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I06a80a4a0a9fd422b50860e8cd8bf0e12973cd43
Gerrit-Change-Number: 21004
Gerrit-PatchSet: 1
Gerrit-Owner: Ádám Bakai <ab...@cloudera.com>
[kudu-CR] [catalog manager] Tighten leader UUID fallback
Posted by "Ádám Bakai (Code Review)" <ge...@cloudera.org>.
Ádám Bakai has posted comments on this change. ( http://gerrit.cloudera.org:8080/21004 )
Change subject: [catalog_manager] Tighten leader UUID fallback
......................................................................
Patch Set 1:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/21004/1//COMMIT_MSG
Commit Message:
PS1:
> Thank you for getting to the bottom of this!
I added an extension to the TestRecreateCMeta so that it is tested in single TServer configuration,too. That is a sure way to reproduce the error. Maybe it's enough to only run with a single TServer, but I thought multiple TServer test is nice to have, too.
If I understand correctly, the KUDU-2335 is still present. I tried to reproduce it without the patch and I had no luck. I think it happens because some very rare race condition between the raft consensus thread and of tablet report generation thread that is triggered by the voting process.
http://gerrit.cloudera.org:8080/#/c/21004/1/src/kudu/master/catalog_manager.cc
File src/kudu/master/catalog_manager.cc:
http://gerrit.cloudera.org:8080/#/c/21004/1/src/kudu/master/catalog_manager.cc@5456
PS1, Line 5456: // previous cstate, and the leader was known for that term.
> Could you please add an extra comment, explaining the essence of the extra
Done
--
To view, visit http://gerrit.cloudera.org:8080/21004
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I06a80a4a0a9fd422b50860e8cd8bf0e12973cd43
Gerrit-Change-Number: 21004
Gerrit-PatchSet: 1
Gerrit-Owner: Ádám Bakai <ab...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Ádám Bakai <ab...@cloudera.com>
Gerrit-Comment-Date: Wed, 07 Feb 2024 15:30:09 +0000
Gerrit-HasComments: Yes
[kudu-CR] [catalog manager] Tighten leader UUID fallback
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/21004 )
Change subject: [catalog_manager] Tighten leader UUID fallback
......................................................................
Patch Set 3: Code-Review+2
Thank you for performing the root cause analysis and fixing the issue!
--
To view, visit http://gerrit.cloudera.org:8080/21004
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I06a80a4a0a9fd422b50860e8cd8bf0e12973cd43
Gerrit-Change-Number: 21004
Gerrit-PatchSet: 3
Gerrit-Owner: Ádám Bakai <ab...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Ádám Bakai <ab...@cloudera.com>
Gerrit-Comment-Date: Fri, 09 Feb 2024 00:40:03 +0000
Gerrit-HasComments: No
[kudu-CR] [catalog manager] Tighten leader UUID fallback
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/21004 )
Change subject: [catalog_manager] Tighten leader UUID fallback
......................................................................
Patch Set 1: Code-Review+1
(2 comments)
http://gerrit.cloudera.org:8080/#/c/21004/1//COMMIT_MSG
Commit Message:
PS1:
Thank you for getting to the bottom of this!
Do you think it makes sense to add a new test or update the existing one (TestRecreateCMeta?) to make the issue happening more consistently when the fix is not present? My concern is that KUDU-2335 was manifesting itself rarely, and if there is a regression in the code once it's updated, it would be hard to detect the issue right away.
http://gerrit.cloudera.org:8080/#/c/21004/1/src/kudu/master/catalog_manager.cc
File src/kudu/master/catalog_manager.cc:
http://gerrit.cloudera.org:8080/#/c/21004/1/src/kudu/master/catalog_manager.cc@5456
PS1, Line 5456: // previous cstate, and the leader was known for that term.
Could you please add an extra comment, explaining the essence of the extra criterion added?
The extra comment might be something like below (but it's up to you to decide what's the best way to describe the rationale, of course):
An extra condition is to check whether it's a report from a former leader replica which currently doesn't maintain the leadership role. Such a situation is possible when the replica's cmeta file had been deleted and then recreated (e.g. by the "kudu local_replica cmeta unsafe_recreate" CLI tool). The code below assumes that the replica effectively has the leadership if the 'leader_uuid' field is set, but that's not so (see KUDU-2335).
--
To view, visit http://gerrit.cloudera.org:8080/21004
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I06a80a4a0a9fd422b50860e8cd8bf0e12973cd43
Gerrit-Change-Number: 21004
Gerrit-PatchSet: 1
Gerrit-Owner: Ádám Bakai <ab...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Wed, 07 Feb 2024 01:11:35 +0000
Gerrit-HasComments: Yes
[kudu-CR] [catalog manager] Tighten leader UUID fallback
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has removed a vote on this change.
Change subject: [catalog_manager] Tighten leader UUID fallback
......................................................................
Removed Verified-1 by Kudu Jenkins (120)
--
To view, visit http://gerrit.cloudera.org:8080/21004
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: I06a80a4a0a9fd422b50860e8cd8bf0e12973cd43
Gerrit-Change-Number: 21004
Gerrit-PatchSet: 3
Gerrit-Owner: Ádám Bakai <ab...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Ádám Bakai <ab...@cloudera.com>
[kudu-CR] [catalog manager] Tighten leader UUID fallback
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/21004 )
Change subject: [catalog_manager] Tighten leader UUID fallback
......................................................................
Patch Set 2: Code-Review+1
(2 comments)
http://gerrit.cloudera.org:8080/#/c/21004/1//COMMIT_MSG
Commit Message:
PS1:
> I added an extension to the TestRecreateCMeta so that it is tested in singl
The reason for KUDU-2335 is still a mystery to me. Maybe, that's indeed some race, but at least your fix helps to address one of the cases when the DCHECK() in quorum_util.cc on non-healthy leader triggered.
Thanks!
http://gerrit.cloudera.org:8080/#/c/21004/1/src/kudu/master/catalog_manager.cc
File src/kudu/master/catalog_manager.cc:
http://gerrit.cloudera.org:8080/#/c/21004/1/src/kudu/master/catalog_manager.cc@5456
PS1, Line 5456: // previous cstate, and the leader was known for that term.
> Done
This comment says 'Done', but I could not see the requested extra comment in PS2. Did I miss something?
--
To view, visit http://gerrit.cloudera.org:8080/21004
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I06a80a4a0a9fd422b50860e8cd8bf0e12973cd43
Gerrit-Change-Number: 21004
Gerrit-PatchSet: 2
Gerrit-Owner: Ádám Bakai <ab...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Ádám Bakai <ab...@cloudera.com>
Gerrit-Comment-Date: Thu, 08 Feb 2024 06:18:19 +0000
Gerrit-HasComments: Yes
[kudu-CR] [catalog manager] Tighten leader UUID fallback
Posted by "Ádám Bakai (Code Review)" <ge...@cloudera.org>.
Hello Alexey Serbin, Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/21004
to look at the new patch set (#2).
Change subject: [catalog_manager] Tighten leader UUID fallback
......................................................................
[catalog_manager] Tighten leader UUID fallback
It is safe to assume that if the term is the same in the current cstate
as in the previous cstate then even if the leader is not set, it will
be the same. But it is possible that cmeta file is deleted then
recreated with "local_replica cmeta unsafe_recreate" command. In this
case the leader_uuid is empty in the new cmeta file. This means that the
peer doesn't consider itself a leader, so no health report is generated
in tablet report and it has no leader_uuid set either. When a master
receives tablet report like this and there isn't a new term, then the
catalog master will treat this peer as a leader, but it will fail on a
check because the leader has to be in healthy status. This happened in
ToolTest::TestRecreateCMeta. As a reproduction step, the same test now
runs with a single TServer configuration, too. In this configuration the
error is reproducible 100% of the times, since the term is not increased
and the leader's cmeta file is changed.
The solution is that catalog manager only assumes the previous leader
for the peer if the previous leader is not the peer itself. This gives
time for the peers to form a consensus about the leader.
Change-Id: I06a80a4a0a9fd422b50860e8cd8bf0e12973cd43
---
M src/kudu/master/catalog_manager.cc
M src/kudu/tools/kudu-tool-test.cc
2 files changed, 11 insertions(+), 3 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/04/21004/2
--
To view, visit http://gerrit.cloudera.org:8080/21004
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I06a80a4a0a9fd422b50860e8cd8bf0e12973cd43
Gerrit-Change-Number: 21004
Gerrit-PatchSet: 2
Gerrit-Owner: Ádám Bakai <ab...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Ádám Bakai <ab...@cloudera.com>
[kudu-CR] [catalog manager] Tighten leader UUID fallback
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/21004 )
Change subject: [catalog_manager] Tighten leader UUID fallback
......................................................................
Patch Set 3: Verified+1
unrelated test failures
--
To view, visit http://gerrit.cloudera.org:8080/21004
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I06a80a4a0a9fd422b50860e8cd8bf0e12973cd43
Gerrit-Change-Number: 21004
Gerrit-PatchSet: 3
Gerrit-Owner: Ádám Bakai <ab...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Ádám Bakai <ab...@cloudera.com>
Gerrit-Comment-Date: Fri, 09 Feb 2024 00:39:33 +0000
Gerrit-HasComments: No
[kudu-CR] [catalog manager] Tighten leader UUID fallback
Posted by "Ádám Bakai (Code Review)" <ge...@cloudera.org>.
Hello Alexey Serbin, Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/21004
to look at the new patch set (#3).
Change subject: [catalog_manager] Tighten leader UUID fallback
......................................................................
[catalog_manager] Tighten leader UUID fallback
It is safe to assume that if the term is the same in the current cstate
as in the previous cstate then even if the leader is not set, it will
be the same. But it is possible that cmeta file is deleted then
recreated with "local_replica cmeta unsafe_recreate" command. In this
case the leader_uuid is empty in the new cmeta file. This means that the
peer doesn't consider itself a leader, so no health report is generated
in tablet report and it has no leader_uuid set either. When a master
receives tablet report like this and there isn't a new term, then the
catalog master will treat this peer as a leader, but it will fail on a
check because the leader has to be in healthy status. This happened in
ToolTest::TestRecreateCMeta. As a reproduction step, the same test now
runs with a single TServer configuration, too. In this configuration the
error is reproducible 100% of the times, since the term is not increased
and the leader's cmeta file is changed.
The solution is that catalog manager only assumes the previous leader
for the peer if the previous leader is not the peer itself. This gives
time for the peers to form a consensus about the leader.
Change-Id: I06a80a4a0a9fd422b50860e8cd8bf0e12973cd43
---
M src/kudu/master/catalog_manager.cc
M src/kudu/tools/kudu-tool-test.cc
2 files changed, 19 insertions(+), 4 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/04/21004/3
--
To view, visit http://gerrit.cloudera.org:8080/21004
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I06a80a4a0a9fd422b50860e8cd8bf0e12973cd43
Gerrit-Change-Number: 21004
Gerrit-PatchSet: 3
Gerrit-Owner: Ádám Bakai <ab...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Ádám Bakai <ab...@cloudera.com>
[kudu-CR] [catalog manager] Tighten leader UUID fallback
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/21004 )
Change subject: [catalog_manager] Tighten leader UUID fallback
......................................................................
[catalog_manager] Tighten leader UUID fallback
It is safe to assume that if the term is the same in the current cstate
as in the previous cstate then even if the leader is not set, it will
be the same. But it is possible that cmeta file is deleted then
recreated with "local_replica cmeta unsafe_recreate" command. In this
case the leader_uuid is empty in the new cmeta file. This means that the
peer doesn't consider itself a leader, so no health report is generated
in tablet report and it has no leader_uuid set either. When a master
receives tablet report like this and there isn't a new term, then the
catalog master will treat this peer as a leader, but it will fail on a
check because the leader has to be in healthy status. This happened in
ToolTest::TestRecreateCMeta. As a reproduction step, the same test now
runs with a single TServer configuration, too. In this configuration the
error is reproducible 100% of the times, since the term is not increased
and the leader's cmeta file is changed.
The solution is that catalog manager only assumes the previous leader
for the peer if the previous leader is not the peer itself. This gives
time for the peers to form a consensus about the leader.
Change-Id: I06a80a4a0a9fd422b50860e8cd8bf0e12973cd43
Reviewed-on: http://gerrit.cloudera.org:8080/21004
Tested-by: Alexey Serbin <al...@apache.org>
Reviewed-by: Alexey Serbin <al...@apache.org>
---
M src/kudu/master/catalog_manager.cc
M src/kudu/tools/kudu-tool-test.cc
2 files changed, 19 insertions(+), 4 deletions(-)
Approvals:
Alexey Serbin: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/21004
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I06a80a4a0a9fd422b50860e8cd8bf0e12973cd43
Gerrit-Change-Number: 21004
Gerrit-PatchSet: 4
Gerrit-Owner: Ádám Bakai <ab...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Ádám Bakai <ab...@cloudera.com>