You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Tim Armstrong (Code Review)" <ge...@cloudera.org> on 2020/03/10 03:31:09 UTC
[kudu-CR] WIP - KUDU-3050: recover from corrupt kerberos ccache
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15394
Change subject: WIP - KUDU-3050: recover from corrupt kerberos ccache
......................................................................
WIP - KUDU-3050: recover from corrupt kerberos ccache
Also add some additional logging and limit the max backoff time
to make it easier to debug other kinds of renewal errors.
Test:
Add a test that exercises the recovery logic after truncating
the credential cache. The test failed before this change.
Change-Id: I2d6e06c3ea65708896a6bf0134cc84838b3f1b58
---
M src/kudu/mini-cluster/external_mini_cluster-test.cc
M src/kudu/security/init.cc
A src/kudu/security/kinit_context.h
M src/kudu/security/test/mini_kdc.cc
M src/kudu/security/test/mini_kdc.h
5 files changed, 161 insertions(+), 66 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/94/15394/1
--
To view, visit http://gerrit.cloudera.org:8080/15394
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I2d6e06c3ea65708896a6bf0134cc84838b3f1b58
Gerrit-Change-Number: 15394
Gerrit-PatchSet: 1
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
[kudu-CR] KUDU-3050: recover from corrupt kerberos ccache
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Alexey Serbin, Kudu Jenkins, Adar Dembo,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15394
to look at the new patch set (#5).
Change subject: KUDU-3050: recover from corrupt kerberos ccache
......................................................................
KUDU-3050: recover from corrupt kerberos ccache
This handles two failure modes:
* krb5_cc_start_seq_get() can fail if the kerberos credential cache gets
corrupted on disk, e.g. is truncated.
* the renewal can fail to find a credential in the credential cache,
either if it is missing or the renewal thread hits an error while
reading through credentials.
Also add some additional logging and limit the max backoff time
to make it easier to debug other kinds of renewal errors.
The test triggers a pre-existing memory leak bug in some older
Kerberos libraries. Added a suppression for leak sanitizer
to ClientNegotiation::CheckGSSAPI() to suppress it.
Test:
Add a test that exercises the recovery logic after truncating
the credential cache. The test failed before this change.
Change-Id: I2d6e06c3ea65708896a6bf0134cc84838b3f1b58
---
M src/kudu/integration-tests/security-itest.cc
M src/kudu/rpc/client_negotiation.cc
M src/kudu/security/init.cc
A src/kudu/security/kinit_context.h
M src/kudu/security/test/mini_kdc.cc
M src/kudu/security/test/mini_kdc.h
6 files changed, 200 insertions(+), 62 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/94/15394/5
--
To view, visit http://gerrit.cloudera.org:8080/15394
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2d6e06c3ea65708896a6bf0134cc84838b3f1b58
Gerrit-Change-Number: 15394
Gerrit-PatchSet: 5
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[kudu-CR] KUDU-3050: recover from corrupt kerberos ccache
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15394 )
Change subject: KUDU-3050: recover from corrupt kerberos ccache
......................................................................
Patch Set 4:
Might be https://krbdev.mit.edu/rt/Ticket/Display.html?id=8437 or similar. I don't see any evidence that Kudu is misusing the APIs. It looks like the memory is allocated in either krb5_cc_resolve() or krb5_cc_dup(), by tracing back from fcc_resolve.
--
To view, visit http://gerrit.cloudera.org:8080/15394
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2d6e06c3ea65708896a6bf0134cc84838b3f1b58
Gerrit-Change-Number: 15394
Gerrit-PatchSet: 4
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 10 Mar 2020 22:44:21 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-3050: recover from corrupt kerberos ccache
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/15394 )
Change subject: KUDU-3050: recover from corrupt kerberos ccache
......................................................................
Patch Set 4:
(5 comments)
> > Patch Set 4:
> >
> > Might be https://krbdev.mit.edu/rt/Ticket/Display.html?id=8437 or
> similar. I don't see any evidence that Kudu is misusing the APIs.
> It looks like the memory is allocated in either krb5_cc_resolve()
> or krb5_cc_dup(), by tracing back from fcc_resolve.
>
> Try running the test with ASAN_OPTIONS=fast_unwind_on_malloc=0 in
> the environment. That should give you a more complete stack trace.
Should we call krb5_cc_end_seq_get() even if prior call to krb5_cc_start_seq_get() returned an error? Maybe, that's the reason?
I.e., maybe add
SCOPED_CLEANUP({krb5_cc_end_seq_get(g_krb5_ctx, ccache_, &cursor); });
before call to krb5_cc_start_seq_get().
http://gerrit.cloudera.org:8080/#/c/15394/4/src/kudu/integration-tests/security-itest.cc
File src/kudu/integration-tests/security-itest.cc:
http://gerrit.cloudera.org:8080/#/c/15394/4/src/kudu/integration-tests/security-itest.cc@403
PS4, Line 403: That
nit: Test that ?
http://gerrit.cloudera.org:8080/#/c/15394/4/src/kudu/integration-tests/security-itest.cc@420
PS4, Line 420: LOG(INFO) << "Truncating ccache at '" << cc_path << "' to " << trunc_len;
nit: since the test is already working as expected, to reduce chatter, consider using 'SCOPED_TRACE(message)' instead of 'LOG(INFO) << message' to output the message only when any assertion below fails.
http://gerrit.cloudera.org:8080/#/c/15394/4/src/kudu/integration-tests/security-itest.cc@421
PS4, Line 421: RWFileOptions opts;
: opts.mode = Env::MUST_EXIST;
: unique_ptr<RWFile> cc_file;
: ASSERT_OK(env_->NewRWFile(opts, cc_path, &cc_file));
: ASSERT_OK(cc_file->Truncate(trunc_len));
: cc_file->Close();
nit: move these lines into a sub-scope and remove the cc_file->Close()?
http://gerrit.cloudera.org:8080/#/c/15394/4/src/kudu/integration-tests/security-itest.cc@431
PS4, Line 431: LOG(INFO) << s.ToString();
nit: our tests are super-chatty; maybe it makes sense to drop this once the test is worked as expected given there is ASSERT_STR_CONTAINS() next line?
http://gerrit.cloudera.org:8080/#/c/15394/4/src/kudu/security/init.cc
File src/kudu/security/init.cc:
http://gerrit.cloudera.org:8080/#/c/15394/4/src/kudu/security/init.cc@244
PS4, Line 244: LOG(WARNING) << "Error " << code << " while opening credential cache: "
> I suppose I should also switch this to Substitute()
+1
--
To view, visit http://gerrit.cloudera.org:8080/15394
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2d6e06c3ea65708896a6bf0134cc84838b3f1b58
Gerrit-Change-Number: 15394
Gerrit-PatchSet: 4
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Wed, 11 Mar 2020 00:25:52 +0000
Gerrit-HasComments: Yes
[kudu-CR] WIP - KUDU-3050: recover from corrupt kerberos ccache
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/15394 )
Change subject: WIP - KUDU-3050: recover from corrupt kerberos ccache
......................................................................
Patch Set 3:
(10 comments)
http://gerrit.cloudera.org:8080/#/c/15394/3/src/kudu/mini-cluster/external_mini_cluster-test.cc
File src/kudu/mini-cluster/external_mini_cluster-test.cc:
PS3:
Maybe integration-tests/security-itest would be a better home for this new test?
http://gerrit.cloudera.org:8080/#/c/15394/3/src/kudu/mini-cluster/external_mini_cluster-test.cc@20
PS3, Line 20: #include <errno.h>
Prefer cerrno.
http://gerrit.cloudera.org:8080/#/c/15394/3/src/kudu/mini-cluster/external_mini_cluster-test.cc@296
PS3, Line 296: ASSERT_EQ(0, truncate(cc_path, 10)) << errno << " " << cc_path;
I think it'd be more elegant (albeit more lines of code) to open cc_path as an RWFile (see env.h) and call Truncate on it. You'll get a Status you can ASSERT_OK() on.
http://gerrit.cloudera.org:8080/#/c/15394/3/src/kudu/mini-cluster/external_mini_cluster-test.cc@298
PS3, Line 298: // Sleep long enough to ensure that the tserver's ticket expires so that renewal is
: // needed.
: SleepFor(MonoDelta::FromSeconds(16));
Could we lower the ticket lifetime so that this sleep could be shorter?
Alternatively, is there some way we could "sleep until ticket expiration"?
http://gerrit.cloudera.org:8080/#/c/15394/3/src/kudu/mini-cluster/external_mini_cluster-test.cc@306
PS3, Line 306: cluster.master(0)->Shutdown();
With just one master in the cluster, you can use the simpler master() accessor.
http://gerrit.cloudera.org:8080/#/c/15394/3/src/kudu/mini-cluster/external_mini_cluster-test.cc@310
PS3, Line 310: cluster.Shutdown();
Don't need to do this; it'll happen in the destructor.
http://gerrit.cloudera.org:8080/#/c/15394/3/src/kudu/security/init.cc
File src/kudu/security/init.cc:
http://gerrit.cloudera.org:8080/#/c/15394/3/src/kudu/security/init.cc@98
PS3, Line 98: RWMutex g_kerberos_reinit_lock(RWMutex::Priority::PREFER_WRITING);
Global non-POD fields are generally a bad idea because it's tough to guarantee exactly when the constructors/destructors run. That's why the existing code used a pointer here.
http://gerrit.cloudera.org:8080/#/c/15394/3/src/kudu/security/init.cc@160
PS3, Line 160: << "Renew thread sleeping after " << failure_retries << " failures for "
: << renew_interval_s << "s";
Nit: we generally use strings::Substitute() for constructing messages like these; it's easier to see the full message with substitution variables than with interspersed '<<'.
http://gerrit.cloudera.org:8080/#/c/15394/3/src/kudu/security/init.cc@220
PS3, Line 220: krb5_get_error_message
Isn't this a memory leak? Use Krb5CallToStatus to handle this safely.
Looking at your test results, seems like LSAN caught this.
http://gerrit.cloudera.org:8080/#/c/15394/3/src/kudu/security/test/mini_kdc.cc
File src/kudu/security/test/mini_kdc.cc:
http://gerrit.cloudera.org:8080/#/c/15394/3/src/kudu/security/test/mini_kdc.cc@301
PS3, Line 301: kt_path = JoinPathSegments(options_.data_root, kt_path) + ".keytab";
: return kt_path;
Nit: could combine.
--
To view, visit http://gerrit.cloudera.org:8080/15394
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2d6e06c3ea65708896a6bf0134cc84838b3f1b58
Gerrit-Change-Number: 15394
Gerrit-PatchSet: 3
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Comment-Date: Tue, 10 Mar 2020 06:42:19 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-3050: recover from corrupt kerberos ccache
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Alexey Serbin, Kudu Jenkins, Adar Dembo,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15394
to look at the new patch set (#6).
Change subject: KUDU-3050: recover from corrupt kerberos ccache
......................................................................
KUDU-3050: recover from corrupt kerberos ccache
This handles two failure modes:
* krb5_cc_start_seq_get() can fail if the kerberos credential cache gets
corrupted on disk, e.g. is truncated.
* the renewal can fail to find a credential in the credential cache,
either if it is missing or the renewal thread hits an error while
reading through credentials.
Also add some additional logging and limit the max backoff time
to make it easier to debug other kinds of renewal errors.
The test triggers a pre-existing memory leak bug in some older
Kerberos libraries. Added a suppression for leak sanitizer
to ClientNegotiation::CheckGSSAPI() to suppress it.
Test:
Add a test that exercises the recovery logic after truncating
the credential cache. The test failed before this change.
Change-Id: I2d6e06c3ea65708896a6bf0134cc84838b3f1b58
---
M src/kudu/integration-tests/security-itest.cc
M src/kudu/rpc/client_negotiation.cc
M src/kudu/security/init.cc
A src/kudu/security/kinit_context.h
M src/kudu/security/test/mini_kdc.cc
M src/kudu/security/test/mini_kdc.h
6 files changed, 200 insertions(+), 62 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/94/15394/6
--
To view, visit http://gerrit.cloudera.org:8080/15394
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2d6e06c3ea65708896a6bf0134cc84838b3f1b58
Gerrit-Change-Number: 15394
Gerrit-PatchSet: 6
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[kudu-CR] KUDU-3050: recover from corrupt kerberos ccache
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15394 )
Change subject: KUDU-3050: recover from corrupt kerberos ccache
......................................................................
Patch Set 4:
(8 comments)
Thanks for your help with the memory leak. With Adar's help I was able to get a full backtrace and I figured out that it's actually a known bug in MIT Kerberos that our versions don't have the fix for. I added a suppression to the function where the leak occurs.
http://gerrit.cloudera.org:8080/#/c/15394/4/src/kudu/integration-tests/security-itest.cc
File src/kudu/integration-tests/security-itest.cc:
http://gerrit.cloudera.org:8080/#/c/15394/4/src/kudu/integration-tests/security-itest.cc@403
PS4, Line 403: That
> nit: Test that ?
Done
http://gerrit.cloudera.org:8080/#/c/15394/4/src/kudu/integration-tests/security-itest.cc@410
PS4, Line 410: // KinitContext is generally a global singleton so does not clean up memory. Just leak
: // it for this test.
> Not needed anymore?
Done
http://gerrit.cloudera.org:8080/#/c/15394/4/src/kudu/integration-tests/security-itest.cc@420
PS4, Line 420: LOG(INFO) << "Truncating ccache at '" << cc_path << "' to " << trunc_len;
> nit: since the test is already working as expected, to reduce chatter, cons
Done
http://gerrit.cloudera.org:8080/#/c/15394/4/src/kudu/integration-tests/security-itest.cc@421
PS4, Line 421: RWFileOptions opts;
: opts.mode = Env::MUST_EXIST;
: unique_ptr<RWFile> cc_file;
: ASSERT_OK(env_->NewRWFile(opts, cc_path, &cc_file));
: ASSERT_OK(cc_file->Truncate(trunc_len));
: cc_file->Close();
> nit: move these lines into a sub-scope and remove the cc_file->Close()?
Done
http://gerrit.cloudera.org:8080/#/c/15394/4/src/kudu/integration-tests/security-itest.cc@431
PS4, Line 431: LOG(INFO) << s.ToString();
> nit: our tests are super-chatty; maybe it makes sense to drop this once the
Done
http://gerrit.cloudera.org:8080/#/c/15394/4/src/kudu/security/init.cc
File src/kudu/security/init.cc:
http://gerrit.cloudera.org:8080/#/c/15394/4/src/kudu/security/init.cc@181
PS4, Line 181: // Free memory associated with these objects.
> Will these functions no-op if the values are null? If Kinit() fails mid-way
One problem with this is that we don't initialise the members (they're all pointers). So I added null initialization.
I also explicitly null check below since the kerberos functions don't explicitly document what they do if they are passed a null argument.
I looked at the implementations of the functions called in KinitInternal() and it looks like, on an error, they all either set the output to NULL or leave it unmodified.
http://gerrit.cloudera.org:8080/#/c/15394/4/src/kudu/security/init.cc@245
PS4, Line 245: << Krb5CallToStatus(g_krb5_ctx, code).ToString();
> Could also wrap krb5_cc_start_seq_get with Krb5CallToStatus and then do the
Done. I also added some info about the credential cache.
http://gerrit.cloudera.org:8080/#/c/15394/4/src/kudu/security/init.cc@299
PS4, Line 299: LOG(WARNING) << "Could not find kerberos principal in credential cache";
> Can we add more useful information to this warning? Such as the location of
Done
--
To view, visit http://gerrit.cloudera.org:8080/15394
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2d6e06c3ea65708896a6bf0134cc84838b3f1b58
Gerrit-Change-Number: 15394
Gerrit-PatchSet: 4
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Wed, 11 Mar 2020 01:04:57 +0000
Gerrit-HasComments: Yes
[kudu-CR] WIP - KUDU-3050: recover from corrupt kerberos ccache
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15394
to look at the new patch set (#2).
Change subject: WIP - KUDU-3050: recover from corrupt kerberos ccache
......................................................................
WIP - KUDU-3050: recover from corrupt kerberos ccache
Also add some additional logging and limit the max backoff time
to make it easier to debug other kinds of renewal errors.
Test:
Add a test that exercises the recovery logic after truncating
the credential cache. The test failed before this change.
Change-Id: I2d6e06c3ea65708896a6bf0134cc84838b3f1b58
---
M src/kudu/mini-cluster/external_mini_cluster-test.cc
M src/kudu/security/init.cc
A src/kudu/security/kinit_context.h
M src/kudu/security/test/mini_kdc.cc
M src/kudu/security/test/mini_kdc.h
5 files changed, 161 insertions(+), 66 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/94/15394/2
--
To view, visit http://gerrit.cloudera.org:8080/15394
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2d6e06c3ea65708896a6bf0134cc84838b3f1b58
Gerrit-Change-Number: 15394
Gerrit-PatchSet: 2
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
[kudu-CR] KUDU-3050: recover from corrupt kerberos ccache
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15394 )
Change subject: KUDU-3050: recover from corrupt kerberos ccache
......................................................................
Patch Set 4:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/15394/4/src/kudu/security/init.cc
File src/kudu/security/init.cc:
http://gerrit.cloudera.org:8080/#/c/15394/4/src/kudu/security/init.cc@244
PS4, Line 244: LOG(WARNING) << "Error " << code << " while opening credential cache: "
I suppose I should also switch this to Substitute()
--
To view, visit http://gerrit.cloudera.org:8080/15394
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2d6e06c3ea65708896a6bf0134cc84838b3f1b58
Gerrit-Change-Number: 15394
Gerrit-PatchSet: 4
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 10 Mar 2020 21:23:45 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-3050: recover from corrupt kerberos ccache
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/15394 )
Change subject: KUDU-3050: recover from corrupt kerberos ccache
......................................................................
Patch Set 5: Code-Review+2
(1 comment)
http://gerrit.cloudera.org:8080/#/c/15394/5/src/kudu/security/test/mini_kdc.h
File src/kudu/security/test/mini_kdc.h:
http://gerrit.cloudera.org:8080/#/c/15394/5/src/kudu/security/test/mini_kdc.h@104
PS5, Line 104: std::string GetKeytabPathForPrincipal(const std::string& spn);
nit: add const specifier for the method?
--
To view, visit http://gerrit.cloudera.org:8080/15394
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2d6e06c3ea65708896a6bf0134cc84838b3f1b58
Gerrit-Change-Number: 15394
Gerrit-PatchSet: 5
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Wed, 11 Mar 2020 04:31:25 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-3050: recover from corrupt kerberos ccache
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15394 )
Change subject: KUDU-3050: recover from corrupt kerberos ccache
......................................................................
Patch Set 4:
(10 comments)
Thanks for the feedback, I reworked a lot of this in response to comments.
I also found and fixed some issues with the recovery logic - I didn't consider the case where it open the file OK but then didn't find the right credential.
http://gerrit.cloudera.org:8080/#/c/15394/3/src/kudu/mini-cluster/external_mini_cluster-test.cc
File src/kudu/mini-cluster/external_mini_cluster-test.cc:
PS3:
> Maybe integration-tests/security-itest would be a better home for this new
That's a good suggestion. Moved it there and rewrote it, it got a lot simpler and faster.
http://gerrit.cloudera.org:8080/#/c/15394/3/src/kudu/mini-cluster/external_mini_cluster-test.cc@20
PS3, Line 20: #include <iosfwd>
> Prefer cerrno.
Done
http://gerrit.cloudera.org:8080/#/c/15394/3/src/kudu/mini-cluster/external_mini_cluster-test.cc@296
PS3, Line 296: opts.master_rpc_addresses = master_rpc_addresses;
> I think it'd be more elegant (albeit more lines of code) to open cc_path as
Done
http://gerrit.cloudera.org:8080/#/c/15394/3/src/kudu/mini-cluster/external_mini_cluster-test.cc@298
PS3, Line 298: NO_FATALS(SmokeExternalMiniCluster(opts, cluster.get(), &master_rpc_addresses));
: ASSERT_EQ(opts.master_rpc_addresses, master_rpc_addresses);
:
> Could we lower the ticket lifetime so that this sleep could be shorter?
After the move of the test to an integration test this is no longer required.
http://gerrit.cloudera.org:8080/#/c/15394/3/src/kudu/mini-cluster/external_mini_cluster-test.cc@306
PS3, Line 306: cluster->Shutdown();
> With just one master in the cluster, you can use the simpler master() acces
Done
http://gerrit.cloudera.org:8080/#/c/15394/3/src/kudu/mini-cluster/external_mini_cluster-test.cc@310
PS3, Line 310: } // namespace kudu
> Don't need to do this; it'll happen in the destructor.
Done
http://gerrit.cloudera.org:8080/#/c/15394/3/src/kudu/security/init.cc
File src/kudu/security/init.cc:
http://gerrit.cloudera.org:8080/#/c/15394/3/src/kudu/security/init.cc@98
PS3, Line 98: RWMutex* g_kerberos_reinit_lock;
> Global non-POD fields are generally a bad idea because it's tough to guaran
I ran into the problem that the test doesn't call InitKerberosForServer() but does call methods that use the lock.
I added this to InitKrb5Ctx(). I could also expose a different method to init the lock, but that seemed ugly.
http://gerrit.cloudera.org:8080/#/c/15394/3/src/kudu/security/init.cc@160
PS3, Line 160: retries > 0) {
: // Log in the abnormal case where som
> Nit: we generally use strings::Substitute() for constructing messages like
Done
http://gerrit.cloudera.org:8080/#/c/15394/3/src/kudu/security/init.cc@220
PS3, Line 220: ibution<> dist(0.5, 1.
> Isn't this a memory leak? Use Krb5CallToStatus to handle this safely.
Done
http://gerrit.cloudera.org:8080/#/c/15394/3/src/kudu/security/test/mini_kdc.cc
File src/kudu/security/test/mini_kdc.cc:
http://gerrit.cloudera.org:8080/#/c/15394/3/src/kudu/security/test/mini_kdc.cc@301
PS3, Line 301: return JoinPathSegments(options_.data_root, kt_path) + ".keytab";
: }
> Nit: could combine.
Done
--
To view, visit http://gerrit.cloudera.org:8080/15394
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2d6e06c3ea65708896a6bf0134cc84838b3f1b58
Gerrit-Change-Number: 15394
Gerrit-PatchSet: 4
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 10 Mar 2020 21:21:01 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-3050: recover from corrupt kerberos ccache
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/15394 )
Change subject: KUDU-3050: recover from corrupt kerberos ccache
......................................................................
Patch Set 6: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/15394
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2d6e06c3ea65708896a6bf0134cc84838b3f1b58
Gerrit-Change-Number: 15394
Gerrit-PatchSet: 6
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Wed, 11 Mar 2020 16:10:13 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-3050: recover from corrupt kerberos ccache
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/15394 )
Change subject: KUDU-3050: recover from corrupt kerberos ccache
......................................................................
Patch Set 4:
(4 comments)
Looks good overall; Alexey should also take a look.
http://gerrit.cloudera.org:8080/#/c/15394/4/src/kudu/integration-tests/security-itest.cc
File src/kudu/integration-tests/security-itest.cc:
http://gerrit.cloudera.org:8080/#/c/15394/4/src/kudu/integration-tests/security-itest.cc@410
PS4, Line 410: // KinitContext is generally a global singleton so does not clean up memory. Just leak
: // it for this test.
Not needed anymore?
http://gerrit.cloudera.org:8080/#/c/15394/4/src/kudu/security/init.cc
File src/kudu/security/init.cc:
http://gerrit.cloudera.org:8080/#/c/15394/4/src/kudu/security/init.cc@181
PS4, Line 181: // Free memory associated with these objects.
Will these functions no-op if the values are null? If Kinit() fails mid-way, will the remaining values be null, or garbage?
http://gerrit.cloudera.org:8080/#/c/15394/4/src/kudu/security/init.cc@245
PS4, Line 245: << Krb5CallToStatus(g_krb5_ctx, code).ToString();
Could also wrap krb5_cc_start_seq_get with Krb5CallToStatus and then do the error check via the more idiomatic "if !status.ok()". Not a big deal either way.
http://gerrit.cloudera.org:8080/#/c/15394/4/src/kudu/security/init.cc@299
PS4, Line 299: LOG(WARNING) << "Could not find kerberos principal in credential cache";
Can we add more useful information to this warning? Such as the location of the cache, if it's on disk?
--
To view, visit http://gerrit.cloudera.org:8080/15394
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2d6e06c3ea65708896a6bf0134cc84838b3f1b58
Gerrit-Change-Number: 15394
Gerrit-PatchSet: 4
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 10 Mar 2020 21:35:07 +0000
Gerrit-HasComments: Yes
[kudu-CR] WIP - KUDU-3050: recover from corrupt kerberos ccache
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15394
to look at the new patch set (#3).
Change subject: WIP - KUDU-3050: recover from corrupt kerberos ccache
......................................................................
WIP - KUDU-3050: recover from corrupt kerberos ccache
Also add some additional logging and limit the max backoff time
to make it easier to debug other kinds of renewal errors.
Test:
Add a test that exercises the recovery logic after truncating
the credential cache. The test failed before this change.
Change-Id: I2d6e06c3ea65708896a6bf0134cc84838b3f1b58
---
M src/kudu/mini-cluster/external_mini_cluster-test.cc
M src/kudu/security/init.cc
A src/kudu/security/kinit_context.h
M src/kudu/security/test/mini_kdc.cc
M src/kudu/security/test/mini_kdc.h
5 files changed, 168 insertions(+), 66 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/94/15394/3
--
To view, visit http://gerrit.cloudera.org:8080/15394
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2d6e06c3ea65708896a6bf0134cc84838b3f1b58
Gerrit-Change-Number: 15394
Gerrit-PatchSet: 3
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
[kudu-CR] KUDU-3050: recover from corrupt kerberos ccache
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/15394 )
Change subject: KUDU-3050: recover from corrupt kerberos ccache
......................................................................
Patch Set 4:
> Patch Set 4:
>
> Might be https://krbdev.mit.edu/rt/Ticket/Display.html?id=8437 or similar. I don't see any evidence that Kudu is misusing the APIs. It looks like the memory is allocated in either krb5_cc_resolve() or krb5_cc_dup(), by tracing back from fcc_resolve.
Try running the test with ASAN_OPTIONS=fast_unwind_on_malloc=0 in the environment. That should give you a more complete stack trace.
--
To view, visit http://gerrit.cloudera.org:8080/15394
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2d6e06c3ea65708896a6bf0134cc84838b3f1b58
Gerrit-Change-Number: 15394
Gerrit-PatchSet: 4
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 10 Mar 2020 22:57:50 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-3050: recover from corrupt kerberos ccache
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15394 )
Change subject: KUDU-3050: recover from corrupt kerberos ccache
......................................................................
KUDU-3050: recover from corrupt kerberos ccache
This handles two failure modes:
* krb5_cc_start_seq_get() can fail if the kerberos credential cache gets
corrupted on disk, e.g. is truncated.
* the renewal can fail to find a credential in the credential cache,
either if it is missing or the renewal thread hits an error while
reading through credentials.
Also add some additional logging and limit the max backoff time
to make it easier to debug other kinds of renewal errors.
The test triggers a pre-existing memory leak bug in some older
Kerberos libraries. Added a suppression for leak sanitizer
to ClientNegotiation::CheckGSSAPI() to suppress it.
Test:
Add a test that exercises the recovery logic after truncating
the credential cache. The test failed before this change.
Change-Id: I2d6e06c3ea65708896a6bf0134cc84838b3f1b58
---
M src/kudu/integration-tests/security-itest.cc
M src/kudu/rpc/client_negotiation.cc
M src/kudu/security/init.cc
A src/kudu/security/kinit_context.h
M src/kudu/security/test/mini_kdc.cc
M src/kudu/security/test/mini_kdc.h
6 files changed, 200 insertions(+), 62 deletions(-)
Approvals:
Adar Dembo: Looks good to me, approved
Kudu Jenkins: Verified
--
To view, visit http://gerrit.cloudera.org:8080/15394
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I2d6e06c3ea65708896a6bf0134cc84838b3f1b58
Gerrit-Change-Number: 15394
Gerrit-PatchSet: 6
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[kudu-CR] KUDU-3050: recover from corrupt kerberos ccache
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Hello Tidy Bot, Kudu Jenkins, Adar Dembo,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15394
to look at the new patch set (#4).
Change subject: KUDU-3050: recover from corrupt kerberos ccache
......................................................................
KUDU-3050: recover from corrupt kerberos ccache
This handles two failure modes:
* krb5_cc_start_seq_get() can fail if the kerberos credential cache gets
corrupted on disk, e.g. is truncated.
* the renewal can fail to find a credential in the credential cache,
either if it is missing or the renewal thread hits an error while
reading through credentials.
Also add some additional logging and limit the max backoff time
to make it easier to debug other kinds of renewal errors.
Test:
Add a test that exercises the recovery logic after truncating
the credential cache. The test failed before this change.
Change-Id: I2d6e06c3ea65708896a6bf0134cc84838b3f1b58
---
M src/kudu/integration-tests/security-itest.cc
M src/kudu/security/init.cc
A src/kudu/security/kinit_context.h
M src/kudu/security/test/mini_kdc.cc
M src/kudu/security/test/mini_kdc.h
5 files changed, 191 insertions(+), 62 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/94/15394/4
--
To view, visit http://gerrit.cloudera.org:8080/15394
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2d6e06c3ea65708896a6bf0134cc84838b3f1b58
Gerrit-Change-Number: 15394
Gerrit-PatchSet: 4
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
[kudu-CR] KUDU-3050: recover from corrupt kerberos ccache
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/15394 )
Change subject: KUDU-3050: recover from corrupt kerberos ccache
......................................................................
Patch Set 4:
> (5 comments)
>
> > > Patch Set 4:
> > >
> > > Might be https://krbdev.mit.edu/rt/Ticket/Display.html?id=8437
> or
> > similar. I don't see any evidence that Kudu is misusing the APIs.
> > It looks like the memory is allocated in either krb5_cc_resolve()
> > or krb5_cc_dup(), by tracing back from fcc_resolve.
> >
> > Try running the test with ASAN_OPTIONS=fast_unwind_on_malloc=0 in
> > the environment. That should give you a more complete stack
> trace.
>
> Should we call krb5_cc_end_seq_get() even if prior call to
> krb5_cc_start_seq_get() returned an error? Maybe, that's the
> reason?
>
> I.e., maybe add
> SCOPED_CLEANUP({krb5_cc_end_seq_get(g_krb5_ctx, ccache_, &cursor);
> });
>
> before call to krb5_cc_start_seq_get().
Ah, most likely not the case.
--
To view, visit http://gerrit.cloudera.org:8080/15394
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2d6e06c3ea65708896a6bf0134cc84838b3f1b58
Gerrit-Change-Number: 15394
Gerrit-PatchSet: 4
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Wed, 11 Mar 2020 00:32:33 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-3050: recover from corrupt kerberos ccache
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15394 )
Change subject: KUDU-3050: recover from corrupt kerberos ccache
......................................................................
Patch Set 4:
After installing libkrb5-dbg i get Direct leak of 24 byte(s) in 1 object(s) allocated from:
#0 0x4f7d22 in malloc /home/tarmstrong/repos/kudu/thirdparty/src/llvm-9.0.0.src/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:145
#1 0x7fc7243dc122 in fcc_resolve ../../../src/lib/krb5/ccache/cc_file.c:1021
will continue digging
--
To view, visit http://gerrit.cloudera.org:8080/15394
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2d6e06c3ea65708896a6bf0134cc84838b3f1b58
Gerrit-Change-Number: 15394
Gerrit-PatchSet: 4
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 10 Mar 2020 22:09:28 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-3050: recover from corrupt kerberos ccache
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15394 )
Change subject: KUDU-3050: recover from corrupt kerberos ccache
......................................................................
Patch Set 4:
I'm still hitting a LSAN error that I need to debug. It seems to go away if I comment out the failing call to TrySetFlagOnTS(). Going to see if I can get a better backtrace somehow.
=================================================================
==14191==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 24 byte(s) in 1 object(s) allocated from:
#0 0x514b01 in malloc /home/jenkins-slave/workspace/kudu-master/0/thirdparty/src/llvm-9.0.0.src/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:145
#1 0x7fddd01fddd2 (/usr/lib/x86_64-linux-gnu/libkrb5.so.3+0x29dd2)
SUMMARY: AddressSanitizer: 24 byte(s) leaked in 1 allocation(s).
--
To view, visit http://gerrit.cloudera.org:8080/15394
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2d6e06c3ea65708896a6bf0134cc84838b3f1b58
Gerrit-Change-Number: 15394
Gerrit-PatchSet: 4
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 10 Mar 2020 22:04:02 +0000
Gerrit-HasComments: No