You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Will Berkeley (Code Review)" <ge...@cloudera.org> on 2019/05/28 23:01:13 UTC
[kudu-CR] [java] Attempt to deflake SecureKuduSinkTest
Will Berkeley has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13454
Change subject: [java] Attempt to deflake SecureKuduSinkTest
......................................................................
[java] Attempt to deflake SecureKuduSinkTest
Occasionally, SecureKuduSinkTest fails when running with TSAN binaries
because of the following sequence of operations:
1. The Kerberos ticket lifetime is set to 10s.
2. The test sets up a mini kudu cluster. This first sets up the KDC,
which creates credentials for all of the Kudu servers and kinits
using test user credentials for the test process.
3. The setup of the cluster takes > 10s.
4. At the end of the cluster setup, the test checks that setup succeeded
in part by issuing a ListTabletServers RPC. This fails because the test
user ticket has expired.
5. The test fails because it can't set up the cluster.
The failure looks like
21:50:06.500 [ERROR - main] (RetryRule.java:217) org.apache.kudu.flume.sink.SecureKuduSinkTest.testEventsWithShortTickets: failed attempt 1
java.io.IOException: ListTabletServers RPC failed: Client connection negotiation failed: client connection to 127.12.111.60:36425: server requires authentication, but client does not have Kerberos credentials available
at org.apache.kudu.test.cluster.MiniKuduCluster.sendRequestToCluster(MiniKuduCluster.java:169) ~[kudu-test-utils-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.apache.kudu.test.cluster.MiniKuduCluster.start(MiniKuduCluster.java:234) ~[kudu-test-utils-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.apache.kudu.test.cluster.MiniKuduCluster.access$300(MiniKuduCluster.java:71) ~[kudu-test-utils-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.apache.kudu.test.cluster.MiniKuduCluster$MiniKuduClusterBuilder.build(MiniKuduCluster.java:658) ~[kudu-test-utils-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.apache.kudu.test.KuduTestHarness.before(KuduTestHarness.java:140) ~[kudu-test-utils-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:46) ~[junit-4.12.jar:4.12]
at org.apache.kudu.test.junit.RetryRule$RetryStatement.doOneAttempt(RetryRule.java:215) [kudu-test-utils-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.apache.kudu.test.junit.RetryRule$RetryStatement.evaluate(RetryRule.java:232) [kudu-test-utils-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
...
This patch attempts to deflake the test a bit by doubling the ticket
lifetime to 20s. It also raises the renewal lifetime to 35s from 30s,
to provide a bit of extra time between the ticket expiring and when
Flume needs to renew the ticket. Since the test waits 2x the renewal
time, this adds 10s to the running time of the test.
I tried to test this on dist-test using TSAN binaries. With the new
patch I saw 0/1000 failures, but without it I saw 830/1000 failures.
That's *way* flakier than any previous indication, so I don't trust
those results. The failures I sampled did seem to be related to the same
issue, but it was ConnectToCluster RPCs failing instead.
Change-Id: Icc936878d7f1496905e83ddaf93b9b049f417f72
---
M java/kudu-flume-sink/src/test/java/org/apache/kudu/flume/sink/SecureKuduSinkTest.java
1 file changed, 2 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/54/13454/1
--
To view, visit http://gerrit.cloudera.org:8080/13454
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Icc936878d7f1496905e83ddaf93b9b049f417f72
Gerrit-Change-Number: 13454
Gerrit-PatchSet: 1
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
[kudu-CR] [java] Attempt to deflake SecureKuduSinkTest
Posted by "Will Berkeley (Code Review)" <ge...@cloudera.org>.
Will Berkeley has removed a vote on this change.
Change subject: [java] Attempt to deflake SecureKuduSinkTest
......................................................................
Removed Verified-1 by Kudu Jenkins (120)
--
To view, visit http://gerrit.cloudera.org:8080/13454
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: Icc936878d7f1496905e83ddaf93b9b049f417f72
Gerrit-Change-Number: 13454
Gerrit-PatchSet: 1
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
[kudu-CR] [java] Attempt to deflake SecureKuduSinkTest
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/13454 )
Change subject: [java] Attempt to deflake SecureKuduSinkTest
......................................................................
Patch Set 2: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/13454
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icc936878d7f1496905e83ddaf93b9b049f417f72
Gerrit-Change-Number: 13454
Gerrit-PatchSet: 2
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Thu, 30 May 2019 00:18:47 +0000
Gerrit-HasComments: No
[kudu-CR] [java] Attempt to deflake SecureKuduSinkTest
Posted by "Will Berkeley (Code Review)" <ge...@cloudera.org>.
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/13454 )
Change subject: [java] Attempt to deflake SecureKuduSinkTest
......................................................................
Patch Set 1: Verified+1
Unrelated flake in DeleteTabletITest
--
To view, visit http://gerrit.cloudera.org:8080/13454
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icc936878d7f1496905e83ddaf93b9b049f417f72
Gerrit-Change-Number: 13454
Gerrit-PatchSet: 1
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Wed, 29 May 2019 17:21:00 +0000
Gerrit-HasComments: No
[kudu-CR] [java] Attempt to deflake SecureKuduSinkTest
Posted by "Will Berkeley (Code Review)" <ge...@cloudera.org>.
Hello Alexey Serbin, Kudu Jenkins, Adar Dembo,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/13454
to look at the new patch set (#2).
Change subject: [java] Attempt to deflake SecureKuduSinkTest
......................................................................
[java] Attempt to deflake SecureKuduSinkTest
Occasionally, SecureKuduSinkTest fails when running with TSAN binaries
because of the following sequence of operations:
1. The Kerberos ticket lifetime is set to 10s.
2. The test sets up a mini kudu cluster. This first sets up the KDC,
which creates credentials for all of the Kudu servers and kinits
using test user credentials for the test process.
3. The setup of the cluster takes > 10s.
4. At the end of the cluster setup, the test checks that setup succeeded
in part by issuing a ListTabletServers RPC. This fails because the test
user ticket has expired.
5. The test fails because it can't set up the cluster.
The failure looks like
21:50:06.500 [ERROR - main] (RetryRule.java:217) org.apache.kudu.flume.sink.SecureKuduSinkTest.testEventsWithShortTickets: failed attempt 1
java.io.IOException: ListTabletServers RPC failed: Client connection negotiation failed: client connection to 127.12.111.60:36425: server requires authentication, but client does not have Kerberos credentials available
at org.apache.kudu.test.cluster.MiniKuduCluster.sendRequestToCluster(MiniKuduCluster.java:169) ~[kudu-test-utils-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.apache.kudu.test.cluster.MiniKuduCluster.start(MiniKuduCluster.java:234) ~[kudu-test-utils-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.apache.kudu.test.cluster.MiniKuduCluster.access$300(MiniKuduCluster.java:71) ~[kudu-test-utils-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.apache.kudu.test.cluster.MiniKuduCluster$MiniKuduClusterBuilder.build(MiniKuduCluster.java:658) ~[kudu-test-utils-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.apache.kudu.test.KuduTestHarness.before(KuduTestHarness.java:140) ~[kudu-test-utils-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:46) ~[junit-4.12.jar:4.12]
at org.apache.kudu.test.junit.RetryRule$RetryStatement.doOneAttempt(RetryRule.java:215) [kudu-test-utils-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.apache.kudu.test.junit.RetryRule$RetryStatement.evaluate(RetryRule.java:232) [kudu-test-utils-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
...
This patch attempts to deflake the test a bit by doubling the ticket
lifetime to 20s. It also raises the renewal lifetime to 35s from 30s,
to provide a bit of extra time between the ticket expiring and when
Flume needs to renew the ticket. Before, the test waited 2x the
renewable ticket lifetime. I made it so the test waits until the
renewable ticket lifetime plus one second has passed, including the
time spent in the test so far.
I tried to test this on dist-test using TSAN binaries. With the new
patch I saw 0/1000 failures, but without it I saw 830/1000 failures.
That's *way* flakier than any previous indication, so I don't trust
those results. The failures I sampled did seem to be related to the same
issue, but it was ConnectToCluster RPCs failing instead.
Change-Id: Icc936878d7f1496905e83ddaf93b9b049f417f72
---
M java/kudu-flume-sink/src/test/java/org/apache/kudu/flume/sink/SecureKuduSinkTest.java
1 file changed, 9 insertions(+), 3 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/54/13454/2
--
To view, visit http://gerrit.cloudera.org:8080/13454
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Icc936878d7f1496905e83ddaf93b9b049f417f72
Gerrit-Change-Number: 13454
Gerrit-PatchSet: 2
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
[kudu-CR] [java] Attempt to deflake SecureKuduSinkTest
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/13454 )
Change subject: [java] Attempt to deflake SecureKuduSinkTest
......................................................................
Patch Set 1:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/13454/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/13454/1//COMMIT_MSG@25
PS1, Line 25: java.io.IOException: ListTabletServers RPC failed: Client connection negotiation failed: client connection to 127.12.111.60:36425: server requires authentication, but client does not have Kerberos credentials available
> I recall looking at a handful of these failures and marveling at the error
As I understand, the client didn't use its kerberos creds during negotiation because kerberos tgt has expired, but for some reason client decided to go with some other authn mechanism.
I would expect an error to be put into errorsByMech in java/kudu-client/src/main/java/org/apache/kudu/client/Negotiator.java:428 "client Kerberos credentials (TGT) have expired". However, that didn't happen.
If my reading is correct, that might be a bug.
Is it possible to enable debug logs and collect information reported at java/kudu-client/src/main/java/org/apache/kudu/client/Negotiator.java:460 when the test fails? Maybe, the logs already contain that info? It's something like LOG.debug("SASL mechanism {} chosen for peer {}", chosenMech.name(), remoteHostname);
http://gerrit.cloudera.org:8080/#/c/13454/1/java/kudu-flume-sink/src/test/java/org/apache/kudu/flume/sink/SecureKuduSinkTest.java
File java/kudu-flume-sink/src/test/java/org/apache/kudu/flume/sink/SecureKuduSinkTest.java:
http://gerrit.cloudera.org:8080/#/c/13454/1/java/kudu-flume-sink/src/test/java/org/apache/kudu/flume/sink/SecureKuduSinkTest.java@90
PS1, Line 90: LOG.info("Waiting for tickets to expire");
: TimeUnit.SECONDS.sleep(RENEWABLE_LIFETIME_SECONDS * 2);
> Isn't this way too conservative? If we took the time at the start of the te
If my understanding is correct, this seems too conservative, indeed.
--
To view, visit http://gerrit.cloudera.org:8080/13454
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icc936878d7f1496905e83ddaf93b9b049f417f72
Gerrit-Change-Number: 13454
Gerrit-PatchSet: 1
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Wed, 29 May 2019 18:26:08 +0000
Gerrit-HasComments: Yes
[kudu-CR] [java] Attempt to deflake SecureKuduSinkTest
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/13454 )
Change subject: [java] Attempt to deflake SecureKuduSinkTest
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/13454/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/13454/1//COMMIT_MSG@25
PS1, Line 25: java.io.IOException: ListTabletServers RPC failed: Client connection negotiation failed: client connection to 127.12.111.60:36425: server requires authentication, but client does not have Kerberos credentials available
> As I understand, the client didn't use its kerberos creds during negotiatio
Ah, ignore that: here we should be talking only C++ client/server negotiations, no Java involved.
I think the following happened: the code in src/kudu/rpc/client_negotiation.cc:438 removed the GSSAPI mechanism for SASL authn because CheckGSSAPI() returned non-OK due to expired TGT. However, I don't understand why the trace of failed negotiation is not output by the client side.
I think we can do as Adar suggested: in case if GSSAPI SASL mech was removed from available ones, add the error messages from the status returned by CheckGSSAPI into Status::NotAuthorized() message returned from ClientNegotiation::HandleNegotiate()
--
To view, visit http://gerrit.cloudera.org:8080/13454
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icc936878d7f1496905e83ddaf93b9b049f417f72
Gerrit-Change-Number: 13454
Gerrit-PatchSet: 1
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Wed, 29 May 2019 20:39:56 +0000
Gerrit-HasComments: Yes
[kudu-CR] [java] Attempt to deflake SecureKuduSinkTest
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/13454 )
Change subject: [java] Attempt to deflake SecureKuduSinkTest
......................................................................
Patch Set 2: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/13454
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icc936878d7f1496905e83ddaf93b9b049f417f72
Gerrit-Change-Number: 13454
Gerrit-PatchSet: 2
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Thu, 30 May 2019 02:36:47 +0000
Gerrit-HasComments: No
[kudu-CR] [java] Attempt to deflake SecureKuduSinkTest
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/13454 )
Change subject: [java] Attempt to deflake SecureKuduSinkTest
......................................................................
Patch Set 1:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/13454/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/13454/1//COMMIT_MSG@25
PS1, Line 25: java.io.IOException: ListTabletServers RPC failed: Client connection negotiation failed: client connection to 127.12.111.60:36425: server requires authentication, but client does not have Kerberos credentials available
I recall looking at a handful of these failures and marveling at the error message. Now that I understand what actually happened, is there any way we can improve it to explain that the client's credentials expired?
http://gerrit.cloudera.org:8080/#/c/13454/1/java/kudu-flume-sink/src/test/java/org/apache/kudu/flume/sink/SecureKuduSinkTest.java
File java/kudu-flume-sink/src/test/java/org/apache/kudu/flume/sink/SecureKuduSinkTest.java:
http://gerrit.cloudera.org:8080/#/c/13454/1/java/kudu-flume-sink/src/test/java/org/apache/kudu/flume/sink/SecureKuduSinkTest.java@90
PS1, Line 90: LOG.info("Waiting for tickets to expire");
: TimeUnit.SECONDS.sleep(RENEWABLE_LIFETIME_SECONDS * 2);
Isn't this way too conservative? If we took the time at the start of the test (i.e. after the tickets were created), took it again here, then waited RENEWABLE_LIFETIME_SECONDS - (end - start), wouldn't that suffice? With maybe a + 1 thrown in there?
It's not purely academic either: if we waited less time here, we could be more aggressive about increasing the lifetimes and warding off future flakiness if startup slows down further (i.e. if we enable HMS or Sentry integration by default).
--
To view, visit http://gerrit.cloudera.org:8080/13454
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icc936878d7f1496905e83ddaf93b9b049f417f72
Gerrit-Change-Number: 13454
Gerrit-PatchSet: 1
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Wed, 29 May 2019 17:25:17 +0000
Gerrit-HasComments: Yes
[kudu-CR] [java] Attempt to deflake SecureKuduSinkTest
Posted by "Will Berkeley (Code Review)" <ge...@cloudera.org>.
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/13454 )
Change subject: [java] Attempt to deflake SecureKuduSinkTest
......................................................................
Patch Set 1:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/13454/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/13454/1//COMMIT_MSG@25
PS1, Line 25: java.io.IOException: ListTabletServers RPC failed: Client connection negotiation failed: client connection to 127.12.111.60:36425: server requires authentication, but client does not have Kerberos credentials available
> Ah, ignore that: here we should be talking only C++ client/server negotiati
Alexey said he'd address this C++ logging improvement in another patch.
http://gerrit.cloudera.org:8080/#/c/13454/1/java/kudu-flume-sink/src/test/java/org/apache/kudu/flume/sink/SecureKuduSinkTest.java
File java/kudu-flume-sink/src/test/java/org/apache/kudu/flume/sink/SecureKuduSinkTest.java:
http://gerrit.cloudera.org:8080/#/c/13454/1/java/kudu-flume-sink/src/test/java/org/apache/kudu/flume/sink/SecureKuduSinkTest.java@90
PS1, Line 90: LOG.info("Waiting for tickets to expire");
: TimeUnit.SECONDS.sleep(RENEWABLE_LIFETIME_SECONDS * 2);
> If my understanding is correct, this seems too conservative, indeed.
Done
--
To view, visit http://gerrit.cloudera.org:8080/13454
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icc936878d7f1496905e83ddaf93b9b049f417f72
Gerrit-Change-Number: 13454
Gerrit-PatchSet: 1
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Thu, 30 May 2019 00:14:13 +0000
Gerrit-HasComments: Yes
[kudu-CR] [java] Attempt to deflake SecureKuduSinkTest
Posted by "Will Berkeley (Code Review)" <ge...@cloudera.org>.
Will Berkeley has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13454 )
Change subject: [java] Attempt to deflake SecureKuduSinkTest
......................................................................
[java] Attempt to deflake SecureKuduSinkTest
Occasionally, SecureKuduSinkTest fails when running with TSAN binaries
because of the following sequence of operations:
1. The Kerberos ticket lifetime is set to 10s.
2. The test sets up a mini kudu cluster. This first sets up the KDC,
which creates credentials for all of the Kudu servers and kinits
using test user credentials for the test process.
3. The setup of the cluster takes > 10s.
4. At the end of the cluster setup, the test checks that setup succeeded
in part by issuing a ListTabletServers RPC. This fails because the test
user ticket has expired.
5. The test fails because it can't set up the cluster.
The failure looks like
21:50:06.500 [ERROR - main] (RetryRule.java:217) org.apache.kudu.flume.sink.SecureKuduSinkTest.testEventsWithShortTickets: failed attempt 1
java.io.IOException: ListTabletServers RPC failed: Client connection negotiation failed: client connection to 127.12.111.60:36425: server requires authentication, but client does not have Kerberos credentials available
at org.apache.kudu.test.cluster.MiniKuduCluster.sendRequestToCluster(MiniKuduCluster.java:169) ~[kudu-test-utils-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.apache.kudu.test.cluster.MiniKuduCluster.start(MiniKuduCluster.java:234) ~[kudu-test-utils-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.apache.kudu.test.cluster.MiniKuduCluster.access$300(MiniKuduCluster.java:71) ~[kudu-test-utils-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.apache.kudu.test.cluster.MiniKuduCluster$MiniKuduClusterBuilder.build(MiniKuduCluster.java:658) ~[kudu-test-utils-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.apache.kudu.test.KuduTestHarness.before(KuduTestHarness.java:140) ~[kudu-test-utils-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:46) ~[junit-4.12.jar:4.12]
at org.apache.kudu.test.junit.RetryRule$RetryStatement.doOneAttempt(RetryRule.java:215) [kudu-test-utils-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
at org.apache.kudu.test.junit.RetryRule$RetryStatement.evaluate(RetryRule.java:232) [kudu-test-utils-1.10.0-SNAPSHOT.jar:1.10.0-SNAPSHOT]
...
This patch attempts to deflake the test a bit by doubling the ticket
lifetime to 20s. It also raises the renewal lifetime to 35s from 30s,
to provide a bit of extra time between the ticket expiring and when
Flume needs to renew the ticket. Before, the test waited 2x the
renewable ticket lifetime. I made it so the test waits until the
renewable ticket lifetime plus one second has passed, including the
time spent in the test so far.
I tried to test this on dist-test using TSAN binaries. With the new
patch I saw 0/1000 failures, but without it I saw 830/1000 failures.
That's *way* flakier than any previous indication, so I don't trust
those results. The failures I sampled did seem to be related to the same
issue, but it was ConnectToCluster RPCs failing instead.
Change-Id: Icc936878d7f1496905e83ddaf93b9b049f417f72
Reviewed-on: http://gerrit.cloudera.org:8080/13454
Reviewed-by: Adar Dembo <ad...@cloudera.com>
Tested-by: Kudu Jenkins
Reviewed-by: Alexey Serbin <as...@cloudera.com>
---
M java/kudu-flume-sink/src/test/java/org/apache/kudu/flume/sink/SecureKuduSinkTest.java
1 file changed, 9 insertions(+), 3 deletions(-)
Approvals:
Adar Dembo: Looks good to me, approved
Kudu Jenkins: Verified
Alexey Serbin: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/13454
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Icc936878d7f1496905e83ddaf93b9b049f417f72
Gerrit-Change-Number: 13454
Gerrit-PatchSet: 3
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>