You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Jean-Daniel Cryans (Code Review)" <ge...@cloudera.org> on 2016/09/30 02:01:38 UTC

[kudu-CR] [java client] Tight-ish loop in master lookups if a tablet doesn't have a leader 

Jean-Daniel Cryans has uploaded a new change for review.

  http://gerrit.cloudera.org:8080/4570

Change subject: [java client] Tight-ish loop in master lookups if a tablet doesn't have a leader
......................................................................

[java client] Tight-ish loop in master lookups if a tablet doesn't have a leader

There's currently the possiblity of a situation like this:
1. sendRpcToTablet finds the tablet that the RPC is going to doesn't have a leader
2. a master lookup is sent
3. discoverTablet is called back and still doesn't find a leader
4. RetryRpcCallback is invoked right away, going back to step 1

This quickly gets us with this exception:

Too many attempts: KuduRpc(method=Write, tablet=redacted, attempt=101, DeadlineTracker(timeout=30000, elapsed=4747)

Notice how it retried 101 times in less than 5 seconds.

This patch changes step 3 so that an exception is thrown so that RetryRpcErrback
is invoked instead, which will add delay before retrying the RPC.

This bug was found by ITClient. It's not _just_ a flaky test!

Change-Id: Ibf2bd53b03551642e4d036d322e1e592b7c2cfec
---
M java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduClient.java
A java/kudu-client/src/main/java/org/apache/kudu/client/NoSuitableReplicaException.java
M java/kudu-client/src/test/java/org/apache/kudu/client/TestAsyncKuduClient.java
3 files changed, 98 insertions(+), 19 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/70/4570/1
-- 
To view, visit http://gerrit.cloudera.org:8080/4570
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ibf2bd53b03551642e4d036d322e1e592b7c2cfec
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Jean-Daniel Cryans <jd...@apache.org>

[kudu-CR] [java client] Tight-ish loop in master lookups if a tablet doesn't have a leader

Posted by "Jean-Daniel Cryans (Code Review)" <ge...@cloudera.org>.
Jean-Daniel Cryans has posted comments on this change.

Change subject: [java client] Tight-ish loop in master lookups if a tablet doesn't have a leader
......................................................................


Patch Set 1:

(7 comments)

http://gerrit.cloudera.org:8080/#/c/4570/1//COMMIT_MSG
Commit Message:

PS1, Line 9: possiblity
> Nit: possibility
Done


PS1, Line 19: 101
> While we're at it, can we remove this upper bound? It's a completely arbitr
It's useful for cases exactly like this one. If you hit "attempt=101" then it means you hit bug.


http://gerrit.cloudera.org:8080/#/c/4570/1/java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduClient.java
File java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduClient.java:

PS1, Line 1409: send
> Nit: throw
Done


Line 1498:     // TODO: Handle the situation when multiple in-flight RPCs are queued waiting
> I know you just moved this comment, but I didn't really understand it origi
They all try to find who the leader is. If you have 1000 RPCs, they will all hit all the masters.


PS1, Line 1499: determine
> Nit: determined
Done


http://gerrit.cloudera.org:8080/#/c/4570/1/java/kudu-client/src/test/java/org/apache/kudu/client/TestAsyncKuduClient.java
File java/kudu-client/src/test/java/org/apache/kudu/client/TestAsyncKuduClient.java:

PS1, Line 50: "testDisconnect"
> This means that if testDisconnect() fails and is retried by maven, it'll fa
Done


Line 157:     KuduTable table = createTable("testNoLeader", basicSchema, options);
> Same here.
Done


-- 
To view, visit http://gerrit.cloudera.org:8080/4570
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ibf2bd53b03551642e4d036d322e1e592b7c2cfec
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Jean-Daniel Cryans <jd...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@cloudera.com>
Gerrit-Reviewer: Jean-Daniel Cryans <jd...@apache.org>
Gerrit-HasComments: Yes

[kudu-CR] [java client] Tight-ish loop in master lookups if a tablet doesn't have a leader

Posted by "Jean-Daniel Cryans (Code Review)" <ge...@cloudera.org>.
Jean-Daniel Cryans has posted comments on this change.

Change subject: [java client] Tight-ish loop in master lookups if a tablet doesn't have a leader
......................................................................


Patch Set 2:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/4570/1//COMMIT_MSG
Commit Message:

PS1, Line 19: 101
> It's useful for development, yes, but not useful (and I would argue potenti
Not having this limit would be hurtful in production, it means runaway operations can start hammering master and tablet servers as long as their timeout allows it. If the bug this jira is fixing happened in prod, you'd hit the master with GetTabletLocations as often as you can for 30 seconds.


http://gerrit.cloudera.org:8080/#/c/4570/1/java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduClient.java
File java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduClient.java:

Line 1498:     // TODO: Handle the situation when multiple in-flight RPCs are queued waiting
> Hmm, maybe we should acquire a "master lookup permit" here then? Not releva
Yup.


-- 
To view, visit http://gerrit.cloudera.org:8080/4570
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ibf2bd53b03551642e4d036d322e1e592b7c2cfec
Gerrit-PatchSet: 2
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Jean-Daniel Cryans <jd...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@cloudera.com>
Gerrit-Reviewer: Jean-Daniel Cryans <jd...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-HasComments: Yes

[kudu-CR] [java client] Tight-ish loop in master lookups if a tablet doesn't have a leader

Posted by "Jean-Daniel Cryans (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins,

I'd like you to reexamine a change.  Please visit

    http://gerrit.cloudera.org:8080/4570

to look at the new patch set (#3).

Change subject: [java client] Tight-ish loop in master lookups if a tablet doesn't have a leader
......................................................................

[java client] Tight-ish loop in master lookups if a tablet doesn't have a leader

There's currently the possibility of a situation like this:
1. sendRpcToTablet finds the tablet that the RPC is going to doesn't have a leader
2. a master lookup is sent
3. discoverTablet is called back and still doesn't find a leader
4. RetryRpcCallback is invoked right away, going back to step 1

This quickly gets us with this exception:

Too many attempts: KuduRpc(method=Write, tablet=redacted, attempt=101, DeadlineTracker(timeout=30000, elapsed=4747)

Notice how it retried 101 times in less than 5 seconds.

This patch changes step 3 so that an exception is thrown so that RetryRpcErrback
is invoked instead, which will add delay before retrying the RPC.

This bug was found by ITClient. It's not _just_ a flaky test!

Change-Id: Ibf2bd53b03551642e4d036d322e1e592b7c2cfec
---
M java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduClient.java
A java/kudu-client/src/main/java/org/apache/kudu/client/NoSuitableReplicaException.java
M java/kudu-client/src/test/java/org/apache/kudu/client/TestAsyncKuduClient.java
3 files changed, 100 insertions(+), 19 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/70/4570/3
-- 
To view, visit http://gerrit.cloudera.org:8080/4570
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ibf2bd53b03551642e4d036d322e1e592b7c2cfec
Gerrit-PatchSet: 3
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Jean-Daniel Cryans <jd...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@cloudera.com>
Gerrit-Reviewer: Jean-Daniel Cryans <jd...@apache.org>
Gerrit-Reviewer: Kudu Jenkins

[kudu-CR] [java client] Tight-ish loop in master lookups if a tablet doesn't have a leader

Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change.

Change subject: [java client] Tight-ish loop in master lookups if a tablet doesn't have a leader
......................................................................


Patch Set 2:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/4570/1//COMMIT_MSG
Commit Message:

PS1, Line 19: 101
> It's useful for cases exactly like this one. If you hit "attempt=101" then 
It's useful for development, yes, but not useful (and I would argue potentially harmful) in production.

Instead, perhaps we can noisily log when we exceed 100 attempts? Or make the 100 attempt limit test-only?


http://gerrit.cloudera.org:8080/#/c/4570/1/java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduClient.java
File java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduClient.java:

Line 1498:     // TODO: Handle the situation when multiple in-flight RPCs are queued waiting
> They all try to find who the leader is. If you have 1000 RPCs, they will al
Hmm, maybe we should acquire a "master lookup permit" here then? Not relevant to the current patch in any case.

But, since you're moving the comment, can you reword it to make it clear why this is a TODO? That is, we should implement the optimization you just told me about.


-- 
To view, visit http://gerrit.cloudera.org:8080/4570
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ibf2bd53b03551642e4d036d322e1e592b7c2cfec
Gerrit-PatchSet: 2
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Jean-Daniel Cryans <jd...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@cloudera.com>
Gerrit-Reviewer: Jean-Daniel Cryans <jd...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-HasComments: Yes

[kudu-CR] [java client] Tight-ish loop in master lookups if a tablet doesn't have a leader

Posted by "Jean-Daniel Cryans (Code Review)" <ge...@cloudera.org>.
Jean-Daniel Cryans has uploaded a new patch set (#2).

Change subject: [java client] Tight-ish loop in master lookups if a tablet doesn't have a leader
......................................................................

[java client] Tight-ish loop in master lookups if a tablet doesn't have a leader

There's currently the possibility of a situation like this:
1. sendRpcToTablet finds the tablet that the RPC is going to doesn't have a leader
2. a master lookup is sent
3. discoverTablet is called back and still doesn't find a leader
4. RetryRpcCallback is invoked right away, going back to step 1

This quickly gets us with this exception:

Too many attempts: KuduRpc(method=Write, tablet=redacted, attempt=101, DeadlineTracker(timeout=30000, elapsed=4747)

Notice how it retried 101 times in less than 5 seconds.

This patch changes step 3 so that an exception is thrown so that RetryRpcErrback
is invoked instead, which will add delay before retrying the RPC.

This bug was found by ITClient. It's not _just_ a flaky test!

Change-Id: Ibf2bd53b03551642e4d036d322e1e592b7c2cfec
---
M java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduClient.java
A java/kudu-client/src/main/java/org/apache/kudu/client/NoSuitableReplicaException.java
M java/kudu-client/src/test/java/org/apache/kudu/client/TestAsyncKuduClient.java
3 files changed, 104 insertions(+), 19 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/70/4570/2
-- 
To view, visit http://gerrit.cloudera.org:8080/4570
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ibf2bd53b03551642e4d036d322e1e592b7c2cfec
Gerrit-PatchSet: 2
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Jean-Daniel Cryans <jd...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@cloudera.com>
Gerrit-Reviewer: Jean-Daniel Cryans <jd...@apache.org>

[kudu-CR] [java client] Tight-ish loop in master lookups if a tablet doesn't have a leader

Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change.

Change subject: [java client] Tight-ish loop in master lookups if a tablet doesn't have a leader
......................................................................


Patch Set 2:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/4570/1//COMMIT_MSG
Commit Message:

PS1, Line 19: 101
> Not having this limit would be hurtful in production, it means runaway oper
I don't like how the limit may trigger accidentally if the cluster is under heavy load and the user has asked for long timeouts. I also don't like that it violates the (otherwise universally applied) contract that timeouts reign supreme. And for what it's worth, we've had a tight loop or two in the C++ client before and we've managed to find them without such a limit, which is partly why no such thing exists there.

Since it appears that we agree to disagree, I'll ask around for a third opinion.


-- 
To view, visit http://gerrit.cloudera.org:8080/4570
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ibf2bd53b03551642e4d036d322e1e592b7c2cfec
Gerrit-PatchSet: 2
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Jean-Daniel Cryans <jd...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@cloudera.com>
Gerrit-Reviewer: Jean-Daniel Cryans <jd...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-HasComments: Yes

[kudu-CR] [java client] Tight-ish loop in master lookups if a tablet doesn't have a leader

Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change.

Change subject: [java client] Tight-ish loop in master lookups if a tablet doesn't have a leader
......................................................................


Patch Set 3: Code-Review+2

-- 
To view, visit http://gerrit.cloudera.org:8080/4570
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ibf2bd53b03551642e4d036d322e1e592b7c2cfec
Gerrit-PatchSet: 3
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Jean-Daniel Cryans <jd...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@cloudera.com>
Gerrit-Reviewer: Jean-Daniel Cryans <jd...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-HasComments: No

[kudu-CR] [java client] Tight-ish loop in master lookups if a tablet doesn't have a leader

Posted by "Jean-Daniel Cryans (Code Review)" <ge...@cloudera.org>.
Jean-Daniel Cryans has posted comments on this change.

Change subject: [java client] Tight-ish loop in master lookups if a tablet doesn't have a leader
......................................................................


Patch Set 3: Verified+1

Dan messed up my build, but I did run all the Java tests locally before pushing the latest patchset.

-- 
To view, visit http://gerrit.cloudera.org:8080/4570
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ibf2bd53b03551642e4d036d322e1e592b7c2cfec
Gerrit-PatchSet: 3
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Jean-Daniel Cryans <jd...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@cloudera.com>
Gerrit-Reviewer: Jean-Daniel Cryans <jd...@apache.org>
Gerrit-HasComments: No

[kudu-CR] [java client] Tight-ish loop in master lookups if a tablet doesn't have a leader

Posted by "Dan Burkert (Code Review)" <ge...@cloudera.org>.
Dan Burkert has posted comments on this change.

Change subject: [java client] Tight-ish loop in master lookups if a tablet doesn't have a leader
......................................................................


Patch Set 2:

(2 comments)

I agree with JD: we routinely find tight loops like this.  The loop limiter is necessary as a backstop until we can comprehensively implement a better solution.

http://gerrit.cloudera.org:8080/#/c/4570/2/java/kudu-client/src/main/java/org/apache/kudu/client/NoSuitableReplicaException.java
File java/kudu-client/src/main/java/org/apache/kudu/client/NoSuitableReplicaException.java:

Line 23: final class NoSuitableReplicaException extends RecoverableException {
How about renaming NoLeaderMasterFoundException to NoLeaderFoundException and reusing that?


http://gerrit.cloudera.org:8080/#/c/4570/2/java/kudu-client/src/test/java/org/apache/kudu/client/TestAsyncKuduClient.java
File java/kudu-client/src/test/java/org/apache/kudu/client/TestAsyncKuduClient.java:

PS2, Line 196: insert.partitionKey()
this could just be the empty array, no need to create an insert.


-- 
To view, visit http://gerrit.cloudera.org:8080/4570
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ibf2bd53b03551642e4d036d322e1e592b7c2cfec
Gerrit-PatchSet: 2
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Jean-Daniel Cryans <jd...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@cloudera.com>
Gerrit-Reviewer: Jean-Daniel Cryans <jd...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-HasComments: Yes

[kudu-CR] [java client] Tight-ish loop in master lookups if a tablet doesn't have a leader

Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change.

Change subject: [java client] Tight-ish loop in master lookups if a tablet doesn't have a leader
......................................................................


Patch Set 1:

(7 comments)

http://gerrit.cloudera.org:8080/#/c/4570/1//COMMIT_MSG
Commit Message:

PS1, Line 9: possiblity
Nit: possibility


PS1, Line 19: 101
While we're at it, can we remove this upper bound? It's a completely arbitrary number and it doesn't make sense; only the deadline expiring should make us stop retrying.


http://gerrit.cloudera.org:8080/#/c/4570/1/java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduClient.java
File java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduClient.java:

PS1, Line 1409: send
Nit: throw


Line 1498:     // TODO: Handle the situation when multiple in-flight RPCs are queued waiting
I know you just moved this comment, but I didn't really understand it originally either. What makes this situation special such that we need additional logic for it? What happens today when multiple RPCs are waiting to find out who the leader master is?


PS1, Line 1499: determine
Nit: determined


http://gerrit.cloudera.org:8080/#/c/4570/1/java/kudu-client/src/test/java/org/apache/kudu/client/TestAsyncKuduClient.java
File java/kudu-client/src/test/java/org/apache/kudu/client/TestAsyncKuduClient.java:

PS1, Line 50: "testDisconnect"
This means that if testDisconnect() fails and is retried by maven, it'll fail again because the testDisconnect table will already exist.

Until such a time that we clean up the mini cluster state after each test like the C++ tests, we need to retain the idiom of appending the time to the table name.


Line 157:     KuduTable table = createTable("testNoLeader", basicSchema, options);
Same here.


-- 
To view, visit http://gerrit.cloudera.org:8080/4570
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ibf2bd53b03551642e4d036d322e1e592b7c2cfec
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Jean-Daniel Cryans <jd...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@cloudera.com>
Gerrit-HasComments: Yes

[kudu-CR] [java client] Tight-ish loop in master lookups if a tablet doesn't have a leader

Posted by "Jean-Daniel Cryans (Code Review)" <ge...@cloudera.org>.
Jean-Daniel Cryans has posted comments on this change.

Change subject: [java client] Tight-ish loop in master lookups if a tablet doesn't have a leader
......................................................................


Patch Set 2:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/4570/2/java/kudu-client/src/main/java/org/apache/kudu/client/NoSuitableReplicaException.java
File java/kudu-client/src/main/java/org/apache/kudu/client/NoSuitableReplicaException.java:

Line 23: final class NoSuitableReplicaException extends RecoverableException {
> How about renaming NoLeaderMasterFoundException to NoLeaderFoundException a
It's not really the same thing, I think. There could be other reasons why we can't find a suitable replica, although at the moment we only have the missing leader situation.


http://gerrit.cloudera.org:8080/#/c/4570/2/java/kudu-client/src/test/java/org/apache/kudu/client/TestAsyncKuduClient.java
File java/kudu-client/src/test/java/org/apache/kudu/client/TestAsyncKuduClient.java:

PS2, Line 196: insert.partitionKey()
> this could just be the empty array, no need to create an insert.
Done


-- 
To view, visit http://gerrit.cloudera.org:8080/4570
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ibf2bd53b03551642e4d036d322e1e592b7c2cfec
Gerrit-PatchSet: 2
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Jean-Daniel Cryans <jd...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@cloudera.com>
Gerrit-Reviewer: Jean-Daniel Cryans <jd...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-HasComments: Yes

[kudu-CR] [java client] Tight-ish loop in master lookups if a tablet doesn't have a leader

Posted by "Jean-Daniel Cryans (Code Review)" <ge...@cloudera.org>.
Jean-Daniel Cryans has submitted this change and it was merged.

Change subject: [java client] Tight-ish loop in master lookups if a tablet doesn't have a leader
......................................................................


[java client] Tight-ish loop in master lookups if a tablet doesn't have a leader

There's currently the possibility of a situation like this:
1. sendRpcToTablet finds the tablet that the RPC is going to doesn't have a leader
2. a master lookup is sent
3. discoverTablet is called back and still doesn't find a leader
4. RetryRpcCallback is invoked right away, going back to step 1

This quickly gets us with this exception:

Too many attempts: KuduRpc(method=Write, tablet=redacted, attempt=101, DeadlineTracker(timeout=30000, elapsed=4747)

Notice how it retried 101 times in less than 5 seconds.

This patch changes step 3 so that an exception is thrown so that RetryRpcErrback
is invoked instead, which will add delay before retrying the RPC.

This bug was found by ITClient. It's not _just_ a flaky test!

Change-Id: Ibf2bd53b03551642e4d036d322e1e592b7c2cfec
Reviewed-on: http://gerrit.cloudera.org:8080/4570
Reviewed-by: Adar Dembo <ad...@cloudera.com>
Tested-by: Jean-Daniel Cryans <jd...@apache.org>
---
M java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduClient.java
A java/kudu-client/src/main/java/org/apache/kudu/client/NoSuitableReplicaException.java
M java/kudu-client/src/test/java/org/apache/kudu/client/TestAsyncKuduClient.java
3 files changed, 100 insertions(+), 19 deletions(-)

Approvals:
  Jean-Daniel Cryans: Verified
  Adar Dembo: Looks good to me, approved



-- 
To view, visit http://gerrit.cloudera.org:8080/4570
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ibf2bd53b03551642e4d036d322e1e592b7c2cfec
Gerrit-PatchSet: 4
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Jean-Daniel Cryans <jd...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@cloudera.com>
Gerrit-Reviewer: Jean-Daniel Cryans <jd...@apache.org>