[kudu-CR] rpc: use full configured timeout for connection negotiation

["Todd Lipcon (Code Review)" <ge...@cloudera.org>]

Hello David Ribeiro Alves, Mike Percy, Adar Dembo,

I'd like you to do a code review.  Please visit

    http://gerrit.cloudera.org:8080/2948

to review the following change.

Change subject: rpc: use full configured timeout for connection negotiation
......................................................................

rpc: use full configured timeout for connection negotiation

Previously, if a call was submitted with a short timeout, that timeout
would be used for the connection negotiation itself. This meant that
the following sequence could fail:

- submit a call with a short timeout (eg 10ms)
- connection negotiation starts with short timeout
- submit a call with a long timeout (eg 5s)
- the call sees a connection negotiation is already in progress, and just
  enqueues itself to the existing connection
- the connection negotiation times out, and both calls fail with a
  TimedOut status

In particular, this caused a lot of flakiness in TestRpc.TestCallTimeout,
where the first call made has a short timeout.

This patch changes the code to always use the full connection negotiation
timeout regardless of the call's time out. If a call has a short timeout,
the connection will continue to be set up in the background.

Note that this means that a call can time out while connection negotiation
is still ongoing, and thus the connection may have an OutboundTransfer on
its queue for a timed-out call. We don't want to send these already-failed
calls to the server, so this patch also makes the connection skip sending
outbound transfers when the associated call has already failed.

Change-Id: Ie3e284f8d879f2b0e0a62e4f29c445ae53c5ea2c
---
M src/kudu/rpc/connection.cc
M src/kudu/rpc/negotiation.cc
M src/kudu/rpc/outbound_call.cc
M src/kudu/rpc/outbound_call.h
M src/kudu/rpc/reactor.cc
M src/kudu/rpc/reactor.h
M src/kudu/rpc/rpc-test.cc
M src/kudu/rpc/rpc_header.proto
M src/kudu/rpc/transfer.cc
M src/kudu/rpc/transfer.h
10 files changed, 127 insertions(+), 76 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/48/2948/1
-- 
To view, visit http://gerrit.cloudera.org:8080/2948
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ie3e284f8d879f2b0e0a62e4f29c445ae53c5ea2c
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: David Ribeiro Alves <da...@cloudera.com>
Gerrit-Reviewer: Mike Percy <mp...@apache.org>