[kudu-CR] rpc: add support for passing an open file descriptor across a unix socket

["Todd Lipcon (Code Review)" <ge...@cloudera.org>]

Hello Tidy Bot, Andrew Wong, Kudu Jenkins, Grant Henke, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/15706

to look at the new patch set (#3).

Change subject: rpc: add support for passing an open file descriptor across a unix socket
......................................................................

rpc: add support for passing an open file descriptor across a unix socket

This adds RpcController::SendFileDescriptor() which allows an RPC client
to pass an open file to the server. The server can retrieve the file
from the RpcContext.

The eventual goal here is that a client can create a memfd file (or an
unlinked file in /dev/shm if memfd isn't supported) and pass it to a
tserver in a Scan RPC. The server can then write the scan results into
the file instead of passing the data back via the normal RPC sidecar
mechanism. Assuming the client has mmapped the file, this should results
in the elimination of a bunch of copies and load on the reactor threads.

This includes a benchmark that compares various methods for doing this
shared memory transport. Results from running the benchmark on various
systems show that the shared memory can be substantially (2-5x) faster
than the existing sidecar approach:

88-thread Xeon E5-2699 v4 @ 2.20GHz, el7:
  https://gist.github.com/toddlipcon/c691afcb72ddea7d1c6c1e54904e94a6

8-thread Core i7-7820HQ @ 2.9Ghz, Ubuntu 18:
  https://gist.github.com/toddlipcon/28193dd2d2017bba2aaebdea78bcee6a

AWS r5d.2xlarge 8vCPU (Xeon Platinum 8175M CPU @ 2.50GHz), Ubuntu 18:
  https://gist.github.com/toddlipcon/f84a09a92d24939d9d49dd7747d8aa9e

Change-Id: I89baff83fa85f9109429fdc29d6eefc9f0f8b4a2
---
M src/kudu/gutil/linux_syscall_support.h
M src/kudu/rpc/connection.cc
M src/kudu/rpc/inbound_call.cc
M src/kudu/rpc/inbound_call.h
M src/kudu/rpc/messenger.h
M src/kudu/rpc/outbound_call.cc
M src/kudu/rpc/outbound_call.h
M src/kudu/rpc/reactor.h
M src/kudu/rpc/rpc-test-base.h
M src/kudu/rpc/rpc_context.cc
M src/kudu/rpc/rpc_context.h
M src/kudu/rpc/rpc_controller.cc
M src/kudu/rpc/rpc_controller.h
M src/kudu/rpc/rpc_stub-test.cc
M src/kudu/rpc/rtest.proto
M src/kudu/rpc/service_pool.h
M src/kudu/rpc/transfer.cc
M src/kudu/rpc/transfer.h
M src/kudu/security/tls_socket.cc
M src/kudu/security/tls_socket.h
A src/kudu/util/fd.h
M src/kudu/util/net/socket.cc
M src/kudu/util/net/socket.h
23 files changed, 889 insertions(+), 31 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/06/15706/3
-- 
To view, visit http://gerrit.cloudera.org:8080/15706
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I89baff83fa85f9109429fdc29d6eefc9f0f8b4a2
Gerrit-Change-Number: 15706
Gerrit-PatchSet: 3
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Andrew Wong <an...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)