[kudu-CR] KUDU-3357 endpoints for proxied RPCs

["Alexey Serbin (Code Review)" <ge...@cloudera.org>]

Hello Zoltan Chovan, Kurt Deschler, Attila Bukor, Kudu Jenkins, Abhishek Chennaka, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/19231

to look at the new patch set (#5).

Change subject: KUDU-3357 endpoints for proxied RPCs
......................................................................

KUDU-3357 endpoints for proxied RPCs

This patch introduces a solution to the problem outlined in KUDU-3357.

The idea is to establish separate RPC endpoint(s) for Kudu servers to
handle traffic proxied from external network(s).  So, when a Kudu server
receives an RPC request, it has enough information to decide whether
to handle the request as arriving from the internal or some external
network.  All the communications of Kudu components in the cluster
should be routed through the standard RPC endpoints, but the requests
proxied from external networks should be routed through those dedicated
RPC endpoints.  When a Kudu server receives an RPC through such an
endpoint, it can substitute internal RPC addresses of Kudu servers with
corresponding RPC addresses reachable to the client through a TCP proxy.

With that, the following new flags have been introduced, both accepting
comma-separated list of strings of form '<hostname>:<port>':

--rpc_proxy_advertised_addresses

  That's to set the server's RPC endpoints exposed to the external
  network via a TCP proxy.

--rpc_proxied_addresses

  That's to define RPC endpoints in the inner network to handle
  RPC requests forwarded/proxied from outside networks.  It's possible
  to use a wildcard for IP address (i.e. 0.0.0.0)
  and the port number (i.e. 0) for the elements of this address list.

The newly introduced --rpc_proxy_advertised_addresses is orthogonal
to already existing --rpc_advertised_addresses, so it's possible to use
both simultaneously if the network environment for Docker containers
in the private internal network is configured in a funny way.

This approach allows for separating the internal and the external
traffic, meanwhile providing the connectivity for Kudu clients running
in external networks, where the internal traffic is never routed through
a proxy's or a loadbalancer's endpoint.  The approach with having only
--rpc_advertised_addresses for public cloud deployments (referred
by KUDU-3357) routes _all_ the Kudu traffic through the endpoints
exposed by the proxy/loadbalancer, and that's the problem this
patch addresses.

I verified this patch works as expected in k8s environment running in
AWS/EC2 cloud where Kudu cluster was deployed in a containerized manner
using Kudu Docker images.  In particular, RPC calls from a client
running in the external network (I was running it from my laptop behind
a firewall) were forwarded/proxied via a TCP proxy (NGINX) to Kudu
servers running in a AWS cluster deployed behind a load balancer.
I used the "kudu perf loadgen" CLI tool to create tables and write
data, and "kudu perf table_scan" to read data.  A test Kudu Java client
application worked as well.

NOTE: even if "kudu cluster ksck" tool also worked, it's not yet a goal
      to be able to run "kudu cluster ksck" and other administrative
      CLI tools from the outside; those tasks are expected to be
      performed from within Kudu cluster's internal network

Follow-up patches should also add:
  * proper advertising of a proxy/loadbalancer endpoint to be forwarded
    to the embedded web server's endpoint for master and tablet servers
  * support for multi-master configurations when forwarding RPCs
    from external networks

Change-Id: Ic300250556d3f6e522a71923bed6aa5cd45375ea
---
M src/kudu/client/client-test.cc
M src/kudu/common/wire_protocol.proto
M src/kudu/integration-tests/CMakeLists.txt
A src/kudu/integration-tests/client-proxied-rpc-test.cc
M src/kudu/integration-tests/consistency-itest.cc
M src/kudu/integration-tests/create-table-itest.cc
M src/kudu/integration-tests/create-table-stress-test.cc
M src/kudu/integration-tests/registration-test.cc
M src/kudu/integration-tests/table_locations-itest.cc
M src/kudu/master/auto_leader_rebalancer.cc
M src/kudu/master/auto_rebalancer.cc
M src/kudu/master/catalog_manager.cc
M src/kudu/master/catalog_manager.h
M src/kudu/master/master-test-util.h
M src/kudu/master/master-test.cc
M src/kudu/master/master.cc
M src/kudu/master/master.h
M src/kudu/master/master_path_handlers.cc
M src/kudu/master/master_service.cc
M src/kudu/master/ts_descriptor.cc
M src/kudu/master/ts_descriptor.h
M src/kudu/server/rpc_server-test.cc
M src/kudu/server/rpc_server.cc
M src/kudu/server/rpc_server.h
M src/kudu/tserver/heartbeater.cc
M src/kudu/util/net/net_util.cc
M src/kudu/util/net/net_util.h
27 files changed, 1,110 insertions(+), 175 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/31/19231/5
-- 
To view, visit http://gerrit.cloudera.org:8080/19231
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic300250556d3f6e522a71923bed6aa5cd45375ea
Gerrit-Change-Number: 19231
Gerrit-PatchSet: 5
Gerrit-Owner: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Zoltan Chovan <zc...@cloudera.com>