You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org> on 2022/03/30 16:27:48 UTC
[Impala-ASF-CR] WIP IMPALA-11129: Support running KRPC over Unix Domain Socket
Wenzhe Zhou has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18369
Change subject: WIP IMPALA-11129: Support running KRPC over Unix Domain Socket
......................................................................
WIP IMPALA-11129: Support running KRPC over Unix Domain Socket
This patch make following changes to support running KRPC over UDS.
- Add a FLAG_rpc_use_unix_domain_socket to enable running KRPC over
UDS.
- Add field uds_address in NetworkAddressPB.
Replace TNetworkAddress with NetworkAddressPB for RPC related
class variables and APIs.
- Set UDS address for each daemon as @impala-kprc:<BackendId>
during initialization.
- When FLAG_rpc_use_unix_domain_socket is true, the socket of KRPC
server will be binded to the UDS address of the daemon.
KRPC Client will connect to KRPC server with the UDS address of
the server when creating proxy service, which in turn call
kudu::Socket::Connect() function to connect KRPC server.
TODO:
- Find a way to advertise BackendId of admissiond in global admission
control mode.
- Fix rpcz web page when FLAG_rpc_use_unix_domain_socket is true.
webserver.test_web_pages.TestWebPage.test_krpc_rpcz is skiped now
when FLAG_rpc_use_unix_domain_socket is set as true.
There are two issues:
- rpcz web page show TCP related stats which are not available for
UDS. We may use a different tmpl file for UDS.
- Cannot get remote address from Unix domain socket for inbound
connection.
- Add more unit-tests.
Testing:
- Passed core test with FLAG_rpc_use_unix_domain_socket as fault
value false.
- Passed core test with FLAG_rpc_use_unix_domain_socket as true.
Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7
---
M be/src/benchmarks/expr-benchmark.cc
M be/src/rpc/impala-service-pool.cc
M be/src/rpc/impala-service-pool.h
M be/src/rpc/rpc-mgr-kerberized-test.cc
M be/src/rpc/rpc-mgr-test.cc
M be/src/rpc/rpc-mgr-test.h
M be/src/rpc/rpc-mgr.cc
M be/src/rpc/rpc-mgr.h
M be/src/rpc/rpc-mgr.inline.h
M be/src/runtime/coordinator-backend-state.cc
M be/src/runtime/data-stream-test.cc
M be/src/runtime/exec-env.cc
M be/src/runtime/exec-env.h
M be/src/runtime/krpc-data-stream-mgr.cc
M be/src/runtime/krpc-data-stream-sender.cc
M be/src/runtime/query-driver.cc
M be/src/runtime/query-state.cc
M be/src/runtime/runtime-filter-bank.cc
M be/src/runtime/test-env.cc
M be/src/runtime/tmp-file-mgr.cc
M be/src/scheduling/admissiond-env.cc
M be/src/scheduling/admissiond-env.h
M be/src/service/client-request-state.cc
M be/src/service/control-service.cc
M be/src/service/control-service.h
M be/src/service/data-stream-service.cc
M be/src/service/data-stream-service.h
M be/src/service/impala-server.cc
M be/src/service/impala-server.h
M be/src/testutil/in-process-servers.cc
M be/src/util/container-util.h
M be/src/util/network-util.cc
M be/src/util/network-util.h
M common/protobuf/common.proto
M tests/custom_cluster/test_restart_services.py
M tests/webserver/test_web_pages.py
36 files changed, 348 insertions(+), 116 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/18369/1
--
To view, visit http://gerrit.cloudera.org:8080/18369
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7
Gerrit-Change-Number: 18369
Gerrit-PatchSet: 1
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
[Impala-ASF-CR] WIP IMPALA-11129: Support running KRPC over Unix Domain Socket
Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/18369 )
Change subject: WIP IMPALA-11129: Support running KRPC over Unix Domain Socket
......................................................................
WIP IMPALA-11129: Support running KRPC over Unix Domain Socket
This patch make following changes to support running KRPC over UDS.
- Add a FLAG_rpc_use_unix_domain_socket to enable running KRPC over
UDS.
- Add field uds_address in NetworkAddressPB.
Replace TNetworkAddress with NetworkAddressPB for RPC related
class variables and APIs.
- Set UDS address for each daemon as @impala-kprc:<BackendId>
during initialization.
- When FLAG_rpc_use_unix_domain_socket is true, the socket of KRPC
server will be binded to the UDS address of the daemon.
KRPC Client will connect to KRPC server with the UDS address of
the server when creating proxy service, which in turn call
kudu::Socket::Connect() function to connect KRPC server.
TODO:
- Find a way to advertise BackendId of admissiond in global admission
control mode.
- Fix rpcz web page when FLAG_rpc_use_unix_domain_socket is true.
webserver.test_web_pages.TestWebPage.test_krpc_rpcz is skiped now
when FLAG_rpc_use_unix_domain_socket is set as true.
There are two issues:
- rpcz web page show TCP related stats which are not available for
UDS. We may use a different tmpl file for UDS.
- Cannot get remote address from Unix domain socket for inbound
connection.
- Add more unit-tests.
Testing:
- Passed core test with FLAG_rpc_use_unix_domain_socket as fault
value false.
- Passed core test with FLAG_rpc_use_unix_domain_socket as true.
Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7
---
M be/src/benchmarks/expr-benchmark.cc
M be/src/rpc/impala-service-pool.cc
M be/src/rpc/impala-service-pool.h
M be/src/rpc/rpc-mgr-kerberized-test.cc
M be/src/rpc/rpc-mgr-test.cc
M be/src/rpc/rpc-mgr-test.h
M be/src/rpc/rpc-mgr.cc
M be/src/rpc/rpc-mgr.h
M be/src/rpc/rpc-mgr.inline.h
M be/src/runtime/coordinator-backend-state.cc
M be/src/runtime/data-stream-test.cc
M be/src/runtime/exec-env.cc
M be/src/runtime/exec-env.h
M be/src/runtime/krpc-data-stream-mgr.cc
M be/src/runtime/krpc-data-stream-sender.cc
M be/src/runtime/query-driver.cc
M be/src/runtime/query-state.cc
M be/src/runtime/runtime-filter-bank.cc
M be/src/runtime/test-env.cc
M be/src/runtime/tmp-file-mgr.cc
M be/src/scheduling/admissiond-env.cc
M be/src/scheduling/admissiond-env.h
M be/src/service/client-request-state.cc
M be/src/service/control-service.cc
M be/src/service/control-service.h
M be/src/service/data-stream-service.cc
M be/src/service/data-stream-service.h
M be/src/service/impala-server.cc
M be/src/service/impala-server.h
M be/src/testutil/in-process-servers.cc
M be/src/util/container-util.h
M be/src/util/network-util.cc
M be/src/util/network-util.h
M common/protobuf/common.proto
M tests/custom_cluster/test_restart_services.py
M tests/webserver/test_web_pages.py
36 files changed, 348 insertions(+), 116 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/18369/2
--
To view, visit http://gerrit.cloudera.org:8080/18369
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7
Gerrit-Change-Number: 18369
Gerrit-PatchSet: 2
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
[Impala-ASF-CR] IMPALA-11129: Support running KRPC over Unix Domain Socket
Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has uploaded a new patch set (#9). ( http://gerrit.cloudera.org:8080/18369 )
Change subject: IMPALA-11129: Support running KRPC over Unix Domain Socket
......................................................................
IMPALA-11129: Support running KRPC over Unix Domain Socket
This patch make following changes to support running KRPC over UDS.
- Add FLAGS_rpc_use_unix_domain_socket to enable running KRPC over
UDS. Add FLAGS_uds_address_unique_id to specify unique Id for UDS
address. It could be 'ip_address', 'backend_id', or 'none'.
- Add field uds_address in NetworkAddressPB.
Replace TNetworkAddress with NetworkAddressPB for RPC related
class variables and APIs.
- Set UDS address for each daemon as @impala-kprc:<unique_id>
during initialization with unique_id specified by starting flag
FLAGS_uds_address_unique_id.
- When FLAG_rpc_use_unix_domain_socket is true, the socket of KRPC
server will be binded to the UDS address of the daemon.
KRPC Client will connect to KRPC server with the UDS address of
the server when creating proxy service, which in turn call
kudu::Socket::Connect() function to connect KRPC server.
- rpcz Web page show TCP related stats as 'N/A' when using UDS.
Show remote UDS address for KRPC inbound connections on rpcz Web
page as '*' when using UDS since the remote UDS addresses are
not available.
- Add new unit-tests for UDS.
TODO:
- Advertise BackendId of admissiond in global admission control mode.
Testing:
- Passed core test with FLAG_rpc_use_unix_domain_socket as fault
value false.
- Passed core test with FLAG_rpc_use_unix_domain_socket as true.
Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7
---
M be/src/benchmarks/expr-benchmark.cc
M be/src/rpc/impala-service-pool.cc
M be/src/rpc/impala-service-pool.h
M be/src/rpc/rpc-mgr-kerberized-test.cc
M be/src/rpc/rpc-mgr-test.cc
M be/src/rpc/rpc-mgr-test.h
M be/src/rpc/rpc-mgr.cc
M be/src/rpc/rpc-mgr.h
M be/src/rpc/rpc-mgr.inline.h
M be/src/runtime/coordinator-backend-state.cc
M be/src/runtime/data-stream-test.cc
M be/src/runtime/exec-env.cc
M be/src/runtime/exec-env.h
M be/src/runtime/krpc-data-stream-mgr.cc
M be/src/runtime/krpc-data-stream-sender.cc
M be/src/runtime/query-driver.cc
M be/src/runtime/query-state.cc
M be/src/runtime/runtime-filter-bank.cc
M be/src/runtime/test-env.cc
M be/src/runtime/tmp-file-mgr.cc
M be/src/scheduling/admissiond-env.cc
M be/src/scheduling/admissiond-env.h
M be/src/service/client-request-state.cc
M be/src/service/control-service.cc
M be/src/service/control-service.h
M be/src/service/data-stream-service.cc
M be/src/service/data-stream-service.h
M be/src/service/impala-server.cc
M be/src/service/impala-server.h
M be/src/testutil/in-process-servers.cc
M be/src/util/container-util.h
M be/src/util/network-util.cc
M be/src/util/network-util.h
M common/protobuf/common.proto
A tests/custom_cluster/test_krpc_socket.py
M tests/custom_cluster/test_restart_services.py
M tests/webserver/test_web_pages.py
M www/rpcz.tmpl
38 files changed, 618 insertions(+), 159 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/18369/9
--
To view, visit http://gerrit.cloudera.org:8080/18369
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7
Gerrit-Change-Number: 18369
Gerrit-PatchSet: 9
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
[Impala-ASF-CR] IMPALA-11129: Support running KRPC over Unix Domain Socket
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18369 )
Change subject: IMPALA-11129: Support running KRPC over Unix Domain Socket
......................................................................
IMPALA-11129: Support running KRPC over Unix Domain Socket
This patch make following changes to support running KRPC over UDS.
- Add FLAGS_rpc_use_unix_domain_socket to enable running KRPC over
UDS. Add FLAGS_uds_address_unique_id to specify unique Id for UDS
address. It could be 'ip_address', 'backend_id', or 'none'.
- Add variable uds_address in NetworkAddressPB and TNetworkAddress.
Replace TNetworkAddress with NetworkAddressPB for KRPC related
class variables and APIs.
- Set UDS address for each daemon as @impala-kprc:<unique_id>
during initialization with unique_id specified by starting flag
FLAGS_uds_address_unique_id.
- When FLAG_rpc_use_unix_domain_socket is true, the socket of KRPC
server will be binded to the UDS address of the daemon.
KRPC Client will connect to KRPC server with the UDS address of
the server when creating proxy service, which in turn call
kudu::Socket::Connect() function to connect KRPC server.
- rpcz Web page show TCP related stats as 'N/A' when using UDS.
Show remote UDS address for KRPC inbound connections on rpcz Web
page as '*' when using UDS since the remote UDS addresses are
not available.
- Add new unit-tests for UDS.
- BackendId of admissiond is not available. Use admissiond's IP
address as unique ID for UDS.
TODO: Advertise BackendId of admissiond in global admission
control mode.
Testing:
- Passed core test with FLAG_rpc_use_unix_domain_socket as fault
value false.
- Passed core test with FLAG_rpc_use_unix_domain_socket as true.
Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7
Reviewed-on: http://gerrit.cloudera.org:8080/18369
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/benchmarks/expr-benchmark.cc
M be/src/rpc/impala-service-pool.cc
M be/src/rpc/impala-service-pool.h
M be/src/rpc/rpc-mgr-kerberized-test.cc
M be/src/rpc/rpc-mgr-test.cc
M be/src/rpc/rpc-mgr-test.h
M be/src/rpc/rpc-mgr.cc
M be/src/rpc/rpc-mgr.h
M be/src/rpc/rpc-mgr.inline.h
M be/src/runtime/coordinator-backend-state.cc
M be/src/runtime/data-stream-test.cc
M be/src/runtime/exec-env.cc
M be/src/runtime/exec-env.h
M be/src/runtime/krpc-data-stream-mgr.cc
M be/src/runtime/krpc-data-stream-sender.cc
M be/src/runtime/query-driver.cc
M be/src/runtime/query-state.cc
M be/src/runtime/runtime-filter-bank.cc
M be/src/runtime/test-env.cc
M be/src/runtime/tmp-file-mgr.cc
M be/src/scheduling/admissiond-env.cc
M be/src/scheduling/admissiond-env.h
M be/src/service/client-request-state.cc
M be/src/service/control-service.cc
M be/src/service/control-service.h
M be/src/service/data-stream-service.cc
M be/src/service/data-stream-service.h
M be/src/service/impala-server.cc
M be/src/service/impala-server.h
M be/src/testutil/in-process-servers.cc
M be/src/util/container-util.h
M be/src/util/network-util.cc
M be/src/util/network-util.h
M common/protobuf/common.proto
M common/thrift/Types.thrift
A tests/custom_cluster/test_krpc_socket.py
M tests/custom_cluster/test_restart_services.py
M tests/webserver/test_web_pages.py
M www/rpcz.tmpl
39 files changed, 597 insertions(+), 161 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/18369
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7
Gerrit-Change-Number: 18369
Gerrit-PatchSet: 12
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
[Impala-ASF-CR] IMPALA-11129: Support running KRPC over Unix Domain Socket
Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/18369 )
Change subject: IMPALA-11129: Support running KRPC over Unix Domain Socket
......................................................................
IMPALA-11129: Support running KRPC over Unix Domain Socket
This patch make following changes to support running KRPC over UDS.
- Add FLAGS_rpc_use_unix_domain_socket to enable running KRPC over
UDS. Add FLAGS_uds_address_unique_id to specify unique Id for UDS
address. It could be 'ip_address', 'backend_id', or 'none'.
- Add field uds_address in NetworkAddressPB.
Replace TNetworkAddress with NetworkAddressPB for RPC related
class variables and APIs.
- Set UDS address for each daemon as @impala-kprc:<unique_id>
during initialization with unique_id specified by starting flag
FLAGS_uds_address_unique_id.
- When FLAG_rpc_use_unix_domain_socket is true, the socket of KRPC
server will be binded to the UDS address of the daemon.
KRPC Client will connect to KRPC server with the UDS address of
the server when creating proxy service, which in turn call
kudu::Socket::Connect() function to connect KRPC server.
- rpcz Web page show TCP related stats as 'N/A' when using UDS.
Show remote UDS address for KRPC inbound connections on rpcz Web
page as '*' when using UDS since the remote UDS addresses are
not available.
- Add new unit-tests for UDS.
TODO:
- Advertise BackendId of admissiond in global admission control mode.
Testing:
- Passed core test with FLAG_rpc_use_unix_domain_socket as fault
value false.
- Passed core test with FLAG_rpc_use_unix_domain_socket as true.
Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7
---
M be/src/benchmarks/expr-benchmark.cc
M be/src/rpc/impala-service-pool.cc
M be/src/rpc/impala-service-pool.h
M be/src/rpc/rpc-mgr-kerberized-test.cc
M be/src/rpc/rpc-mgr-test.cc
M be/src/rpc/rpc-mgr-test.h
M be/src/rpc/rpc-mgr.cc
M be/src/rpc/rpc-mgr.h
M be/src/rpc/rpc-mgr.inline.h
M be/src/runtime/coordinator-backend-state.cc
M be/src/runtime/data-stream-test.cc
M be/src/runtime/exec-env.cc
M be/src/runtime/exec-env.h
M be/src/runtime/krpc-data-stream-mgr.cc
M be/src/runtime/krpc-data-stream-sender.cc
M be/src/runtime/query-driver.cc
M be/src/runtime/query-state.cc
M be/src/runtime/runtime-filter-bank.cc
M be/src/runtime/test-env.cc
M be/src/runtime/tmp-file-mgr.cc
M be/src/scheduling/admissiond-env.cc
M be/src/scheduling/admissiond-env.h
M be/src/service/client-request-state.cc
M be/src/service/control-service.cc
M be/src/service/control-service.h
M be/src/service/data-stream-service.cc
M be/src/service/data-stream-service.h
M be/src/service/impala-server.cc
M be/src/service/impala-server.h
M be/src/testutil/in-process-servers.cc
M be/src/util/container-util.h
M be/src/util/network-util.cc
M be/src/util/network-util.h
M common/protobuf/common.proto
A tests/custom_cluster/test_krpc_socket.py
M tests/custom_cluster/test_restart_services.py
M tests/webserver/test_web_pages.py
M www/rpcz.tmpl
38 files changed, 593 insertions(+), 138 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/18369/6
--
To view, visit http://gerrit.cloudera.org:8080/18369
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7
Gerrit-Change-Number: 18369
Gerrit-PatchSet: 6
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
[Impala-ASF-CR] IMPALA-11129: Support running KRPC over Unix Domain Socket
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/18369 )
Change subject: IMPALA-11129: Support running KRPC over Unix Domain Socket
......................................................................
Patch Set 9:
(2 comments)
This is looking good to me. I had a couple thoughts, but I'm basically ready to approve this.
http://gerrit.cloudera.org:8080/#/c/18369/8/be/src/rpc/rpc-mgr.cc
File be/src/rpc/rpc-mgr.cc:
http://gerrit.cloudera.org:8080/#/c/18369/8/be/src/rpc/rpc-mgr.cc@120
PS8, Line 120: else if (strcasecmp("ip_address", FLAGS_uds_address_unique_id.c_str()) == 0) {
: uds_addr_unique_id_ = UdsAddressUniqueIdPB::IP_ADDRESS;
: }
> Should we stop running if usd_address_unique_id is not one of allowed value
I see that you disable UDS if the value is invalid.
I think it is also ok to fail startup if uds_address_unique_id has been set to an invalid value. My guess is that the caller of RpcMgr::Init() would handle a not-OK status by halting startup. It can be confusing if the system ignores an invalid value. Admins or developers may think that they have UDS configured when it actually isn't enabled.
http://gerrit.cloudera.org:8080/#/c/18369/8/common/protobuf/common.proto
File common/protobuf/common.proto:
http://gerrit.cloudera.org:8080/#/c/18369/8/common/protobuf/common.proto@28
PS8, Line 28: // Unix Domain Socket address. We use the unique name in "Abstract Namespace" as UDS
: // address in the form of "@impala-krpc:<unique-id>". This field is optional.
: // It is only used for KRPC bind/listen/connect when FLAGS_rpc_use_unix_domain_socket
: // is set as true.
: optional string uds_address = 3;
> We only support UDS for KPRC, not for thrift RPC. I think TNetworkAddress d
My thought is that sometimes we put KRPC addresses in a Thrift structure to send to other daemons. The information is used on the other side to initialize a NetworkAddressPB to use with KRPC, but it is inconvenient to embed a NetworkAddressPB in a Thrift struct.
For example, the code in query-driver.cc and query-state.cc uses a second field to communicate the backend id. That's fine, and this is a rare case. If we wanted TNetworkAddress to have the UDS information, we could, and that case would be a little easier.
I'm also fine with the current way. We should add a sentence to the comment "Refer to Types.thrift for documentation" to say that UDS is limited to KRPC and is only needed for NetworkAddressPB.
--
To view, visit http://gerrit.cloudera.org:8080/18369
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7
Gerrit-Change-Number: 18369
Gerrit-PatchSet: 9
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Tue, 24 May 2022 00:38:52 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11129: Support running KRPC over Unix Domain Socket
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18369 )
Change subject: IMPALA-11129: Support running KRPC over Unix Domain Socket
......................................................................
Patch Set 4:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/10374/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18369
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7
Gerrit-Change-Number: 18369
Gerrit-PatchSet: 4
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Fri, 01 Apr 2022 00:30:12 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11129: Support running KRPC over Unix Domain Socket
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18369 )
Change subject: IMPALA-11129: Support running KRPC over Unix Domain Socket
......................................................................
Patch Set 10:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/10635/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18369
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7
Gerrit-Change-Number: 18369
Gerrit-PatchSet: 10
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Tue, 24 May 2022 19:29:49 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11129: Support running KRPC over Unix Domain Socket
Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/18369 )
Change subject: IMPALA-11129: Support running KRPC over Unix Domain Socket
......................................................................
Patch Set 8:
We can use UDS address as file path or address in the UNIX domain "abstract namespace" (a leading '@' indicates "abstract namespace"). I found that UDS addresses are not released immediately by system when impala daemons are killed if we use UDS address in the form of file path. This cause socket binding failures when restarting impala daemons. Don't see this issue with UDS address in the form of address in abstract namespace. So we use UDS address as address in abstract namespace.
--
To view, visit http://gerrit.cloudera.org:8080/18369
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7
Gerrit-Change-Number: 18369
Gerrit-PatchSet: 8
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Fri, 13 May 2022 21:41:29 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11129: Support running KRPC over Unix Domain Socket
Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/18369 )
Change subject: IMPALA-11129: Support running KRPC over Unix Domain Socket
......................................................................
IMPALA-11129: Support running KRPC over Unix Domain Socket
This patch make following changes to support running KRPC over UDS.
- Add FLAGS_rpc_use_unix_domain_socket to enable running KRPC over
UDS. Add FLAGS_uds_address_unique_id to specify unique Id for UDS
address. It could be 'ip_address', 'backend_id', or 'none'.
- Add field uds_address in NetworkAddressPB.
Replace TNetworkAddress with NetworkAddressPB for RPC related
class variables and APIs.
- Set UDS address for each daemon as @impala-kprc:<BackendId>
during initialization.
- When FLAG_rpc_use_unix_domain_socket is true, the socket of KRPC
server will be binded to the UDS address of the daemon.
KRPC Client will connect to KRPC server with the UDS address of
the server when creating proxy service, which in turn call
kudu::Socket::Connect() function to connect KRPC server.
- rpcz Web page show TCP related stats as 'N/A' when using UDS.
- Add new unit-tests for UDS.
TODO:
- Advertise BackendId of admissiond in global admission control mode.
- Show remote UDS address for KRPC inbound connections on rpcz Web
page.
Testing:
- Passed core test with FLAG_rpc_use_unix_domain_socket as fault
value false.
- Passed core test with FLAG_rpc_use_unix_domain_socket as true.
Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7
---
M be/src/benchmarks/expr-benchmark.cc
M be/src/rpc/impala-service-pool.cc
M be/src/rpc/impala-service-pool.h
M be/src/rpc/rpc-mgr-kerberized-test.cc
M be/src/rpc/rpc-mgr-test.cc
M be/src/rpc/rpc-mgr-test.h
M be/src/rpc/rpc-mgr.cc
M be/src/rpc/rpc-mgr.h
M be/src/rpc/rpc-mgr.inline.h
M be/src/runtime/coordinator-backend-state.cc
M be/src/runtime/data-stream-test.cc
M be/src/runtime/exec-env.cc
M be/src/runtime/exec-env.h
M be/src/runtime/krpc-data-stream-mgr.cc
M be/src/runtime/krpc-data-stream-sender.cc
M be/src/runtime/query-driver.cc
M be/src/runtime/query-state.cc
M be/src/runtime/runtime-filter-bank.cc
M be/src/runtime/test-env.cc
M be/src/runtime/tmp-file-mgr.cc
M be/src/scheduling/admissiond-env.cc
M be/src/scheduling/admissiond-env.h
M be/src/service/client-request-state.cc
M be/src/service/control-service.cc
M be/src/service/control-service.h
M be/src/service/data-stream-service.cc
M be/src/service/data-stream-service.h
M be/src/service/impala-server.cc
M be/src/service/impala-server.h
M be/src/testutil/in-process-servers.cc
M be/src/util/container-util.h
M be/src/util/network-util.cc
M be/src/util/network-util.h
M common/protobuf/common.proto
A tests/custom_cluster/test_krpc_socket.py
M tests/custom_cluster/test_restart_services.py
M tests/webserver/test_web_pages.py
M www/rpcz.tmpl
38 files changed, 582 insertions(+), 139 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/18369/5
--
To view, visit http://gerrit.cloudera.org:8080/18369
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7
Gerrit-Change-Number: 18369
Gerrit-PatchSet: 5
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
[Impala-ASF-CR] IMPALA-11129: Support running KRPC over Unix Domain Socket
Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/18369 )
Change subject: IMPALA-11129: Support running KRPC over Unix Domain Socket
......................................................................
Patch Set 8:
(5 comments)
Joe, thanks for your feedback.
http://gerrit.cloudera.org:8080/#/c/18369/8/be/src/rpc/rpc-mgr-test.cc
File be/src/rpc/rpc-mgr-test.cc:
http://gerrit.cloudera.org:8080/#/c/18369/8/be/src/rpc/rpc-mgr-test.cc@380
PS8, Line 380: // Run tests with Unix domain socket.
: FLAGS_rpc_use_unix_domain_socket = true;
: discard_result(RUN_ALL_TESTS());
: // Run tests with TCP socket.
: FLAGS_rpc_use_unix_domain_socket = false;
: return RUN_ALL_TESTS();
> Couple thoughts:
Will fix it by using INSTANTIATE_TEST_CASE_P in gtest.
http://gerrit.cloudera.org:8080/#/c/18369/8/be/src/rpc/rpc-mgr.cc
File be/src/rpc/rpc-mgr.cc:
http://gerrit.cloudera.org:8080/#/c/18369/8/be/src/rpc/rpc-mgr.cc@120
PS8, Line 120: else {
: uds_addr_unique_id_ = UdsAddressUniqueIdPB::IP_ADDRESS;
: }
> It would be good to match "ip_address" and produce an error if usd_address_
Should we stop running if usd_address_unique_id is not one of allowed values?
http://gerrit.cloudera.org:8080/#/c/18369/8/be/src/runtime/exec-env.cc
File be/src/runtime/exec-env.cc:
http://gerrit.cloudera.org:8080/#/c/18369/8/be/src/runtime/exec-env.cc@273
PS8, Line 273: // TODO: get BackendId of admissiond. Use a fixed UUID now.
: admissiond_backend_id.set_hi(12345678);
: admissiond_backend_id.set_lo(87654321);
> Will this be addressed in a separate JIRA?
I am not sure if we should support to use backend_id as unique id for UDS. But I saw Kudu to use UUID as unique id for UDS. If we decide to support it, I will address it in a separate JIRA.
http://gerrit.cloudera.org:8080/#/c/18369/8/be/src/util/network-util.cc
File be/src/util/network-util.cc:
http://gerrit.cloudera.org:8080/#/c/18369/8/be/src/util/network-util.cc@198
PS8, Line 198: NetworkAddressPB MakeNetworkAddressPB(const std::string& hostname, int port,
: const UniqueIdPB& backend_id, const UdsAddressUniqueIdPB& uds_addr_unique_id) {
> Would it be useful to have a wrapper function that generates its own random
Good suggestion. Will do.
http://gerrit.cloudera.org:8080/#/c/18369/8/common/protobuf/common.proto
File common/protobuf/common.proto:
http://gerrit.cloudera.org:8080/#/c/18369/8/common/protobuf/common.proto@28
PS8, Line 28: // Unix Domain Socket address. We use the unique name in "Abstract Namespace" as UDS
: // address in the form of "@impala-krpc:<unique-id>". This field is optional.
: // It is only used for KRPC bind/listen/connect when FLAGS_rpc_use_unix_domain_socket
: // is set as true.
: optional string uds_address = 3;
> Should we have a symmetric change in TNetworkAddress? The locations doing F
We only support UDS for KPRC, not for thrift RPC. I think TNetworkAddress don't need to know about UDS.
--
To view, visit http://gerrit.cloudera.org:8080/18369
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7
Gerrit-Change-Number: 18369
Gerrit-PatchSet: 8
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Fri, 20 May 2022 01:18:32 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11129: Support running KRPC over Unix Domain Socket
Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/18369 )
Change subject: IMPALA-11129: Support running KRPC over Unix Domain Socket
......................................................................
Patch Set 9:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/18369/8/be/src/rpc/rpc-mgr.cc
File be/src/rpc/rpc-mgr.cc:
http://gerrit.cloudera.org:8080/#/c/18369/8/be/src/rpc/rpc-mgr.cc@120
PS8, Line 120: else if (strcasecmp("ip_address", FLAGS_uds_address_unique_id.c_str()) == 0) {
: uds_addr_unique_id_ = UdsAddressUniqueIdPB::IP_ADDRESS;
: }
> I see that you disable UDS if the value is invalid.
fixed.
http://gerrit.cloudera.org:8080/#/c/18369/8/common/protobuf/common.proto
File common/protobuf/common.proto:
http://gerrit.cloudera.org:8080/#/c/18369/8/common/protobuf/common.proto@28
PS8, Line 28: // Unix Domain Socket address. We use the unique name in "Abstract Namespace" as UDS
: // address in the form of "@impala-krpc:<unique-id>". This field is optional.
: // It is only used for KRPC bind/listen/connect when FLAGS_rpc_use_unix_domain_socket
: // is set as true.
: optional string uds_address = 3;
> My thought is that sometimes we put KRPC addresses in a Thrift structure to
will add uds_address in TNetworkAddress.
--
To view, visit http://gerrit.cloudera.org:8080/18369
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7
Gerrit-Change-Number: 18369
Gerrit-PatchSet: 9
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Tue, 24 May 2022 18:36:31 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP IMPALA-11129: Support running KRPC over Unix Domain Socket
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18369 )
Change subject: WIP IMPALA-11129: Support running KRPC over Unix Domain Socket
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/10364/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18369
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7
Gerrit-Change-Number: 18369
Gerrit-PatchSet: 2
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Comment-Date: Wed, 30 Mar 2022 17:47:04 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11129: Support running KRPC over Unix Domain Socket
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18369 )
Change subject: IMPALA-11129: Support running KRPC over Unix Domain Socket
......................................................................
Patch Set 3:
(6 comments)
http://gerrit.cloudera.org:8080/#/c/18369/3/tests/custom_cluster/test_krpc_socket.py
File tests/custom_cluster/test_krpc_socket.py:
http://gerrit.cloudera.org:8080/#/c/18369/3/tests/custom_cluster/test_krpc_socket.py@24
PS3, Line 24: class TestKrpcSocket(CustomClusterTestSuite):
flake8: E302 expected 2 blank lines, found 1
http://gerrit.cloudera.org:8080/#/c/18369/3/tests/custom_cluster/test_krpc_socket.py@48
PS3, Line 48: +
flake8: E226 missing whitespace around arithmetic operator
http://gerrit.cloudera.org:8080/#/c/18369/3/tests/custom_cluster/test_krpc_socket.py@49
PS3, Line 49: +
flake8: E226 missing whitespace around arithmetic operator
http://gerrit.cloudera.org:8080/#/c/18369/3/tests/custom_cluster/test_krpc_socket.py@80
PS3, Line 80: +
flake8: E226 missing whitespace around arithmetic operator
http://gerrit.cloudera.org:8080/#/c/18369/3/tests/custom_cluster/test_krpc_socket.py@81
PS3, Line 81: +
flake8: E226 missing whitespace around arithmetic operator
http://gerrit.cloudera.org:8080/#/c/18369/3/tests/custom_cluster/test_krpc_socket.py@91
PS3, Line 91:
flake8: W292 no newline at end of file
--
To view, visit http://gerrit.cloudera.org:8080/18369
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7
Gerrit-Change-Number: 18369
Gerrit-PatchSet: 3
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Comment-Date: Fri, 01 Apr 2022 00:01:55 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP IMPALA-11129: Support running KRPC over Unix Domain Socket
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18369 )
Change subject: WIP IMPALA-11129: Support running KRPC over Unix Domain Socket
......................................................................
Patch Set 1:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/10363/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/18369
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7
Gerrit-Change-Number: 18369
Gerrit-PatchSet: 1
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Comment-Date: Wed, 30 Mar 2022 16:46:20 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11129: Support running KRPC over Unix Domain Socket
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18369 )
Change subject: IMPALA-11129: Support running KRPC over Unix Domain Socket
......................................................................
Patch Set 9:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/10617/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18369
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7
Gerrit-Change-Number: 18369
Gerrit-PatchSet: 9
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Mon, 23 May 2022 14:47:39 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11129: Support running KRPC over Unix Domain Socket
Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has uploaded a new patch set (#10). ( http://gerrit.cloudera.org:8080/18369 )
Change subject: IMPALA-11129: Support running KRPC over Unix Domain Socket
......................................................................
IMPALA-11129: Support running KRPC over Unix Domain Socket
This patch make following changes to support running KRPC over UDS.
- Add FLAGS_rpc_use_unix_domain_socket to enable running KRPC over
UDS. Add FLAGS_uds_address_unique_id to specify unique Id for UDS
address. It could be 'ip_address', 'backend_id', or 'none'.
- Add variable uds_address in NetworkAddressPB and TNetworkAddress.
Replace TNetworkAddress with NetworkAddressPB for KRPC related
class variables and APIs.
- Set UDS address for each daemon as @impala-kprc:<unique_id>
during initialization with unique_id specified by starting flag
FLAGS_uds_address_unique_id.
- When FLAG_rpc_use_unix_domain_socket is true, the socket of KRPC
server will be binded to the UDS address of the daemon.
KRPC Client will connect to KRPC server with the UDS address of
the server when creating proxy service, which in turn call
kudu::Socket::Connect() function to connect KRPC server.
- rpcz Web page show TCP related stats as 'N/A' when using UDS.
Show remote UDS address for KRPC inbound connections on rpcz Web
page as '*' when using UDS since the remote UDS addresses are
not available.
- Add new unit-tests for UDS.
- BackendId of admissiond is not available. Use admissiond's IP
address as unique ID for UDS.
TODO: Advertise BackendId of admissiond in global admission
control mode.
Testing:
- Passed core test with FLAG_rpc_use_unix_domain_socket as fault
value false.
- Passed core test with FLAG_rpc_use_unix_domain_socket as true.
Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7
---
M be/src/benchmarks/expr-benchmark.cc
M be/src/rpc/impala-service-pool.cc
M be/src/rpc/impala-service-pool.h
M be/src/rpc/rpc-mgr-kerberized-test.cc
M be/src/rpc/rpc-mgr-test.cc
M be/src/rpc/rpc-mgr-test.h
M be/src/rpc/rpc-mgr.cc
M be/src/rpc/rpc-mgr.h
M be/src/rpc/rpc-mgr.inline.h
M be/src/runtime/coordinator-backend-state.cc
M be/src/runtime/data-stream-test.cc
M be/src/runtime/exec-env.cc
M be/src/runtime/exec-env.h
M be/src/runtime/krpc-data-stream-mgr.cc
M be/src/runtime/krpc-data-stream-sender.cc
M be/src/runtime/query-driver.cc
M be/src/runtime/query-state.cc
M be/src/runtime/runtime-filter-bank.cc
M be/src/runtime/test-env.cc
M be/src/runtime/tmp-file-mgr.cc
M be/src/scheduling/admissiond-env.cc
M be/src/scheduling/admissiond-env.h
M be/src/service/client-request-state.cc
M be/src/service/control-service.cc
M be/src/service/control-service.h
M be/src/service/data-stream-service.cc
M be/src/service/data-stream-service.h
M be/src/service/impala-server.cc
M be/src/service/impala-server.h
M be/src/testutil/in-process-servers.cc
M be/src/util/container-util.h
M be/src/util/network-util.cc
M be/src/util/network-util.h
M common/protobuf/common.proto
M common/thrift/Types.thrift
A tests/custom_cluster/test_krpc_socket.py
M tests/custom_cluster/test_restart_services.py
M tests/webserver/test_web_pages.py
M www/rpcz.tmpl
39 files changed, 597 insertions(+), 161 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/18369/10
--
To view, visit http://gerrit.cloudera.org:8080/18369
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7
Gerrit-Change-Number: 18369
Gerrit-PatchSet: 10
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
[Impala-ASF-CR] IMPALA-11129: Support running KRPC over Unix Domain Socket
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18369 )
Change subject: IMPALA-11129: Support running KRPC over Unix Domain Socket
......................................................................
Patch Set 7:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/10522/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18369
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7
Gerrit-Change-Number: 18369
Gerrit-PatchSet: 7
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Mon, 02 May 2022 17:09:29 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11129: Support running KRPC over Unix Domain Socket
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/18369 )
Change subject: IMPALA-11129: Support running KRPC over Unix Domain Socket
......................................................................
Patch Set 10: Code-Review+2
Thanks, this looks good to me.
--
To view, visit http://gerrit.cloudera.org:8080/18369
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7
Gerrit-Change-Number: 18369
Gerrit-PatchSet: 10
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Wed, 25 May 2022 00:48:39 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11129: Support running KRPC over Unix Domain Socket
Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/18369 )
Change subject: IMPALA-11129: Support running KRPC over Unix Domain Socket
......................................................................
IMPALA-11129: Support running KRPC over Unix Domain Socket
This patch make following changes to support running KRPC over UDS.
- Add a FLAG_rpc_use_unix_domain_socket to enable running KRPC over
UDS.
- Add field uds_address in NetworkAddressPB.
Replace TNetworkAddress with NetworkAddressPB for RPC related
class variables and APIs.
- Set UDS address for each daemon as @impala-kprc:<BackendId>
during initialization.
- When FLAG_rpc_use_unix_domain_socket is true, the socket of KRPC
server will be binded to the UDS address of the daemon.
KRPC Client will connect to KRPC server with the UDS address of
the server when creating proxy service, which in turn call
kudu::Socket::Connect() function to connect KRPC server.
- rpcz Web page show TCP related stats as 'N/A' when using UDS.
- Add new unit-tests for UDS.
TODO:
- Advertise BackendId of admissiond in global admission control mode.
- Show remote address for KRPC inbound connections on rpcz Web page.
Testing:
- Passed core test with FLAG_rpc_use_unix_domain_socket as fault
value false.
- Passed core test with FLAG_rpc_use_unix_domain_socket as true.
Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7
---
M be/src/benchmarks/expr-benchmark.cc
M be/src/rpc/impala-service-pool.cc
M be/src/rpc/impala-service-pool.h
M be/src/rpc/rpc-mgr-kerberized-test.cc
M be/src/rpc/rpc-mgr-test.cc
M be/src/rpc/rpc-mgr-test.h
M be/src/rpc/rpc-mgr.cc
M be/src/rpc/rpc-mgr.h
M be/src/rpc/rpc-mgr.inline.h
M be/src/runtime/coordinator-backend-state.cc
M be/src/runtime/data-stream-test.cc
M be/src/runtime/exec-env.cc
M be/src/runtime/exec-env.h
M be/src/runtime/krpc-data-stream-mgr.cc
M be/src/runtime/krpc-data-stream-sender.cc
M be/src/runtime/query-driver.cc
M be/src/runtime/query-state.cc
M be/src/runtime/runtime-filter-bank.cc
M be/src/runtime/test-env.cc
M be/src/runtime/tmp-file-mgr.cc
M be/src/scheduling/admissiond-env.cc
M be/src/scheduling/admissiond-env.h
M be/src/service/client-request-state.cc
M be/src/service/control-service.cc
M be/src/service/control-service.h
M be/src/service/data-stream-service.cc
M be/src/service/data-stream-service.h
M be/src/service/impala-server.cc
M be/src/service/impala-server.h
M be/src/testutil/in-process-servers.cc
M be/src/util/container-util.h
M be/src/util/network-util.cc
M be/src/util/network-util.h
M common/protobuf/common.proto
A tests/custom_cluster/test_krpc_socket.py
M tests/custom_cluster/test_restart_services.py
M tests/webserver/test_web_pages.py
M www/rpcz.tmpl
38 files changed, 478 insertions(+), 137 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/18369/4
--
To view, visit http://gerrit.cloudera.org:8080/18369
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7
Gerrit-Change-Number: 18369
Gerrit-PatchSet: 4
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
[Impala-ASF-CR] IMPALA-11129: Support running KRPC over Unix Domain Socket
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18369 )
Change subject: IMPALA-11129: Support running KRPC over Unix Domain Socket
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/10373/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18369
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7
Gerrit-Change-Number: 18369
Gerrit-PatchSet: 3
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Fri, 01 Apr 2022 00:20:28 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11129: Support running KRPC over Unix Domain Socket
Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/18369 )
Change subject: IMPALA-11129: Support running KRPC over Unix Domain Socket
......................................................................
IMPALA-11129: Support running KRPC over Unix Domain Socket
This patch make following changes to support running KRPC over UDS.
- Add FLAGS_rpc_use_unix_domain_socket to enable running KRPC over
UDS. Add FLAGS_uds_address_unique_id to specify unique Id for UDS
address. It could be 'ip_address', 'backend_id', or 'none'.
- Add field uds_address in NetworkAddressPB.
Replace TNetworkAddress with NetworkAddressPB for RPC related
class variables and APIs.
- Set UDS address for each daemon as @impala-kprc:<unique_id>
during initialization with unique_id specified by starting flag
FLAGS_uds_address_unique_id.
- When FLAG_rpc_use_unix_domain_socket is true, the socket of KRPC
server will be binded to the UDS address of the daemon.
KRPC Client will connect to KRPC server with the UDS address of
the server when creating proxy service, which in turn call
kudu::Socket::Connect() function to connect KRPC server.
- rpcz Web page show TCP related stats as 'N/A' when using UDS.
Show remote UDS address for KRPC inbound connections on rpcz Web
page as '*' when using UDS since the remote UDS addresses are
not available.
- Add new unit-tests for UDS.
TODO:
- Advertise BackendId of admissiond in global admission control mode.
Testing:
- Passed core test with FLAG_rpc_use_unix_domain_socket as fault
value false.
- Passed core test with FLAG_rpc_use_unix_domain_socket as true.
Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7
---
M be/src/benchmarks/expr-benchmark.cc
M be/src/rpc/impala-service-pool.cc
M be/src/rpc/impala-service-pool.h
M be/src/rpc/rpc-mgr-kerberized-test.cc
M be/src/rpc/rpc-mgr-test.cc
M be/src/rpc/rpc-mgr-test.h
M be/src/rpc/rpc-mgr.cc
M be/src/rpc/rpc-mgr.h
M be/src/rpc/rpc-mgr.inline.h
M be/src/runtime/coordinator-backend-state.cc
M be/src/runtime/data-stream-test.cc
M be/src/runtime/exec-env.cc
M be/src/runtime/exec-env.h
M be/src/runtime/krpc-data-stream-mgr.cc
M be/src/runtime/krpc-data-stream-sender.cc
M be/src/runtime/query-driver.cc
M be/src/runtime/query-state.cc
M be/src/runtime/runtime-filter-bank.cc
M be/src/runtime/test-env.cc
M be/src/runtime/tmp-file-mgr.cc
M be/src/scheduling/admissiond-env.cc
M be/src/scheduling/admissiond-env.h
M be/src/service/client-request-state.cc
M be/src/service/control-service.cc
M be/src/service/control-service.h
M be/src/service/data-stream-service.cc
M be/src/service/data-stream-service.h
M be/src/service/impala-server.cc
M be/src/service/impala-server.h
M be/src/testutil/in-process-servers.cc
M be/src/util/container-util.h
M be/src/util/network-util.cc
M be/src/util/network-util.h
M common/protobuf/common.proto
A tests/custom_cluster/test_krpc_socket.py
M tests/custom_cluster/test_restart_services.py
M tests/webserver/test_web_pages.py
M www/rpcz.tmpl
38 files changed, 593 insertions(+), 138 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/18369/7
--
To view, visit http://gerrit.cloudera.org:8080/18369
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7
Gerrit-Change-Number: 18369
Gerrit-PatchSet: 7
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
[Impala-ASF-CR] IMPALA-11129: Support running KRPC over Unix Domain Socket
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18369 )
Change subject: IMPALA-11129: Support running KRPC over Unix Domain Socket
......................................................................
Patch Set 6:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/10432/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18369
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7
Gerrit-Change-Number: 18369
Gerrit-PatchSet: 6
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Wed, 13 Apr 2022 03:22:22 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11129: Support running KRPC over Unix Domain Socket
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18369 )
Change subject: IMPALA-11129: Support running KRPC over Unix Domain Socket
......................................................................
Patch Set 11: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/18369
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7
Gerrit-Change-Number: 18369
Gerrit-PatchSet: 11
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Wed, 25 May 2022 01:49:31 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11129: Support running KRPC over Unix Domain Socket
Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/18369 )
Change subject: IMPALA-11129: Support running KRPC over Unix Domain Socket
......................................................................
IMPALA-11129: Support running KRPC over Unix Domain Socket
This patch make following changes to support running KRPC over UDS.
- Add FLAGS_rpc_use_unix_domain_socket to enable running KRPC over
UDS. Add FLAGS_uds_address_unique_id to specify unique Id for UDS
address. It could be 'ip_address', 'backend_id', or 'none'.
- Add field uds_address in NetworkAddressPB.
Replace TNetworkAddress with NetworkAddressPB for RPC related
class variables and APIs.
- Set UDS address for each daemon as @impala-kprc:<unique_id>
during initialization with unique_id specified by starting flag
FLAGS_uds_address_unique_id.
- When FLAG_rpc_use_unix_domain_socket is true, the socket of KRPC
server will be binded to the UDS address of the daemon.
KRPC Client will connect to KRPC server with the UDS address of
the server when creating proxy service, which in turn call
kudu::Socket::Connect() function to connect KRPC server.
- rpcz Web page show TCP related stats as 'N/A' when using UDS.
Show remote UDS address for KRPC inbound connections on rpcz Web
page as '*' when using UDS since the remote UDS addresses are
not available.
- Add new unit-tests for UDS.
TODO:
- Advertise BackendId of admissiond in global admission control mode.
Testing:
- Passed core test with FLAG_rpc_use_unix_domain_socket as fault
value false.
- Passed core test with FLAG_rpc_use_unix_domain_socket as true.
Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7
---
M be/src/benchmarks/expr-benchmark.cc
M be/src/rpc/impala-service-pool.cc
M be/src/rpc/impala-service-pool.h
M be/src/rpc/rpc-mgr-kerberized-test.cc
M be/src/rpc/rpc-mgr-test.cc
M be/src/rpc/rpc-mgr-test.h
M be/src/rpc/rpc-mgr.cc
M be/src/rpc/rpc-mgr.h
M be/src/rpc/rpc-mgr.inline.h
M be/src/runtime/coordinator-backend-state.cc
M be/src/runtime/data-stream-test.cc
M be/src/runtime/exec-env.cc
M be/src/runtime/exec-env.h
M be/src/runtime/krpc-data-stream-mgr.cc
M be/src/runtime/krpc-data-stream-sender.cc
M be/src/runtime/query-driver.cc
M be/src/runtime/query-state.cc
M be/src/runtime/runtime-filter-bank.cc
M be/src/runtime/test-env.cc
M be/src/runtime/tmp-file-mgr.cc
M be/src/scheduling/admissiond-env.cc
M be/src/scheduling/admissiond-env.h
M be/src/service/client-request-state.cc
M be/src/service/control-service.cc
M be/src/service/control-service.h
M be/src/service/data-stream-service.cc
M be/src/service/data-stream-service.h
M be/src/service/impala-server.cc
M be/src/service/impala-server.h
M be/src/testutil/in-process-servers.cc
M be/src/util/container-util.h
M be/src/util/network-util.cc
M be/src/util/network-util.h
M common/protobuf/common.proto
A tests/custom_cluster/test_krpc_socket.py
M tests/custom_cluster/test_restart_services.py
M tests/webserver/test_web_pages.py
M www/rpcz.tmpl
38 files changed, 593 insertions(+), 138 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/18369/8
--
To view, visit http://gerrit.cloudera.org:8080/18369
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7
Gerrit-Change-Number: 18369
Gerrit-PatchSet: 8
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
[Impala-ASF-CR] IMPALA-11129: Support running KRPC over Unix Domain Socket
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18369 )
Change subject: IMPALA-11129: Support running KRPC over Unix Domain Socket
......................................................................
Patch Set 8:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/10574/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18369
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7
Gerrit-Change-Number: 18369
Gerrit-PatchSet: 8
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Fri, 13 May 2022 18:55:55 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11129: Support running KRPC over Unix Domain Socket
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18369 )
Change subject: IMPALA-11129: Support running KRPC over Unix Domain Socket
......................................................................
Patch Set 11: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/18369
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7
Gerrit-Change-Number: 18369
Gerrit-PatchSet: 11
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Wed, 25 May 2022 06:27:23 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11129: Support running KRPC over Unix Domain Socket
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/18369 )
Change subject: IMPALA-11129: Support running KRPC over Unix Domain Socket
......................................................................
Patch Set 8:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/18369/8/be/src/rpc/rpc-mgr-test.cc
File be/src/rpc/rpc-mgr-test.cc:
http://gerrit.cloudera.org:8080/#/c/18369/8/be/src/rpc/rpc-mgr-test.cc@380
PS8, Line 380: // Run tests with Unix domain socket.
: FLAGS_rpc_use_unix_domain_socket = true;
: discard_result(RUN_ALL_TESTS());
: // Run tests with TCP socket.
: FLAGS_rpc_use_unix_domain_socket = false;
: return RUN_ALL_TESTS();
Couple thoughts:
1. We want to notice failures in the UDS case, so we want to care about the return code.
2. One way to handle this is to use the INSTANTIATE_TEST_CASE_P functionality in gtest. This lets you run the same set of tests in different configurations. Then, there is a single RUN_ALL_TESTS() call. See thrift-server-test.cc for some examples (and there are a few others in the codebase).
3. The other way to do this is to exit if the UDS RUN_ALL_TESTS() fails. I think the way it works is that each RUN_ALL_TESTS() invocation would write out a JUnitXML file and the second invocation would overwrite the first. Exiting on failure keeps the JUnitXML state consistent (i.e. it shows the failures), but the downside is that in the success case, the JUnitXML file doesn't contain the first round of tests.
http://gerrit.cloudera.org:8080/#/c/18369/8/be/src/rpc/rpc-mgr.cc
File be/src/rpc/rpc-mgr.cc:
http://gerrit.cloudera.org:8080/#/c/18369/8/be/src/rpc/rpc-mgr.cc@120
PS8, Line 120: else {
: uds_addr_unique_id_ = UdsAddressUniqueIdPB::IP_ADDRESS;
: }
It would be good to match "ip_address" and produce an error if usd_address_unique_id is not one of the allowed values.
http://gerrit.cloudera.org:8080/#/c/18369/8/be/src/runtime/exec-env.cc
File be/src/runtime/exec-env.cc:
http://gerrit.cloudera.org:8080/#/c/18369/8/be/src/runtime/exec-env.cc@273
PS8, Line 273: // TODO: get BackendId of admissiond. Use a fixed UUID now.
: admissiond_backend_id.set_hi(12345678);
: admissiond_backend_id.set_lo(87654321);
Will this be addressed in a separate JIRA?
http://gerrit.cloudera.org:8080/#/c/18369/8/be/src/util/network-util.cc
File be/src/util/network-util.cc:
http://gerrit.cloudera.org:8080/#/c/18369/8/be/src/util/network-util.cc@198
PS8, Line 198: NetworkAddressPB MakeNetworkAddressPB(const std::string& hostname, int port,
: const UniqueIdPB& backend_id, const UdsAddressUniqueIdPB& uds_addr_unique_id) {
Would it be useful to have a wrapper function that generates its own random backend_id and calls MakeNetworkAddressPB with it? There are several spots that do something like this:
UniqueIdPB backend_id;
UUIDToUniqueIdPB(boost::uuids::random_generator()(), &backend_id);
tls_krpc_address = MakeNetworkAddressPB(
ip, tls_service_port, backend_id,
tls_rpc_mgr.GetUdsAddressUniqueId());
http://gerrit.cloudera.org:8080/#/c/18369/8/common/protobuf/common.proto
File common/protobuf/common.proto:
http://gerrit.cloudera.org:8080/#/c/18369/8/common/protobuf/common.proto@28
PS8, Line 28: // Unix Domain Socket address. We use the unique name in "Abstract Namespace" as UDS
: // address in the form of "@impala-krpc:<unique-id>". This field is optional.
: // It is only used for KRPC bind/listen/connect when FLAGS_rpc_use_unix_domain_socket
: // is set as true.
: optional string uds_address = 3;
Should we have a symmetric change in TNetworkAddress? The locations doing FromTNetworkAddress() would be simpler if we didn't need the extra args.
More generally, does TNetworkAddress need to know about UDS?
--
To view, visit http://gerrit.cloudera.org:8080/18369
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7
Gerrit-Change-Number: 18369
Gerrit-PatchSet: 8
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Thu, 19 May 2022 23:47:55 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11129: Support running KRPC over Unix Domain Socket
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18369 )
Change subject: IMPALA-11129: Support running KRPC over Unix Domain Socket
......................................................................
Patch Set 11:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8157/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/18369
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7
Gerrit-Change-Number: 18369
Gerrit-PatchSet: 11
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Wed, 25 May 2022 01:49:32 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11129: Support running KRPC over Unix Domain Socket
Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/18369 )
Change subject: IMPALA-11129: Support running KRPC over Unix Domain Socket
......................................................................
IMPALA-11129: Support running KRPC over Unix Domain Socket
This patch make following changes to support running KRPC over UDS.
- Add a FLAG_rpc_use_unix_domain_socket to enable running KRPC over
UDS.
- Add field uds_address in NetworkAddressPB.
Replace TNetworkAddress with NetworkAddressPB for RPC related
class variables and APIs.
- Set UDS address for each daemon as @impala-kprc:<BackendId>
during initialization.
- When FLAG_rpc_use_unix_domain_socket is true, the socket of KRPC
server will be binded to the UDS address of the daemon.
KRPC Client will connect to KRPC server with the UDS address of
the server when creating proxy service, which in turn call
kudu::Socket::Connect() function to connect KRPC server.
- rpcz Web page show TCP related stats as 'N/A' when using UDS.
- Add new unit-tests for UDS.
TODO:
- Advertise BackendId of admissiond in global admission control mode.
- Show remote address for KRPC inbound connections on rpcz Web page.
Testing:
- Passed core test with FLAG_rpc_use_unix_domain_socket as fault
value false.
- Passed core test with FLAG_rpc_use_unix_domain_socket as true.
Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7
---
M be/src/benchmarks/expr-benchmark.cc
M be/src/rpc/impala-service-pool.cc
M be/src/rpc/impala-service-pool.h
M be/src/rpc/rpc-mgr-kerberized-test.cc
M be/src/rpc/rpc-mgr-test.cc
M be/src/rpc/rpc-mgr-test.h
M be/src/rpc/rpc-mgr.cc
M be/src/rpc/rpc-mgr.h
M be/src/rpc/rpc-mgr.inline.h
M be/src/runtime/coordinator-backend-state.cc
M be/src/runtime/data-stream-test.cc
M be/src/runtime/exec-env.cc
M be/src/runtime/exec-env.h
M be/src/runtime/krpc-data-stream-mgr.cc
M be/src/runtime/krpc-data-stream-sender.cc
M be/src/runtime/query-driver.cc
M be/src/runtime/query-state.cc
M be/src/runtime/runtime-filter-bank.cc
M be/src/runtime/test-env.cc
M be/src/runtime/tmp-file-mgr.cc
M be/src/scheduling/admissiond-env.cc
M be/src/scheduling/admissiond-env.h
M be/src/service/client-request-state.cc
M be/src/service/control-service.cc
M be/src/service/control-service.h
M be/src/service/data-stream-service.cc
M be/src/service/data-stream-service.h
M be/src/service/impala-server.cc
M be/src/service/impala-server.h
M be/src/testutil/in-process-servers.cc
M be/src/util/container-util.h
M be/src/util/network-util.cc
M be/src/util/network-util.h
M common/protobuf/common.proto
A tests/custom_cluster/test_krpc_socket.py
M tests/custom_cluster/test_restart_services.py
M tests/webserver/test_web_pages.py
M www/rpcz.tmpl
38 files changed, 477 insertions(+), 137 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/18369/3
--
To view, visit http://gerrit.cloudera.org:8080/18369
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7
Gerrit-Change-Number: 18369
Gerrit-PatchSet: 3
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
[Impala-ASF-CR] IMPALA-11129: Support running KRPC over Unix Domain Socket
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18369 )
Change subject: IMPALA-11129: Support running KRPC over Unix Domain Socket
......................................................................
Patch Set 5:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/10400/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18369
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I439f5a03eb425c17451bcaa96a154bb0bca17ee7
Gerrit-Change-Number: 18369
Gerrit-PatchSet: 5
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: David Rorke <dr...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Tue, 05 Apr 2022 17:02:38 +0000
Gerrit-HasComments: No