You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@impala.apache.org by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org> on 2022/01/18 17:13:18 UTC

[Impala-ASF-CR] WIP IMPALA-10931 (part2): Fixed rebased Kudu source to compile

Wenzhe Zhou has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18155


Change subject: WIP IMPALA-10931 (part2): Fixed rebased Kudu source to compile
......................................................................

WIP IMPALA-10931 (part2): Fixed rebased Kudu source to compile

This patch applies various fixes to Impala and to the copied Kudu
source code in be/src/kudu/* to allow everything to compile.

Highlights of the changes made:
- Some flags that have a DEFINE in both Kudu and Impala are modified
  to change one of the DEFINEs to a DECLARE.
- Fixed backend unit-test bloom-filter-test.cc due to the changes for
  block bloom filter false positive rate correction in
  kudu/util/block_bloom_filter.cc.
- Set Sockaddr as Sockaddr::Wildcard() if FLAGS_rpc_use_loopback is
  true since the Sockaddr's constructor is changed.

This patch was in part based on the patches that were applied the last
time we rebased the Kudu code in IMPALA-9335. All changes from those
commits that are still relevant were included here.

Went through all commits that have been applied to the be/src/kudu
directory since the last rebase and ensured that all relevant changes
from those are included here. These include patches for IMPALA-9182,
IMPALA-10779, IMPALA-9940 and commit d9a38c0bac (Limit short versions
of KUDU_* macros to Kudu source).

Testing:
- Passed core debug build and core ASAN build except one backend
  test rpc-mgr-kerberized-test.
  TODO: Fix backend unit-test rpc-mgr-kerberized-test.cc
- TODO: Fix end-end test for exhaustive build.

Change-Id: Ia9919c06e60d132d997093abb7b14825847e07c7
---
M be/src/kudu/rpc/CMakeLists.txt
M be/src/kudu/rpc/transfer.cc
M be/src/kudu/security/CMakeLists.txt
M be/src/kudu/security/init.cc
M be/src/kudu/security/test/mini_kdc.cc
M be/src/kudu/security/tls_handshake.cc
M be/src/kudu/util/CMakeLists.txt
M be/src/kudu/util/flags.cc
A be/src/kudu/util/kudu_export.h
M be/src/kudu/util/logging.cc
M be/src/kudu/util/logging.h
M be/src/kudu/util/random_util.h
M be/src/kudu/util/web_callback_registry.h
M be/src/rpc/authentication.cc
M be/src/rpc/rpc-mgr-kerberized-test.cc
M be/src/rpc/rpc-mgr.cc
M be/src/rpc/sidecar-util.h
M be/src/rpc/thrift-server.cc
M be/src/runtime/query-state.cc
M be/src/util/bloom-filter-test.cc
M be/src/util/webserver.cc
21 files changed, 214 insertions(+), 78 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/55/18155/1
-- 
To view, visit http://gerrit.cloudera.org:8080/18155
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ia9919c06e60d132d997093abb7b14825847e07c7
Gerrit-Change-Number: 18155
Gerrit-PatchSet: 1
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>

[Impala-ASF-CR] IMPALA-10931 (part2): Fixed rebased Kudu source to compile

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18155 )

Change subject: IMPALA-10931 (part2): Fixed rebased Kudu source to compile
......................................................................


Patch Set 4:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7837/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/18155
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia9919c06e60d132d997093abb7b14825847e07c7
Gerrit-Change-Number: 18155
Gerrit-PatchSet: 4
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Comment-Date: Sat, 12 Feb 2022 00:32:19 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10931 (part2): Fixed rebased Kudu source to compile

Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.

Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/18155 )

Change subject: IMPALA-10931 (part2): Fixed rebased Kudu source to compile
......................................................................


Patch Set 4:

Hit irrelevant IMPALA-10983.


-- 
To view, visit http://gerrit.cloudera.org:8080/18155
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia9919c06e60d132d997093abb7b14825847e07c7
Gerrit-Change-Number: 18155
Gerrit-PatchSet: 4
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Mon, 14 Feb 2022 05:50:26 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10931 (part2): Fixed rebased Kudu source to compile

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18155 )

Change subject: IMPALA-10931 (part2): Fixed rebased Kudu source to compile
......................................................................


Patch Set 5:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7848/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/18155
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia9919c06e60d132d997093abb7b14825847e07c7
Gerrit-Change-Number: 18155
Gerrit-PatchSet: 5
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Wed, 16 Feb 2022 07:36:17 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10931 (part2): Fixed rebased Kudu source to compile

Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.

Wenzhe Zhou has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/18155 )

Change subject: IMPALA-10931 (part2): Fixed rebased Kudu source to compile
......................................................................

IMPALA-10931 (part2): Fixed rebased Kudu source to compile

This patch applies various fixes to Impala and to the copied Kudu
source code in be/src/kudu/* to allow everything to compile and pass
Impala exhaustive tests.

Highlights of the changes made:
- Some flags that have a DEFINE in both Kudu and Impala are modified
  to change one of the DEFINEs to a DECLARE.
- Fixed backend unit-test bloom-filter-test.cc due to the changes for
  block bloom filter false positive rate correction in
  kudu/util/block_bloom_filter.cc.
- kudu::Sockaddr was changed to support different families of address.
  The storage of address is not cleaned in new constructor. This cause
  the padding bytes in IPv4 address sockaddr_in is not cleaned. We have
  to set initial values of Sockaddr variables as Sockaddr::Wildcard()
  as workaround.
- The parameter type of kudu::RpcSidecar::FromFaststring() was changed,
  need to change the caller code in Impala.
- openssl_util.h was moved from directory kudu/security to kudu/util.
  need to change including path in Impala.

This patch was in part based on the patches that were applied the last
time we rebased the Kudu code in IMPALA-9335. All changes from those
commits that are still relevant were included here.

Went through all commits that have been applied to the be/src/kudu
directory since the last rebase and ensured that all relevant changes
from those are included here. These include patches for IMPALA-9182,
IMPALA-10779, IMPALA-9940 and commit d9a38c0bac (Limit short versions
of KUDU_* macros to Kudu source).

Testing:
- Passed core debug build, core ASAN build and exhaustive release
  build.

Change-Id: Ia9919c06e60d132d997093abb7b14825847e07c7
---
M be/src/kudu/rpc/CMakeLists.txt
M be/src/kudu/rpc/transfer.cc
M be/src/kudu/security/CMakeLists.txt
M be/src/kudu/security/init.cc
M be/src/kudu/security/test/mini_kdc.cc
M be/src/kudu/security/tls_handshake.cc
M be/src/kudu/util/CMakeLists.txt
M be/src/kudu/util/flags.cc
A be/src/kudu/util/kudu_export.h
M be/src/kudu/util/logging.cc
M be/src/kudu/util/logging.h
M be/src/kudu/util/random_util.h
M be/src/kudu/util/web_callback_registry.h
M be/src/rpc/authentication-util.cc
M be/src/rpc/authentication.cc
M be/src/rpc/rpc-mgr.cc
M be/src/rpc/rpc-mgr.inline.h
M be/src/rpc/sidecar-util.h
M be/src/rpc/thrift-server.cc
M be/src/runtime/query-state.cc
M be/src/util/bloom-filter-test.cc
M be/src/util/webserver.cc
22 files changed, 217 insertions(+), 77 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/55/18155/4
-- 
To view, visit http://gerrit.cloudera.org:8080/18155
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia9919c06e60d132d997093abb7b14825847e07c7
Gerrit-Change-Number: 18155
Gerrit-PatchSet: 4
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>

[Impala-ASF-CR] IMPALA-10931 (part2): Fixed rebased Kudu source to compile

Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.

Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/18155 )

Change subject: IMPALA-10931 (part2): Fixed rebased Kudu source to compile
......................................................................


Patch Set 5:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/18155/5/be/src/kudu/security/test/mini_kdc.cc
File be/src/kudu/security/test/mini_kdc.cc:

http://gerrit.cloudera.org:8080/#/c/18155/5/be/src/kudu/security/test/mini_kdc.cc@314
PS5, Line 314:   string tmp_username = username;
             :   StripString(&tmp_username, "/", '_');
             :   const auto tmp_template = Substitute("kinit-temp-$0.XXXXXX", tmp_username);
> The implementation of MiniKdc::Kinit() was changed. In be/src/rpc/rpc-mgr-k
I will ask Kudu team to make this change so we don't need to change Kudu code in next rebase.


http://gerrit.cloudera.org:8080/#/c/18155/5/be/src/kudu/security/tls_handshake.cc
File be/src/kudu/security/tls_handshake.cc:

http://gerrit.cloudera.org:8080/#/c/18155/5/be/src/kudu/security/tls_handshake.cc@316
PS5, Line 316:     const uint8_t* data = reinterpret_cast<const uint8_t*>(rbio_pending_data_.data());
> The first parameter type of Socket.Write() is "const uint8_t *". Got warnin
I will ask Kudu team to make this change so we don't need to change Kudu code in next rebase.



-- 
To view, visit http://gerrit.cloudera.org:8080/18155
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia9919c06e60d132d997093abb7b14825847e07c7
Gerrit-Change-Number: 18155
Gerrit-PatchSet: 5
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Tue, 15 Feb 2022 19:49:50 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10931 (part2): Fixed rebased Kudu source to compile

Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.

Wenzhe Zhou has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18155 )

Change subject: IMPALA-10931 (part2): Fixed rebased Kudu source to compile
......................................................................

IMPALA-10931 (part2): Fixed rebased Kudu source to compile

This patch applies various fixes to Impala and to the copied Kudu
source code in be/src/kudu/* to allow everything to compile and pass
Impala exhaustive tests.

Highlights of the changes made:
- Some flags that have a DEFINE in both Kudu and Impala are modified
  to change one of the DEFINEs to a DECLARE.
- Fixed backend unit-test bloom-filter-test.cc due to the changes for
  block bloom filter false positive rate correction in
  kudu/util/block_bloom_filter.cc.
- Set initial values of Sockaddr variables as Sockaddr::Wildcard().
- The parameter type of kudu::RpcSidecar::FromFaststring() was changed,
  need to change the caller code in Impala.
- openssl_util.h was moved from directory kudu/security to kudu/util.
  need to change including path in Impala.

This patch was in part based on the patches that were applied the last
time we rebased the Kudu code in IMPALA-9335. All changes from those
commits that are still relevant were included here.

Went through all commits that have been applied to the be/src/kudu
directory since the last rebase and ensured that all relevant changes
from those are included here. These include patches for IMPALA-9182,
IMPALA-10779, IMPALA-9940 and commit d9a38c0bac (Limit short versions
of KUDU_* macros to Kudu source).

Testing:
- Passed core debug build, core ASAN build and exhaustive release
  build.

Change-Id: Ia9919c06e60d132d997093abb7b14825847e07c7
Reviewed-on: http://gerrit.cloudera.org:8080/18155
Reviewed-by: Joe McDonnell <jo...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/kudu/rpc/CMakeLists.txt
M be/src/kudu/rpc/transfer.cc
M be/src/kudu/security/CMakeLists.txt
M be/src/kudu/security/init.cc
M be/src/kudu/security/test/mini_kdc.cc
M be/src/kudu/security/tls_handshake.cc
M be/src/kudu/util/CMakeLists.txt
M be/src/kudu/util/flags.cc
A be/src/kudu/util/kudu_export.h
M be/src/kudu/util/logging.cc
M be/src/kudu/util/logging.h
M be/src/kudu/util/random_util.h
M be/src/kudu/util/web_callback_registry.h
M be/src/rpc/authentication-util.cc
M be/src/rpc/authentication.cc
M be/src/rpc/rpc-mgr.cc
M be/src/rpc/rpc-mgr.inline.h
M be/src/rpc/sidecar-util.h
M be/src/rpc/thrift-server.cc
M be/src/runtime/query-state.cc
M be/src/util/bloom-filter-test.cc
M be/src/util/webserver.cc
22 files changed, 217 insertions(+), 77 deletions(-)

Approvals:
  Joe McDonnell: Looks good to me, approved
  Impala Public Jenkins: Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/18155
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ia9919c06e60d132d997093abb7b14825847e07c7
Gerrit-Change-Number: 18155
Gerrit-PatchSet: 6
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>