You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@impala.apache.org by "Xianqing He (Code Review)" <ge...@cloudera.org> on 2022/09/27 11:37:32 UTC

[Impala-ASF-CR] WIP:IMPALA-11558: Ensure one Kudu client created (FE) for the specified Kudu master addresses

Xianqing He has uploaded this change for review. ( http://gerrit.cloudera.org:8080/19046


Change subject: WIP:IMPALA-11558: Ensure one Kudu client created (FE) for the specified Kudu master addresses
......................................................................

WIP:IMPALA-11558: Ensure one Kudu client created (FE) for the specified Kudu master addresses

Creating Kudu clients is very expensive as each will fetch
metadata from the Kudu master, so we can ensure one Kudu
client created for the specified Kudu master addresses.

The solution is to ensure KuduUtil.getKuduClient created only
one KuduClient for the specified Kudu master addresses by using
'computeIfAbsent' of the ConcurrentHashMap.

Testing:
- Manually ran a stress test: scan of a Kudu table, 1000
  concurrent queries and verified the untracked memory
- Ran the full set of verifications in Impala Public Jenkins

Change-Id: I1003556d3afc8e8216142cac4007a4c99046caeb
---
M fe/src/main/java/org/apache/impala/util/KuduUtil.java
1 file changed, 3 insertions(+), 5 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/46/19046/1
-- 
To view, visit http://gerrit.cloudera.org:8080/19046
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I1003556d3afc8e8216142cac4007a4c99046caeb
Gerrit-Change-Number: 19046
Gerrit-PatchSet: 1
Gerrit-Owner: Xianqing He <he...@126.com>

[Impala-ASF-CR] WIP:IMPALA-11558: Ensure one Kudu client created (FE) for the specified Kudu master addresses

Posted by "Xiang Yang (Code Review)" <ge...@cloudera.org>.

Xiang Yang has posted comments on this change. ( http://gerrit.cloudera.org:8080/19046 )

Change subject: WIP:IMPALA-11558: Ensure one Kudu client created (FE) for the specified Kudu master addresses
......................................................................


Patch Set 1:

Hi xianqing, can you add a UT? for example, call the getKuduClient() with a same parameter concurrently and see whether the returned objects's address are same or not.


-- 
To view, visit http://gerrit.cloudera.org:8080/19046
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1003556d3afc8e8216142cac4007a4c99046caeb
Gerrit-Change-Number: 19046
Gerrit-PatchSet: 1
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Comment-Date: Tue, 27 Sep 2022 14:33:06 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11558: Ensure one Kudu client created (FE) for the specified Kudu master addresses

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19046 )

Change subject: IMPALA-11558: Ensure one Kudu client created (FE) for the specified Kudu master addresses
......................................................................


Patch Set 5:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8647/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/19046
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1003556d3afc8e8216142cac4007a4c99046caeb
Gerrit-Change-Number: 19046
Gerrit-PatchSet: 5
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jian Zhang <zj...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Fri, 30 Sep 2022 16:19:50 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11558: Ensure one Kudu client created (FE) for the specified Kudu master addresses

Posted by "Xiang Yang (Code Review)" <ge...@cloudera.org>.

Xiang Yang has posted comments on this change. ( http://gerrit.cloudera.org:8080/19046 )

Change subject: IMPALA-11558: Ensure one Kudu client created (FE) for the specified Kudu master addresses
......................................................................


Patch Set 6:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/19046/5/fe/src/test/java/org/apache/impala/util/KuduUtilTest.java
File fe/src/test/java/org/apache/impala/util/KuduUtilTest.java:

http://gerrit.cloudera.org:8080/#/c/19046/5/fe/src/test/java/org/apache/impala/util/KuduUtilTest.java@57
PS5, Line 57:     assertEquals(0, KuduUtil.getkuduClients().size());
Hi xianqing, I think exposing the mutable map in patch 6 isn't a good practice.
To fix the unit failure, how about calculate the difference of kudu client size?



-- 
To view, visit http://gerrit.cloudera.org:8080/19046
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1003556d3afc8e8216142cac4007a4c99046caeb
Gerrit-Change-Number: 19046
Gerrit-PatchSet: 6
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jian Zhang <zj...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Sun, 09 Oct 2022 07:09:01 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11558: Ensure one Kudu client created (FE) for the specified Kudu master addresses

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19046 )

Change subject: IMPALA-11558: Ensure one Kudu client created (FE) for the specified Kudu master addresses
......................................................................


Patch Set 8: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/19046
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1003556d3afc8e8216142cac4007a4c99046caeb
Gerrit-Change-Number: 19046
Gerrit-PatchSet: 8
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jian Zhang <zj...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Sun, 09 Oct 2022 20:42:06 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11558: Ensure one Kudu client created (FE) for the specified Kudu master addresses

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19046 )

Change subject: IMPALA-11558: Ensure one Kudu client created (FE) for the specified Kudu master addresses
......................................................................


Patch Set 3:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/11466/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/19046
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1003556d3afc8e8216142cac4007a4c99046caeb
Gerrit-Change-Number: 19046
Gerrit-PatchSet: 3
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jian Zhang <zj...@gmail.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Wed, 28 Sep 2022 08:00:31 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11558: Ensure one Kudu client created (FE) for the specified Kudu master addresses

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19046 )

Change subject: IMPALA-11558: Ensure one Kudu client created (FE) for the specified Kudu master addresses
......................................................................


Patch Set 5: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8647/


-- 
To view, visit http://gerrit.cloudera.org:8080/19046
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1003556d3afc8e8216142cac4007a4c99046caeb
Gerrit-Change-Number: 19046
Gerrit-PatchSet: 5
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jian Zhang <zj...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Fri, 30 Sep 2022 21:25:37 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11558: Ensure one Kudu client created (FE) for the specified Kudu master addresses

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19046 )

Change subject: IMPALA-11558: Ensure one Kudu client created (FE) for the specified Kudu master addresses
......................................................................


Patch Set 8: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/19046
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1003556d3afc8e8216142cac4007a4c99046caeb
Gerrit-Change-Number: 19046
Gerrit-PatchSet: 8
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jian Zhang <zj...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Mon, 10 Oct 2022 01:47:19 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11558: Ensure one Kudu client created (FE) for the specified Kudu master addresses

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19046 )

Change subject: IMPALA-11558: Ensure one Kudu client created (FE) for the specified Kudu master addresses
......................................................................


Patch Set 2:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/11463/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/19046
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1003556d3afc8e8216142cac4007a4c99046caeb
Gerrit-Change-Number: 19046
Gerrit-PatchSet: 2
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Wed, 28 Sep 2022 07:02:10 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11558: Ensure one Kudu client created (FE) for the specified Kudu master addresses

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19046 )

Change subject: IMPALA-11558: Ensure one Kudu client created (FE) for the specified Kudu master addresses
......................................................................


Patch Set 4:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/11480/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/19046
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1003556d3afc8e8216142cac4007a4c99046caeb
Gerrit-Change-Number: 19046
Gerrit-PatchSet: 4
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jian Zhang <zj...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Thu, 29 Sep 2022 02:55:07 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11558: Ensure one Kudu client created (FE) for the specified Kudu master addresses

Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.

Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/19046 )

Change subject: IMPALA-11558: Ensure one Kudu client created (FE) for the specified Kudu master addresses
......................................................................


Patch Set 3: Code-Review+1

(1 comment)

http://gerrit.cloudera.org:8080/#/c/19046/3//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/19046/3//COMMIT_MSG@10
PS3, Line 10: so we can ensure one Kudu
            : client created for the specified Kudu master addresses.
nit: so we have to ensure only one Kudu client created for a given Kudu master address.



-- 
To view, visit http://gerrit.cloudera.org:8080/19046
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1003556d3afc8e8216142cac4007a4c99046caeb
Gerrit-Change-Number: 19046
Gerrit-PatchSet: 3
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jian Zhang <zj...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Wed, 28 Sep 2022 11:35:08 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-11558: Ensure one Kudu client created (FE) for the specified Kudu master addresses

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19046 )

Change subject: IMPALA-11558: Ensure one Kudu client created (FE) for the specified Kudu master addresses
......................................................................


Patch Set 5: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/19046
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1003556d3afc8e8216142cac4007a4c99046caeb
Gerrit-Change-Number: 19046
Gerrit-PatchSet: 5
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jian Zhang <zj...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Fri, 30 Sep 2022 16:19:49 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-11558: Ensure one Kudu client created (FE) for the specified Kudu master addresses

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19046 )

Change subject: IMPALA-11558: Ensure one Kudu client created (FE) for the specified Kudu master addresses
......................................................................


Patch Set 6:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/11574/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/19046
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1003556d3afc8e8216142cac4007a4c99046caeb
Gerrit-Change-Number: 19046
Gerrit-PatchSet: 6
Gerrit-Owner: Xianqing He <he...@126.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jian Zhang <zj...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Xiang Yang <yx...@126.com>
Gerrit-Reviewer: Xianqing He <he...@126.com>
Gerrit-Comment-Date: Sun, 09 Oct 2022 06:56:26 +0000
Gerrit-HasComments: No