You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Quanlong Huang (Code Review)" <ge...@cloudera.org> on 2024/03/21 06:41:08 UTC
[Impala-ASF-CR] IMPALA-12929: Skip loading HDFS permissions in local-catalog mode
Quanlong Huang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21178
Change subject: IMPALA-12929: Skip loading HDFS permissions in local-catalog mode
......................................................................
IMPALA-12929: Skip loading HDFS permissions in local-catalog mode
HDFS file/dir permissions are not used at all in local catalog mode - in
LocalFsTable, hasWriteAccessToBaseDir() always returns true and
getFirstLocationWithoutWriteAccess() always returns null.
However, in catalogd, we still load them (in single thread for a table!)
which could dominant the table loading time when there are lots of
partitions.
This patch skips loading HDFS permissions in local catalog mode. We can
revisit it in IMPALA-7539.
Change-Id: I5baa9f6ab0d3888a78ff161ae5caa19e85bc983a
---
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
2 files changed, 8 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/78/21178/1
--
To view, visit http://gerrit.cloudera.org:8080/21178
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I5baa9f6ab0d3888a78ff161ae5caa19e85bc983a
Gerrit-Change-Number: 21178
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-12929: Skip loading HDFS permissions in local-catalog mode
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21178 )
Change subject: IMPALA-12929: Skip loading HDFS permissions in local-catalog mode
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/15601/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/21178
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5baa9f6ab0d3888a78ff161ae5caa19e85bc983a
Gerrit-Change-Number: 21178
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 21 Mar 2024 07:05:02 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12929: Skip loading HDFS permissions in local-catalog mode
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/21178 )
Change subject: IMPALA-12929: Skip loading HDFS permissions in local-catalog mode
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/21178/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/21178/1//COMMIT_MSG@13
PS1, Line 13: catalogd
Can you mention in the commit message the flag that decides this? I was a bit confused, as use_local_catalog is passed only to the coordinators AFAIK, and if I remember the original design correctly, it is allowed to use local catalog in some coordinators, while disabling it in others. Or this is no longer supported?
--
To view, visit http://gerrit.cloudera.org:8080/21178
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5baa9f6ab0d3888a78ff161ae5caa19e85bc983a
Gerrit-Change-Number: 21178
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 21 Mar 2024 16:18:18 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12929: Skip loading HDFS permissions in local-catalog mode
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21178 )
Change subject: IMPALA-12929: Skip loading HDFS permissions in local-catalog mode
......................................................................
Patch Set 3: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/21178
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5baa9f6ab0d3888a78ff161ae5caa19e85bc983a
Gerrit-Change-Number: 21178
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 22 Mar 2024 14:17:57 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12929: Skip loading HDFS permissions in local-catalog mode
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21178 )
Change subject: IMPALA-12929: Skip loading HDFS permissions in local-catalog mode
......................................................................
Patch Set 1:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10401/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/21178
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5baa9f6ab0d3888a78ff161ae5caa19e85bc983a
Gerrit-Change-Number: 21178
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 21 Mar 2024 07:12:48 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12929: Skip loading HDFS permissions in local-catalog mode
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21178 )
Change subject: IMPALA-12929: Skip loading HDFS permissions in local-catalog mode
......................................................................
Patch Set 1: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/21178
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5baa9f6ab0d3888a78ff161ae5caa19e85bc983a
Gerrit-Change-Number: 21178
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 21 Mar 2024 12:19:57 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12929: Skip loading HDFS permissions in local-catalog mode
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Hello Fang-Yu Rao, Csaba Ringhofer, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/21178
to look at the new patch set (#2).
Change subject: IMPALA-12929: Skip loading HDFS permissions in local-catalog mode
......................................................................
IMPALA-12929: Skip loading HDFS permissions in local-catalog mode
HDFS file/dir permissions are not used at all in local catalog mode - in
LocalFsTable, hasWriteAccessToBaseDir() always returns true and
getFirstLocationWithoutWriteAccess() always returns null.
However, in catalogd, we still load them (in single thread for a table!)
which could dominant the table loading time when there are lots of
partitions. Note that the table loading process in catalogd is the same
no matter what catalog mode is in used. The difference between catalog
modes is mainly in how coordinators get metadata from catalogd. Local
catalog mode is turned on by setting --catalog_topic_mode=minimal on
catalogd and --use_local_catalog=true on coordinators.
This patch skips loading HDFS permissions on catalogd when running in
local catalog mode. We can revisit it in IMPALA-7539.
Tests:
- Ran CORE tests
Change-Id: I5baa9f6ab0d3888a78ff161ae5caa19e85bc983a
---
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
2 files changed, 8 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/78/21178/2
--
To view, visit http://gerrit.cloudera.org:8080/21178
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I5baa9f6ab0d3888a78ff161ae5caa19e85bc983a
Gerrit-Change-Number: 21178
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-12929: Skip loading HDFS permissions in local-catalog mode
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21178 )
Change subject: IMPALA-12929: Skip loading HDFS permissions in local-catalog mode
......................................................................
Patch Set 3:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10407/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/21178
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5baa9f6ab0d3888a78ff161ae5caa19e85bc983a
Gerrit-Change-Number: 21178
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 22 Mar 2024 09:22:25 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12929: Skip loading HDFS permissions in local-catalog mode
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/21178 )
Change subject: IMPALA-12929: Skip loading HDFS permissions in local-catalog mode
......................................................................
IMPALA-12929: Skip loading HDFS permissions in local-catalog mode
HDFS file/dir permissions are not used at all in local catalog mode - in
LocalFsTable, hasWriteAccessToBaseDir() always returns true and
getFirstLocationWithoutWriteAccess() always returns null.
However, in catalogd, we still load them (in single thread for a table!)
which could dominant the table loading time when there are lots of
partitions. Note that the table loading process in catalogd is the same
no matter what catalog mode is in used. The difference between catalog
modes is mainly in how coordinators get metadata from catalogd. Local
catalog mode is turned on by setting --catalog_topic_mode=minimal on
catalogd and --use_local_catalog=true on coordinators.
This patch skips loading HDFS permissions on catalogd when running in
local catalog mode. We can revisit it in IMPALA-7539.
Tests:
- Ran CORE tests
Change-Id: I5baa9f6ab0d3888a78ff161ae5caa19e85bc983a
Reviewed-on: http://gerrit.cloudera.org:8080/21178
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
2 files changed, 8 insertions(+), 0 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/21178
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I5baa9f6ab0d3888a78ff161ae5caa19e85bc983a
Gerrit-Change-Number: 21178
Gerrit-PatchSet: 4
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-12929: Skip loading HDFS permissions in local-catalog mode
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/21178 )
Change subject: IMPALA-12929: Skip loading HDFS permissions in local-catalog mode
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/21178/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/21178/1//COMMIT_MSG@13
PS1, Line 13: catalogd
> Can you mention in the commit message the flag that decides this? I was a b
Yes, use_local_catalog is only used by coordinators. Catalogd uses another flag, catalog_topic_mode, and the mixed mode is still supported.
Legacy catalog mode:
- catalogd_args=--catalog_topic_mode=full
- impalad_args=--use_local_catalog=false
Local catalog modoe:
- catalogd_args=--catalog_topic_mode=minimal
- impalad_args=--use_local_catalog=true
Mixed catalog mode:
- catalogd_args=--catalog_topic_mode=mixed
- impalad_args can set use_local_catalog to true or false
--
To view, visit http://gerrit.cloudera.org:8080/21178
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5baa9f6ab0d3888a78ff161ae5caa19e85bc983a
Gerrit-Change-Number: 21178
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 22 Mar 2024 06:06:57 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12929: Skip loading HDFS permissions in local-catalog mode
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/21178 )
Change subject: IMPALA-12929: Skip loading HDFS permissions in local-catalog mode
......................................................................
Patch Set 2: Code-Review+2
(1 comment)
http://gerrit.cloudera.org:8080/#/c/21178/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/21178/1//COMMIT_MSG@13
PS1, Line 13: catalogd
> Yes, use_local_catalog is only used by coordinators. Catalogd uses another
Thanks for the explanation, I forgot about catalog_topic_mode=mixed
--
To view, visit http://gerrit.cloudera.org:8080/21178
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5baa9f6ab0d3888a78ff161ae5caa19e85bc983a
Gerrit-Change-Number: 21178
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 22 Mar 2024 08:16:33 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12929: Skip loading HDFS permissions in local-catalog mode
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21178 )
Change subject: IMPALA-12929: Skip loading HDFS permissions in local-catalog mode
......................................................................
Patch Set 3: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/21178
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5baa9f6ab0d3888a78ff161ae5caa19e85bc983a
Gerrit-Change-Number: 21178
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 22 Mar 2024 09:22:25 +0000
Gerrit-HasComments: No