You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Anonymous Coward (Code Review)" <ge...@cloudera.org> on 2023/12/14 09:27:24 UTC

[Impala-ASF-CR] IMPALA-9375: Remove DirectMetaProvider usage from CatalogMetaProvider

k.venureddy2103@gmail.com has uploaded this change for review. ( http://gerrit.cloudera.org:8080/20791


Change subject: IMPALA-9375: Remove DirectMetaProvider usage from CatalogMetaProvider
......................................................................

IMPALA-9375: Remove DirectMetaProvider usage from CatalogMetaProvider

This commit removes the CatalogMetaProvider dependence on
DirectMetaProvider. CatalogMetaProvider depends on DirectMetaProvider
for 2 APIs. Implemented the APIs on catalog server and used them
instead. DirectMetaProvider is not referenced anywhere now. But it is
retained for future use.

Testing:
- Manually tested and CatalogdMetaProviderTest covers the tests.

Change-Id: I096c1b1d1a52e979c8b2d8173dae9ca2cc6c36d2
---
M be/src/catalog/catalog-server.cc
M be/src/catalog/catalog-service-client-wrapper.h
M be/src/catalog/catalog.cc
M be/src/catalog/catalog.h
M be/src/exec/catalog-op-executor.cc
M be/src/exec/catalog-op-executor.h
M be/src/service/fe-support.cc
M common/thrift/CatalogService.thrift
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/service/FeSupport.java
M fe/src/main/java/org/apache/impala/service/JniCatalog.java
M fe/src/test/java/org/apache/impala/catalog/local/CatalogdMetaProviderTest.java
13 files changed, 371 insertions(+), 43 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/91/20791/1
-- 
To view, visit http://gerrit.cloudera.org:8080/20791
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I096c1b1d1a52e979c8b2d8173dae9ca2cc6c36d2
Gerrit-Change-Number: 20791
Gerrit-PatchSet: 1
Gerrit-Owner: Anonymous Coward <k....@gmail.com>

[Impala-ASF-CR] IMPALA-9375: Remove DirectMetaProvider usage from CatalogMetaProvider

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20791 )

Change subject: IMPALA-9375: Remove DirectMetaProvider usage from CatalogMetaProvider
......................................................................


Patch Set 3:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/14860/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/20791
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I096c1b1d1a52e979c8b2d8173dae9ca2cc6c36d2
Gerrit-Change-Number: 20791
Gerrit-PatchSet: 3
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 03 Jan 2024 09:48:00 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9375: Remove DirectMetaProvider usage from CatalogMetaProvider

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20791 )

Change subject: IMPALA-9375: Remove DirectMetaProvider usage from CatalogMetaProvider
......................................................................


Patch Set 2:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10035/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/20791
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I096c1b1d1a52e979c8b2d8173dae9ca2cc6c36d2
Gerrit-Change-Number: 20791
Gerrit-PatchSet: 2
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 14 Dec 2023 15:33:55 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9375: Remove DirectMetaProvider usage from CatalogMetaProvider

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20791 )

Change subject: IMPALA-9375: Remove DirectMetaProvider usage from CatalogMetaProvider
......................................................................


Patch Set 4:

(4 comments)

http://gerrit.cloudera.org:8080/#/c/20791/4/be/src/catalog/catalog-server.cc
File be/src/catalog/catalog-server.cc:

http://gerrit.cloudera.org:8080/#/c/20791/4/be/src/catalog/catalog-server.cc@368
PS4, Line 368:     VLOG_RPC << "GetNullPartitionName(): request=" << ThriftDebugString(req);
Please make sure you don't output sensitive data with ThriftDebugString(). If so, use impala::RedactedDebugString() instead.


http://gerrit.cloudera.org:8080/#/c/20791/4/be/src/catalog/catalog-server.cc@377
PS4, Line 377:     VLOG_RPC << "GetNullPartitionName(): response=" << ThriftDebugStringNoThrow(resp);
Please make sure you don't output sensitive data with ThriftDebugString(). If so, use impala::RedactedDebugString() instead.


http://gerrit.cloudera.org:8080/#/c/20791/4/be/src/catalog/catalog-server.cc@382
PS4, Line 382:     VLOG_RPC << "GetLatestCompactions(): request=" << ThriftDebugString(req);
Please make sure you don't output sensitive data with ThriftDebugString(). If so, use impala::RedactedDebugString() instead.


http://gerrit.cloudera.org:8080/#/c/20791/4/be/src/catalog/catalog-server.cc@391
PS4, Line 391:     VLOG_RPC << "GetLatestCompactions(): response=" << ThriftDebugStringNoThrow(resp);
Please make sure you don't output sensitive data with ThriftDebugString(). If so, use impala::RedactedDebugString() instead.



-- 
To view, visit http://gerrit.cloudera.org:8080/20791
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I096c1b1d1a52e979c8b2d8173dae9ca2cc6c36d2
Gerrit-Change-Number: 20791
Gerrit-PatchSet: 4
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 03 Jan 2024 11:30:12 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-9375: Remove DirectMetaProvider usage from CatalogMetaProvider

Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Sai Hemanth Gantasala, Csaba Ringhofer, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/20791

to look at the new patch set (#3).

Change subject: IMPALA-9375: Remove DirectMetaProvider usage from CatalogMetaProvider
......................................................................

IMPALA-9375: Remove DirectMetaProvider usage from CatalogMetaProvider

This commit removes the CatalogMetaProvider dependence on
DirectMetaProvider. CatalogMetaProvider depends on DirectMetaProvider
for 2 APIs. Implemented the APIs on catalog server and used them
instead. DirectMetaProvider is not referenced anywhere now. But it is
retained for future use.

Testing:
- Manually tested and CatalogdMetaProviderTest covers the tests.

Change-Id: I096c1b1d1a52e979c8b2d8173dae9ca2cc6c36d2
---
M be/src/catalog/catalog-server.cc
M be/src/catalog/catalog-service-client-wrapper.h
M be/src/catalog/catalog.cc
M be/src/catalog/catalog.h
M be/src/exec/catalog-op-executor.cc
M be/src/exec/catalog-op-executor.h
M be/src/service/fe-support.cc
M common/thrift/CatalogService.thrift
M fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/service/FeSupport.java
M fe/src/main/java/org/apache/impala/service/JniCatalog.java
M fe/src/test/java/org/apache/impala/catalog/local/CatalogdMetaProviderTest.java
14 files changed, 382 insertions(+), 43 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/91/20791/3
-- 
To view, visit http://gerrit.cloudera.org:8080/20791
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I096c1b1d1a52e979c8b2d8173dae9ca2cc6c36d2
Gerrit-Change-Number: 20791
Gerrit-PatchSet: 3
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>

[Impala-ASF-CR] IMPALA-9375: Remove DirectMetaProvider usage from CatalogMetaProvider

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20791 )

Change subject: IMPALA-9375: Remove DirectMetaProvider usage from CatalogMetaProvider
......................................................................


Patch Set 5: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/20791
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I096c1b1d1a52e979c8b2d8173dae9ca2cc6c36d2
Gerrit-Change-Number: 20791
Gerrit-PatchSet: 5
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 04 Jan 2024 14:14:17 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9375: Remove DirectMetaProvider usage from CatalogMetaProvider

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20791 )

Change subject: IMPALA-9375: Remove DirectMetaProvider usage from CatalogMetaProvider
......................................................................


Patch Set 2:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/14720/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/20791
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I096c1b1d1a52e979c8b2d8173dae9ca2cc6c36d2
Gerrit-Change-Number: 20791
Gerrit-PatchSet: 2
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 14 Dec 2023 15:59:48 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9375: Remove DirectMetaProvider usage from CatalogMetaProvider

Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/20791

to look at the new patch set (#2).

Change subject: IMPALA-9375: Remove DirectMetaProvider usage from CatalogMetaProvider
......................................................................

IMPALA-9375: Remove DirectMetaProvider usage from CatalogMetaProvider

This commit removes the CatalogMetaProvider dependence on
DirectMetaProvider. CatalogMetaProvider depends on DirectMetaProvider
for 2 APIs. Implemented the APIs on catalog server and used them
instead. DirectMetaProvider is not referenced anywhere now. But it is
retained for future use.

Testing:
- Manually tested and CatalogdMetaProviderTest covers the tests.

Change-Id: I096c1b1d1a52e979c8b2d8173dae9ca2cc6c36d2
---
M be/src/catalog/catalog-server.cc
M be/src/catalog/catalog-service-client-wrapper.h
M be/src/catalog/catalog.cc
M be/src/catalog/catalog.h
M be/src/exec/catalog-op-executor.cc
M be/src/exec/catalog-op-executor.h
M be/src/service/fe-support.cc
M common/thrift/CatalogService.thrift
M fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/service/FeSupport.java
M fe/src/main/java/org/apache/impala/service/JniCatalog.java
M fe/src/test/java/org/apache/impala/catalog/local/CatalogdMetaProviderTest.java
14 files changed, 378 insertions(+), 43 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/91/20791/2
-- 
To view, visit http://gerrit.cloudera.org:8080/20791
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I096c1b1d1a52e979c8b2d8173dae9ca2cc6c36d2
Gerrit-Change-Number: 20791
Gerrit-PatchSet: 2
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] IMPALA-9375: Remove DirectMetaProvider usage from CatalogMetaProvider

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20791 )

Change subject: IMPALA-9375: Remove DirectMetaProvider usage from CatalogMetaProvider
......................................................................


Patch Set 1:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10034/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/20791
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I096c1b1d1a52e979c8b2d8173dae9ca2cc6c36d2
Gerrit-Change-Number: 20791
Gerrit-PatchSet: 1
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 14 Dec 2023 09:34:14 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9375: Remove DirectMetaProvider usage from CatalogMetaProvider

Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
k.venureddy2103@gmail.com has posted comments on this change. ( http://gerrit.cloudera.org:8080/20791 )

Change subject: IMPALA-9375: Remove DirectMetaProvider usage from CatalogMetaProvider
......................................................................


Patch Set 2:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/20791/2/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java:

http://gerrit.cloudera.org:8080/#/c/20791/2/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@1713
PS2, Line 1713:     private static final String UNPARTITIONED_NAME = "";
> nit: unused?
Used in checkLatestCompaction method.


http://gerrit.cloudera.org:8080/#/c/20791/2/fe/src/main/java/org/apache/impala/service/JniCatalog.java
File fe/src/main/java/org/apache/impala/service/JniCatalog.java:

http://gerrit.cloudera.org:8080/#/c/20791/2/fe/src/main/java/org/apache/impala/service/JniCatalog.java@568
PS2, Line 568:             MetaStoreUtil.getNullPartitionKeyValue(msClient.getHiveClient()));
> I just realized this triggers HMS RPCs in our offline discussion. Will this
HMS may be restarted with new "hive.exec.default.partition.name" value. I think, it is safe to fetch as and when needed. It is a lightweight operation.



-- 
To view, visit http://gerrit.cloudera.org:8080/20791
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I096c1b1d1a52e979c8b2d8173dae9ca2cc6c36d2
Gerrit-Change-Number: 20791
Gerrit-PatchSet: 2
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Mon, 18 Dec 2023 06:32:14 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-9375: Remove DirectMetaProvider usage from CatalogMetaProvider

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20791 )

Change subject: IMPALA-9375: Remove DirectMetaProvider usage from CatalogMetaProvider
......................................................................


Patch Set 4: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/20791
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I096c1b1d1a52e979c8b2d8173dae9ca2cc6c36d2
Gerrit-Change-Number: 20791
Gerrit-PatchSet: 4
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 03 Jan 2024 16:09:20 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9375: Remove DirectMetaProvider usage from CatalogMetaProvider

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/20791 )

Change subject: IMPALA-9375: Remove DirectMetaProvider usage from CatalogMetaProvider
......................................................................


Patch Set 4: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/20791
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I096c1b1d1a52e979c8b2d8173dae9ca2cc6c36d2
Gerrit-Change-Number: 20791
Gerrit-PatchSet: 4
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 04 Jan 2024 14:13:53 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9375: Remove DirectMetaProvider usage from CatalogMetaProvider

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20791 )

Change subject: IMPALA-9375: Remove DirectMetaProvider usage from CatalogMetaProvider
......................................................................


Patch Set 5: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/20791
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I096c1b1d1a52e979c8b2d8173dae9ca2cc6c36d2
Gerrit-Change-Number: 20791
Gerrit-PatchSet: 5
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 04 Jan 2024 18:52:27 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9375: Remove DirectMetaProvider usage from CatalogMetaProvider

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20791 )

Change subject: IMPALA-9375: Remove DirectMetaProvider usage from CatalogMetaProvider
......................................................................


Patch Set 1:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/14715/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/20791
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I096c1b1d1a52e979c8b2d8173dae9ca2cc6c36d2
Gerrit-Change-Number: 20791
Gerrit-PatchSet: 1
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 14 Dec 2023 09:55:36 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9375: Remove DirectMetaProvider usage from CatalogMetaProvider

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20791 )

Change subject: IMPALA-9375: Remove DirectMetaProvider usage from CatalogMetaProvider
......................................................................


Patch Set 2:

(4 comments)

http://gerrit.cloudera.org:8080/#/c/20791/2/be/src/catalog/catalog-server.cc
File be/src/catalog/catalog-server.cc:

http://gerrit.cloudera.org:8080/#/c/20791/2/be/src/catalog/catalog-server.cc@368
PS2, Line 368:     VLOG_RPC << "GetNullPartitionName(): request=" << ThriftDebugString(req);
Please make sure you don't output sensitive data with ThriftDebugString(). If so, use impala::RedactedDebugString() instead.


http://gerrit.cloudera.org:8080/#/c/20791/2/be/src/catalog/catalog-server.cc@377
PS2, Line 377:     VLOG_RPC << "GetNullPartitionName(): response=" << ThriftDebugStringNoThrow(resp);
Please make sure you don't output sensitive data with ThriftDebugString(). If so, use impala::RedactedDebugString() instead.


http://gerrit.cloudera.org:8080/#/c/20791/2/be/src/catalog/catalog-server.cc@382
PS2, Line 382:     VLOG_RPC << "GetLatestCompactions(): request=" << ThriftDebugString(req);
Please make sure you don't output sensitive data with ThriftDebugString(). If so, use impala::RedactedDebugString() instead.


http://gerrit.cloudera.org:8080/#/c/20791/2/be/src/catalog/catalog-server.cc@391
PS2, Line 391:     VLOG_RPC << "GetLatestCompactions(): response=" << ThriftDebugStringNoThrow(resp);
Please make sure you don't output sensitive data with ThriftDebugString(). If so, use impala::RedactedDebugString() instead.



-- 
To view, visit http://gerrit.cloudera.org:8080/20791
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I096c1b1d1a52e979c8b2d8173dae9ca2cc6c36d2
Gerrit-Change-Number: 20791
Gerrit-PatchSet: 2
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 14 Dec 2023 15:33:30 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-9375: Remove DirectMetaProvider usage from CatalogMetaProvider

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20791 )

Change subject: IMPALA-9375: Remove DirectMetaProvider usage from CatalogMetaProvider
......................................................................


Patch Set 2: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/20791
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I096c1b1d1a52e979c8b2d8173dae9ca2cc6c36d2
Gerrit-Change-Number: 20791
Gerrit-PatchSet: 2
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 14 Dec 2023 19:56:28 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9375: Remove DirectMetaProvider usage from CatalogMetaProvider

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20791 )

Change subject: IMPALA-9375: Remove DirectMetaProvider usage from CatalogMetaProvider
......................................................................


Patch Set 5:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10118/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/20791
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I096c1b1d1a52e979c8b2d8173dae9ca2cc6c36d2
Gerrit-Change-Number: 20791
Gerrit-PatchSet: 5
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 04 Jan 2024 14:14:17 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9375: Remove DirectMetaProvider usage from CatalogMetaProvider

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/20791 )

Change subject: IMPALA-9375: Remove DirectMetaProvider usage from CatalogMetaProvider
......................................................................


Patch Set 2: Code-Review+1

(5 comments)

http://gerrit.cloudera.org:8080/#/c/20791/2/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:

http://gerrit.cloudera.org:8080/#/c/20791/2/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@743
PS2, Line 743:     if (lastCompactionId > 0) { request.setLastCompactionId(lastCompactionId); }
nit: don't need brackets "{}" for one-line statement.


http://gerrit.cloudera.org:8080/#/c/20791/2/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@759
PS2, Line 759:       if (ci != null) { partNameToCompactionId.put(unPartitionedName, ci.getId()); }
nit: don't need brackets "{}"


http://gerrit.cloudera.org:8080/#/c/20791/2/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java:

http://gerrit.cloudera.org:8080/#/c/20791/2/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@1019
PS2, Line 1019:     }
It'd be helpful to keep logging the time taken for checking the latest compaction info. I.e. the logging in MetastoreShim#checkLatestCompaction().


http://gerrit.cloudera.org:8080/#/c/20791/2/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@1713
PS2, Line 1713:     private static final String UNPARTITIONED_NAME = "";
> Used in checkLatestCompaction method.
We can use HdfsTable.DEFAULT_PARTITION_NAME which is the name of default partition for unpartitioned tables.


http://gerrit.cloudera.org:8080/#/c/20791/2/fe/src/main/java/org/apache/impala/service/JniCatalog.java
File fe/src/main/java/org/apache/impala/service/JniCatalog.java:

http://gerrit.cloudera.org:8080/#/c/20791/2/fe/src/main/java/org/apache/impala/service/JniCatalog.java@568
PS2, Line 568:             MetaStoreUtil.getNullPartitionKeyValue(msClient.getHiveClient()));
> HMS may be restarted with new "hive.exec.default.partition.name" value. I t
Ack



-- 
To view, visit http://gerrit.cloudera.org:8080/20791
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I096c1b1d1a52e979c8b2d8173dae9ca2cc6c36d2
Gerrit-Change-Number: 20791
Gerrit-PatchSet: 2
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Sat, 30 Dec 2023 05:28:44 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-9375: Remove DirectMetaProvider usage from CatalogMetaProvider

Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Sai Hemanth Gantasala, Csaba Ringhofer, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/20791

to look at the new patch set (#4).

Change subject: IMPALA-9375: Remove DirectMetaProvider usage from CatalogMetaProvider
......................................................................

IMPALA-9375: Remove DirectMetaProvider usage from CatalogMetaProvider

This commit removes the CatalogMetaProvider dependence on
DirectMetaProvider. CatalogMetaProvider depends on DirectMetaProvider
for 2 APIs. Implemented the APIs on catalog server and used them
instead. DirectMetaProvider is not referenced anywhere now. But it is
retained for future use.

Testing:
- Manually tested and CatalogdMetaProviderTest covers the tests.

Change-Id: I096c1b1d1a52e979c8b2d8173dae9ca2cc6c36d2
---
M be/src/catalog/catalog-server.cc
M be/src/catalog/catalog-service-client-wrapper.h
M be/src/catalog/catalog.cc
M be/src/catalog/catalog.h
M be/src/exec/catalog-op-executor.cc
M be/src/exec/catalog-op-executor.h
M be/src/service/fe-support.cc
M common/thrift/CatalogService.thrift
M fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/service/FeSupport.java
M fe/src/main/java/org/apache/impala/service/JniCatalog.java
M fe/src/test/java/org/apache/impala/catalog/local/CatalogdMetaProviderTest.java
14 files changed, 382 insertions(+), 43 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/91/20791/4
-- 
To view, visit http://gerrit.cloudera.org:8080/20791
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I096c1b1d1a52e979c8b2d8173dae9ca2cc6c36d2
Gerrit-Change-Number: 20791
Gerrit-PatchSet: 4
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>

[Impala-ASF-CR] IMPALA-9375: Remove DirectMetaProvider usage from CatalogMetaProvider

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20791 )

Change subject: IMPALA-9375: Remove DirectMetaProvider usage from CatalogMetaProvider
......................................................................


Patch Set 3:

(4 comments)

http://gerrit.cloudera.org:8080/#/c/20791/3/be/src/catalog/catalog-server.cc
File be/src/catalog/catalog-server.cc:

http://gerrit.cloudera.org:8080/#/c/20791/3/be/src/catalog/catalog-server.cc@368
PS3, Line 368:     VLOG_RPC << "GetNullPartitionName(): request=" << ThriftDebugString(req);
Please make sure you don't output sensitive data with ThriftDebugString(). If so, use impala::RedactedDebugString() instead.


http://gerrit.cloudera.org:8080/#/c/20791/3/be/src/catalog/catalog-server.cc@377
PS3, Line 377:     VLOG_RPC << "GetNullPartitionName(): response=" << ThriftDebugStringNoThrow(resp);
Please make sure you don't output sensitive data with ThriftDebugString(). If so, use impala::RedactedDebugString() instead.


http://gerrit.cloudera.org:8080/#/c/20791/3/be/src/catalog/catalog-server.cc@382
PS3, Line 382:     VLOG_RPC << "GetLatestCompactions(): request=" << ThriftDebugString(req);
Please make sure you don't output sensitive data with ThriftDebugString(). If so, use impala::RedactedDebugString() instead.


http://gerrit.cloudera.org:8080/#/c/20791/3/be/src/catalog/catalog-server.cc@391
PS3, Line 391:     VLOG_RPC << "GetLatestCompactions(): response=" << ThriftDebugStringNoThrow(resp);
Please make sure you don't output sensitive data with ThriftDebugString(). If so, use impala::RedactedDebugString() instead.



-- 
To view, visit http://gerrit.cloudera.org:8080/20791
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I096c1b1d1a52e979c8b2d8173dae9ca2cc6c36d2
Gerrit-Change-Number: 20791
Gerrit-PatchSet: 3
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 03 Jan 2024 09:21:53 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-9375: Remove DirectMetaProvider usage from CatalogMetaProvider

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20791 )

Change subject: IMPALA-9375: Remove DirectMetaProvider usage from CatalogMetaProvider
......................................................................


Patch Set 4:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/14861/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/20791
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I096c1b1d1a52e979c8b2d8173dae9ca2cc6c36d2
Gerrit-Change-Number: 20791
Gerrit-PatchSet: 4
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 03 Jan 2024 11:56:07 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9375: Remove DirectMetaProvider usage from CatalogMetaProvider

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20791 )

Change subject: IMPALA-9375: Remove DirectMetaProvider usage from CatalogMetaProvider
......................................................................


Patch Set 1: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10034/


-- 
To view, visit http://gerrit.cloudera.org:8080/20791
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I096c1b1d1a52e979c8b2d8173dae9ca2cc6c36d2
Gerrit-Change-Number: 20791
Gerrit-PatchSet: 1
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 14 Dec 2023 14:04:51 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9375: Remove DirectMetaProvider usage from CatalogMetaProvider

Posted by "Anonymous Coward (Code Review)" <ge...@cloudera.org>.
k.venureddy2103@gmail.com has posted comments on this change. ( http://gerrit.cloudera.org:8080/20791 )

Change subject: IMPALA-9375: Remove DirectMetaProvider usage from CatalogMetaProvider
......................................................................


Patch Set 4:

(4 comments)

Fixed review comments

http://gerrit.cloudera.org:8080/#/c/20791/2/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:

http://gerrit.cloudera.org:8080/#/c/20791/2/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@743
PS2, Line 743:   }
> nit: don't need brackets "{}" for one-line statement.
Done


http://gerrit.cloudera.org:8080/#/c/20791/2/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@759
PS2, Line 759:       for (CompactionInfoStruct ci : response.getCompactions()) {
> nit: don't need brackets "{}"
Done


http://gerrit.cloudera.org:8080/#/c/20791/2/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java:

http://gerrit.cloudera.org:8080/#/c/20791/2/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@1019
PS2, Line 1019:     TGetLatestCompactionsResponse resp = new TGetLatestCompactionsResponse();
> It'd be helpful to keep logging the time taken for checking the latest comp
Done


http://gerrit.cloudera.org:8080/#/c/20791/2/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@1713
PS2, Line 1713:    * Invalidate cached DataSource for the given DataSource name. If anything was
> We can use HdfsTable.DEFAULT_PARTITION_NAME which is the name of default pa
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/20791
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I096c1b1d1a52e979c8b2d8173dae9ca2cc6c36d2
Gerrit-Change-Number: 20791
Gerrit-PatchSet: 4
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Thu, 04 Jan 2024 06:16:54 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-9375: Remove DirectMetaProvider usage from CatalogMetaProvider

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20791 )

Change subject: IMPALA-9375: Remove DirectMetaProvider usage from CatalogMetaProvider
......................................................................


Patch Set 4:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10106/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/20791
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I096c1b1d1a52e979c8b2d8173dae9ca2cc6c36d2
Gerrit-Change-Number: 20791
Gerrit-PatchSet: 4
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Wed, 03 Jan 2024 11:31:23 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9375: Remove DirectMetaProvider usage from CatalogMetaProvider

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20791 )

Change subject: IMPALA-9375: Remove DirectMetaProvider usage from CatalogMetaProvider
......................................................................

IMPALA-9375: Remove DirectMetaProvider usage from CatalogMetaProvider

This commit removes the CatalogMetaProvider dependence on
DirectMetaProvider. CatalogMetaProvider depends on DirectMetaProvider
for 2 APIs. Implemented the APIs on catalog server and used them
instead. DirectMetaProvider is not referenced anywhere now. But it is
retained for future use.

Testing:
- Manually tested and CatalogdMetaProviderTest covers the tests.

Change-Id: I096c1b1d1a52e979c8b2d8173dae9ca2cc6c36d2
Reviewed-on: http://gerrit.cloudera.org:8080/20791
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/catalog/catalog-server.cc
M be/src/catalog/catalog-service-client-wrapper.h
M be/src/catalog/catalog.cc
M be/src/catalog/catalog.h
M be/src/exec/catalog-op-executor.cc
M be/src/exec/catalog-op-executor.h
M be/src/service/fe-support.cc
M common/thrift/CatalogService.thrift
M fe/src/compat-apache-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/service/FeSupport.java
M fe/src/main/java/org/apache/impala/service/JniCatalog.java
M fe/src/test/java/org/apache/impala/catalog/local/CatalogdMetaProviderTest.java
14 files changed, 382 insertions(+), 43 deletions(-)

Approvals:
  Impala Public Jenkins: Looks good to me, approved; Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/20791
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I096c1b1d1a52e979c8b2d8173dae9ca2cc6c36d2
Gerrit-Change-Number: 20791
Gerrit-PatchSet: 6
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>

[Impala-ASF-CR] IMPALA-9375: Remove DirectMetaProvider usage from CatalogMetaProvider

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/20791 )

Change subject: IMPALA-9375: Remove DirectMetaProvider usage from CatalogMetaProvider
......................................................................


Patch Set 2:

(2 comments)

Thanks for clearing this!

http://gerrit.cloudera.org:8080/#/c/20791/2/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java:

http://gerrit.cloudera.org:8080/#/c/20791/2/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java@1713
PS2, Line 1713:     private static final String UNPARTITIONED_NAME = "";
nit: unused?


http://gerrit.cloudera.org:8080/#/c/20791/2/fe/src/main/java/org/apache/impala/service/JniCatalog.java
File fe/src/main/java/org/apache/impala/service/JniCatalog.java:

http://gerrit.cloudera.org:8080/#/c/20791/2/fe/src/main/java/org/apache/impala/service/JniCatalog.java@568
PS2, Line 568:             MetaStoreUtil.getNullPartitionKeyValue(msClient.getHiveClient()));
I just realized this triggers HMS RPCs in our offline discussion. Will this change frequently? If not, maybe we can get it once at startup.



-- 
To view, visit http://gerrit.cloudera.org:8080/20791
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I096c1b1d1a52e979c8b2d8173dae9ca2cc6c36d2
Gerrit-Change-Number: 20791
Gerrit-PatchSet: 2
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Comment-Date: Fri, 15 Dec 2023 07:48:47 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-9375: Remove DirectMetaProvider usage from CatalogMetaProvider

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20791 )

Change subject: IMPALA-9375: Remove DirectMetaProvider usage from CatalogMetaProvider
......................................................................


Patch Set 1:

(4 comments)

http://gerrit.cloudera.org:8080/#/c/20791/1/be/src/catalog/catalog-server.cc
File be/src/catalog/catalog-server.cc:

http://gerrit.cloudera.org:8080/#/c/20791/1/be/src/catalog/catalog-server.cc@368
PS1, Line 368:     VLOG_RPC << "GetNullPartitionName(): request=" << ThriftDebugString(req);
Please make sure you don't output sensitive data with ThriftDebugString(). If so, use impala::RedactedDebugString() instead.


http://gerrit.cloudera.org:8080/#/c/20791/1/be/src/catalog/catalog-server.cc@377
PS1, Line 377:     VLOG_RPC << "GetNullPartitionName(): response=" << ThriftDebugStringNoThrow(resp);
Please make sure you don't output sensitive data with ThriftDebugString(). If so, use impala::RedactedDebugString() instead.


http://gerrit.cloudera.org:8080/#/c/20791/1/be/src/catalog/catalog-server.cc@382
PS1, Line 382:     VLOG_RPC << "GetLatestCompactions(): request=" << ThriftDebugString(req);
Please make sure you don't output sensitive data with ThriftDebugString(). If so, use impala::RedactedDebugString() instead.


http://gerrit.cloudera.org:8080/#/c/20791/1/be/src/catalog/catalog-server.cc@391
PS1, Line 391:     VLOG_RPC << "GetLatestCompactions(): response=" << ThriftDebugStringNoThrow(resp);
Please make sure you don't output sensitive data with ThriftDebugString(). If so, use impala::RedactedDebugString() instead.



-- 
To view, visit http://gerrit.cloudera.org:8080/20791
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I096c1b1d1a52e979c8b2d8173dae9ca2cc6c36d2
Gerrit-Change-Number: 20791
Gerrit-PatchSet: 1
Gerrit-Owner: Anonymous Coward <k....@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 14 Dec 2023 09:28:26 +0000
Gerrit-HasComments: Yes