You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@impala.apache.org by "Sourabh Goyal (Code Review)" <ge...@cloudera.org> on 2021/04/09 21:36:44 UTC

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table cache for hms ddl apis which modify tables and partitions.

Sourabh Goyal has uploaded this change for review. ( http://gerrit.cloudera.org:8080/17298

Change subject: IMPALA-10648: Invalidate catalogd table cache for hms ddl apis which modify tables and partitions.
......................................................................

IMPALA-10648: Invalidate catalogd table cache for hms ddl apis which modify tables and partitions.

For non transactional tables, invalidate the table from cache if HMS DDL apis are accessed
from catalogd's metastore server. Any subsequent get table request fetches the table from
HMS and loads it in cache. This ensures that any get_table/get_partition requests after DDL
operations on the same table return updated table. This behavior has a performance penalty
(since table loading in cache takes time) but ensures consistency. This change is behind catalogd
server's flag: invalidate_hms_cache_on_ddls which is enabled by default. The flag needs to be
turned off if this change becomes a performance bottleneck.

Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
---
M be/src/catalog/catalog-server.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
M tests/custom_cluster/test_metastore_service.py
5 files changed, 416 insertions(+), 48 deletions(-)

git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/98/17298/1
--
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 1
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................


Patch Set 22:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7202/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 22
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Mon, 07 Jun 2021 19:34:37 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table cache for HMS DDL apis which modify tables and partitions.

Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.

Hello Quanlong Huang, Vihang Karajgaonkar, Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/17298

to look at the new patch set (#6).

Change subject: IMPALA-10648: Invalidate catalogd table cache for HMS DDL apis which modify tables and partitions.
......................................................................

IMPALA-10648: Invalidate catalogd table cache for HMS DDL apis
which modify tables and partitions.

For transactional tables, catalogd already guarantees consitent table
metadata reads based on the writeIdList passed in the request. For
non transactional tables, the reads are eventually consistent as in
event processor thread in the background, processes HMS events for the
table and updates its metadata.
In this patch, to ensure strong consistency guarantees for external tables,
we invalidate the table metadata from cache if HMS DDL apis like alter/drop
table/partition are accessed from catalogd's metastore server. As a result
of which, any subsequent get table request fetches the table from HMS and
loads it in cache. This ensures that any get_table/get_partition requests
after DDL operations on the same table return updated table metadata. This
behavior has a performance penalty since metadata loading in cache takes
time specially for large tables. The change is behind catalogd server's
flag: invalidate_hms_cache_on_ddls which is enabled by default. The flag
needs to be turned off in case of a performance bottleneck.

Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
---
M be/src/catalog/catalog-server.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M tests/custom_cluster/test_metastore_service.py
6 files changed, 511 insertions(+), 44 deletions(-)

git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/98/17298/6
--
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 6
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.

Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................


Patch Set 8:

Thanks. The changes looks good to me.


-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 8
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Mon, 26 Apr 2021 20:02:36 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................


Patch Set 17:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7187/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 17
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Wed, 02 Jun 2021 01:18:10 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................


Patch Set 22:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7206/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 22
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Tue, 08 Jun 2021 18:15:52 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................


Patch Set 15:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7183/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 15
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Tue, 01 Jun 2021 17:46:13 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table cache for HMS DDL apis which modify tables and partitions.

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.

Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table cache for HMS DDL apis which modify tables and partitions.
......................................................................


Patch Set 7:

(5 comments)

http://gerrit.cloudera.org:8080/#/c/17298/5//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/17298/5//COMMIT_MSG@7
PS5, Line 7: 
nit, Can you please remove this. It is redundant and makes the title unnecessary long.


http://gerrit.cloudera.org:8080/#/c/17298/5//COMMIT_MSG@9
PS5, Line 9: 
           : For transactio
nit, Can you please reformat this commit msg to 72 line width as per the convention.


http://gerrit.cloudera.org:8080/#/c/17298/7/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
File fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java:

http://gerrit.cloudera.org:8080/#/c/17298/7/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@2966
PS7, Line 2966: removeTableIfExists
I found that removing the table directly from catalog_ doesn't take the metastoreDdlLock_ which can introduce subtle bugs. I had removed this method from CatalogServiceCatalog and added it to CatalogOpExecutor in my patch https://gerrit.cloudera.org/#/c/17308/. Depending on whether we merge https://gerrit.cloudera.org/#/c/17308/ or the current patch first, I think we will have to resolve that conflict one or the other way. I suggest for now add a //TODO here mentioning that this method should be moved to CatalogOpExecutor as suggested in IMPALA-10502. If we merge IMPALA-10502 first in case it takes us long time to unrevert https://gerrit.cloudera.org/#/c/17244/, it should be handled automatically as we resolve the git conflict.


http://gerrit.cloudera.org:8080/#/c/17298/7/tests/custom_cluster/test_metastore_service.py
File tests/custom_cluster/test_metastore_service.py:

http://gerrit.cloudera.org:8080/#/c/17298/7/tests/custom_cluster/test_metastore_service.py@439
PS7, Line 439: removed
nit, s/removed/invalidated


http://gerrit.cloudera.org:8080/#/c/17298/7/tests/custom_cluster/test_metastore_service.py@442
PS7, Line 442: removed
nit, s/removed/invalidated



-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 7
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Wed, 21 Apr 2021 19:55:51 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table cache for hms ddl apis which modify tables and partitions.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table cache for hms ddl apis which modify tables and partitions.
......................................................................


Patch Set 1:

(51 comments)

http://gerrit.cloudera.org:8080/#/c/17298/1/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
File fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java:

http://gerrit.cloudera.org:8080/#/c/17298/1/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@348
PS1, Line 348:     LOG.info("Invalidate catalogd cache for DDLs on non transactional tables is set to {}",
line too long (91 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/1/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@658
PS1, Line 658:       removeNonTransactionalTableIfExists(dbname, tblname, "drop_table_with_environment_context");
line too long (98 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/1/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@843
PS1, Line 843:       invalidateNonTransactionalTableIfExists(newTable.getDbName(), newTable.getTableName(),
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/1/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@890
PS1, Line 890:       Partition addedPartition = client.getHiveClient().getThriftClient().add_partition(partition);
line too long (99 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/1/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@891
PS1, Line 891:       invalidateNonTransactionalTableIfExists(partition.getDbName(), partition.getTableName(),
line too long (94 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/1/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@904
PS1, Line 904:       invalidateNonTransactionalTableIfExists(partition.getDbName(), partition.getTableName(),
line too long (94 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/1/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@914
PS1, Line 914:       int numPartitionsAdded = client.getHiveClient().getThriftClient().add_partitions(partitionList);
line too long (102 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/1/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@916
PS1, Line 916:         invalidateNonTransactionalTableIfExists(partition.getDbName(), partition.getTableName(),
line too long (96 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/1/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@930
PS1, Line 930:         invalidateNonTransactionalTableIfExists(partitionSpec.getDbName(), partitionSpec.getTableName(),
line too long (104 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/1/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@955
PS1, Line 955:       invalidateNonTransactionalTableIfExists(addPartitionsRequest.getDbName(), addPartitionsRequest.getTblName(),
line too long (114 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/1/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@968
PS1, Line 968:               .append_partition_with_environment_context(dbname, tblname, partVals, environmentContext);
line too long (104 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/1/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@994
PS1, Line 994:               .append_partition_by_name_with_environment_context(dbname, tblname, partName,
line too long (91 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/1/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@1021
PS1, Line 1021:               .drop_partition_with_environment_context(dbname, tblname, partNames, deleteData,
line too long (94 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/1/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@1034
PS1, Line 1034:       boolean partitionsDropped =  client.getHiveClient().getThriftClient().drop_partition_by_name(dbname,
line too long (106 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/1/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@1049
PS1, Line 1049:               .drop_partition_by_name_with_environment_context(dbName, tableName, partName,
line too long (91 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/1/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@1064
PS1, Line 1064:       invalidateNonTransactionalTableIfExists(dropPartitionsRequest.getDbName(), dropPartitionsRequest.getTblName(),
line too long (116 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/1/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@1102
PS1, Line 1102:               .exchange_partitions(partitionSpecs, sourceDb, sourceTable, destDb, destinationTableName);
line too long (104 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/1/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@1457
PS1, Line 1457:               .alter_partitions_with_environment_context(dbName, tblName, list, environmentContext);
line too long (100 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/1/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@1458
PS1, Line 1458:       invalidateNonTransactionalTableIfExists(dbName, tblName, "alter_partitions_with_environment_context");
line too long (108 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/1/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@1483
PS1, Line 1483:       invalidateNonTransactionalTableIfExists(dbName, tblName, "alter_partition_with_environment_context");
line too long (107 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/1/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@2846
PS1, Line 2846:   private void invalidateNonTransactionalTableIfExists(String  dbNameWithCatalog, String tableName, String apiName)
line too long (115 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/1/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@2861
PS1, Line 2861:                       "when parsing dbName {} to invalidate/remove table from cache with error message: {}",
line too long (108 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/1/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@2867
PS1, Line 2867:       LOG.debug("Not invalidating non transactional table {}.{} because db {} does not exist in catalogd cache",
line too long (112 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/1/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@2872
PS1, Line 2872:       LOG.debug("Not invalidating non transactional table {}.{} because it does not exist in catalogd's db cache",
line too long (114 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/1/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@2881
PS1, Line 2881:       LOG.debug("Table {} is transactional. Not removing it from catalogd cache", catalogTbl.getFullName());
line too long (108 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/1/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@2890
PS1, Line 2890:       LOG.info("Invalidated non transactional table {} from catalogd cache due to metastore api: {}",
line too long (101 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/1/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@2901
PS1, Line 2901:   private void removeNonTransactionalTableIfExists(String dbNameWithCatalog, String tableName, String apiName)
line too long (110 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/1/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@2916
PS1, Line 2916:                       "when parsing dbName {} to invalidate/remove table from cache with error message: {}",
line too long (108 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/1/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@2927
PS1, Line 2927:       LOG.debug("Not removing table {}.{} because it does not exist in catalogd's db cache",
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/1/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@2934
PS1, Line 2934:       LOG.debug("Table {} is transactional. Not removing it from catalogd cache", catalogTbl.getFullName());
line too long (108 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/1/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@2942
PS1, Line 2942:             catalog_.removeTableIfExists(catalogTbl.getMetaStoreTable(), tableFound, tableMatched);
line too long (99 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/1/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@2944
PS1, Line 2944:       LOG.info("Removed non transactional table {} from catalogd cache due to HMS api: {}",
line too long (91 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/1/tests/custom_cluster/test_metastore_service.py
File tests/custom_cluster/test_metastore_service.py:

http://gerrit.cloudera.org:8080/#/c/17298/1/tests/custom_cluster/test_metastore_service.py@401
PS1, Line 401:  
flake8: E999 IndentationError: unexpected unindent


http://gerrit.cloudera.org:8080/#/c/17298/1/tests/custom_cluster/test_metastore_service.py@433
PS1, Line 433: h
flake8: E501 line too long (94 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/17298/1/tests/custom_cluster/test_metastore_service.py@436
PS1, Line 436: o
flake8: E501 line too long (102 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/17298/1/tests/custom_cluster/test_metastore_service.py@437
PS1, Line 437:  
flake8: E501 line too long (95 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/17298/1/tests/custom_cluster/test_metastore_service.py@438
PS1, Line 438: c
flake8: E501 line too long (95 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/17298/1/tests/custom_cluster/test_metastore_service.py@441
PS1, Line 441: c
flake8: E501 line too long (93 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/17298/1/tests/custom_cluster/test_metastore_service.py@461
PS1, Line 461: a
flake8: E501 line too long (112 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/17298/1/tests/custom_cluster/test_metastore_service.py@462
PS1, Line 462: a
flake8: E501 line too long (112 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/17298/1/tests/custom_cluster/test_metastore_service.py@466
PS1, Line 466: e
flake8: E501 line too long (92 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/17298/1/tests/custom_cluster/test_metastore_service.py@474
PS1, Line 474: ,
flake8: E501 line too long (95 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/17298/1/tests/custom_cluster/test_metastore_service.py@486
PS1, Line 486:  
flake8: E501 line too long (96 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/17298/1/tests/custom_cluster/test_metastore_service.py@489
PS1, Line 489: t
flake8: E501 line too long (92 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/17298/1/tests/custom_cluster/test_metastore_service.py@512
PS1, Line 512:  
flake8: E501 line too long (96 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/17298/1/tests/custom_cluster/test_metastore_service.py@519
PS1, Line 519: e
flake8: E501 line too long (100 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/17298/1/tests/custom_cluster/test_metastore_service.py@526
PS1, Line 526: q
flake8: E501 line too long (96 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/17298/1/tests/custom_cluster/test_metastore_service.py@536
PS1, Line 536:  
flake8: E251 unexpected spaces around keyword / parameter equals


http://gerrit.cloudera.org:8080/#/c/17298/1/tests/custom_cluster/test_metastore_service.py@536
PS1, Line 536:  
flake8: E251 unexpected spaces around keyword / parameter equals


http://gerrit.cloudera.org:8080/#/c/17298/1/tests/custom_cluster/test_metastore_service.py@537
PS1, Line 537:  
flake8: E251 unexpected spaces around keyword / parameter equals


http://gerrit.cloudera.org:8080/#/c/17298/1/tests/custom_cluster/test_metastore_service.py@537
PS1, Line 537:  
flake8: E251 unexpected spaces around keyword / parameter equals



-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 1
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 09 Apr 2021 21:37:30 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.

Vihang Karajgaonkar has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................

IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

For transactional tables, catalogd already guarantees consitent table
metadata reads based on the writeIdList passed in the request. For
non transactional tables, the reads are eventually consistent as in
event processor thread in the background, processes HMS events for the
table and updates its metadata.
In this patch, to ensure strong consistency guarantees for external
tables,we invalidate the table metadata from cache if HMS DDL apis
like alter/drop table/partition are accessed from catalogd's metastore
server. As a result of which, any subsequent get table request fetches
the table from HMS and loads it in cache. This ensures that any
get_table/get_partition requests after DDL operations on same table
return updated table metadata. This behavior has a performance penalty
since metadata loading in cache takes time specially for large tables.
The change is behind catalogd server's flag:
invalidate_hms_cache_on_ddls which is enabled by default. The flag
needs to be turned off in case of a performance bottleneck.

Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Reviewed-on: http://gerrit.cloudera.org:8080/17298
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Vihang Karajgaonkar <vi...@cloudera.com>
---
M be/src/catalog/catalog-server.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M tests/custom_cluster/test_metastore_service.py
6 files changed, 713 insertions(+), 46 deletions(-)

Approvals:
  Impala Public Jenkins: Verified
  Vihang Karajgaonkar: Looks good to me, approved

-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 24
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table cache for hms ddl apis which modify tables and partitions.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table cache for hms ddl apis which modify tables and partitions.
......................................................................


Patch Set 4:

(18 comments)

http://gerrit.cloudera.org:8080/#/c/17298/4/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
File fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java:

http://gerrit.cloudera.org:8080/#/c/17298/4/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@1067
PS4, Line 1067:           client.getHiveClient().getThriftClient().drop_partitions_req(dropPartitionsRequest);
line too long (94 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/4/tests/custom_cluster/test_metastore_service.py
File tests/custom_cluster/test_metastore_service.py:

http://gerrit.cloudera.org:8080/#/c/17298/4/tests/custom_cluster/test_metastore_service.py@470
PS4, Line 470: .
flake8: E131 continuation line unaligned for hanging indent


http://gerrit.cloudera.org:8080/#/c/17298/4/tests/custom_cluster/test_metastore_service.py@476
PS4, Line 476: e
flake8: E501 line too long (92 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/17298/4/tests/custom_cluster/test_metastore_service.py@489
PS4, Line 489: #
flake8: E265 block comment should start with '# '


http://gerrit.cloudera.org:8080/#/c/17298/4/tests/custom_cluster/test_metastore_service.py@490
PS4, Line 490: e
flake8: E501 line too long (92 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/17298/4/tests/custom_cluster/test_metastore_service.py@496
PS4, Line 496:  
flake8: E501 line too long (96 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/17298/4/tests/custom_cluster/test_metastore_service.py@499
PS4, Line 499: t
flake8: E501 line too long (92 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/17298/4/tests/custom_cluster/test_metastore_service.py@526
PS4, Line 526: #
flake8: E303 too many blank lines (2)


http://gerrit.cloudera.org:8080/#/c/17298/4/tests/custom_cluster/test_metastore_service.py@531
PS4, Line 531:  
flake8: E501 line too long (96 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/17298/4/tests/custom_cluster/test_metastore_service.py@533
PS4, Line 533: t
flake8: E501 line too long (92 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/17298/4/tests/custom_cluster/test_metastore_service.py@550
PS4, Line 550: #
flake8: E303 too many blank lines (2)


http://gerrit.cloudera.org:8080/#/c/17298/4/tests/custom_cluster/test_metastore_service.py@554
PS4, Line 554:  
flake8: E501 line too long (96 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/17298/4/tests/custom_cluster/test_metastore_service.py@561
PS4, Line 561: e
flake8: E501 line too long (100 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/17298/4/tests/custom_cluster/test_metastore_service.py@568
PS4, Line 568: q
flake8: E501 line too long (96 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/17298/4/tests/custom_cluster/test_metastore_service.py@577
PS4, Line 577:  
flake8: E251 unexpected spaces around keyword / parameter equals


http://gerrit.cloudera.org:8080/#/c/17298/4/tests/custom_cluster/test_metastore_service.py@577
PS4, Line 577:  
flake8: E251 unexpected spaces around keyword / parameter equals


http://gerrit.cloudera.org:8080/#/c/17298/4/tests/custom_cluster/test_metastore_service.py@578
PS4, Line 578:  
flake8: E251 unexpected spaces around keyword / parameter equals


http://gerrit.cloudera.org:8080/#/c/17298/4/tests/custom_cluster/test_metastore_service.py@578
PS4, Line 578:  
flake8: E251 unexpected spaces around keyword / parameter equals



-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 4
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Comment-Date: Wed, 14 Apr 2021 23:16:43 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.

Hello Quanlong Huang, Vihang Karajgaonkar, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/17298

to look at the new patch set (#18).

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................

IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

For transactional tables, catalogd already guarantees consitent table
metadata reads based on the writeIdList passed in the request. For
non transactional tables, the reads are eventually consistent as in
event processor thread in the background, processes HMS events for the
table and updates its metadata.
In this patch, to ensure strong consistency guarantees for external
tables,we invalidate the table metadata from cache if HMS DDL apis
like alter/drop table/partition are accessed from catalogd's metastore
server. As a result of which, any subsequent get table request fetches
the table from HMS and loads it in cache. This ensures that any
get_table/get_partition requests after DDL operations on same table
return updated table metadata. This behavior has a performance penalty
since metadata loading in cache takes time specially for large tables.
The change is behind catalogd server's flag:
invalidate_hms_cache_on_ddls which is enabled by default. The flag
needs to be turned off in case of a performance bottleneck.

Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
---
M be/src/catalog/catalog-server.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M tests/custom_cluster/test_metastore_service.py
6 files changed, 605 insertions(+), 46 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/98/17298/18
-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 18
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................


Patch Set 17:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/8828/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 17
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Wed, 02 Jun 2021 00:17:56 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................


Patch Set 8:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/8643/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 8
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Mon, 26 Apr 2021 20:03:43 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.

Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................


Patch Set 8:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/17298/8/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
File fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java:

http://gerrit.cloudera.org:8080/#/c/17298/8/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@843
PS8, Line 843: newTable.getTableName()
I just realize that ALTER TABLE RENAME TO is also handled here. So I think we should also call this on the original table name if it differs from the new table. Otherwise, the old table will still exist.


http://gerrit.cloudera.org:8080/#/c/17298/8/tests/custom_cluster/test_metastore_service.py
File tests/custom_cluster/test_metastore_service.py:

http://gerrit.cloudera.org:8080/#/c/17298/8/tests/custom_cluster/test_metastore_service.py@449
PS8, Line 449:         """
Could you add a test for renaming the table name, and verify that the old table disappear and the new table appear?



-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 8
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Thu, 29 Apr 2021 07:49:28 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................


Patch Set 23:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7207/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 23
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Tue, 08 Jun 2021 18:33:59 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................


Patch Set 21: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7192/


-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 21
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Thu, 03 Jun 2021 12:37:41 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table cache for hms ddl apis which modify tables and partitions.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table cache for hms ddl apis which modify tables and partitions.
......................................................................


Patch Set 5:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/8584/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 5
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Thu, 15 Apr 2021 00:40:57 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table cache for hms ddl apis which modify tables and partitions.

Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.

Hello Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/17298

to look at the new patch set (#3).

Change subject: IMPALA-10648: Invalidate catalogd table cache for hms ddl apis which modify tables and partitions.
......................................................................

IMPALA-10648: Invalidate catalogd table cache for hms ddl apis which modify tables and partitions.

Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
---
M be/src/catalog/catalog-server.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M tests/custom_cluster/test_metastore_service.py
6 files changed, 420 insertions(+), 48 deletions(-)

git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/98/17298/3
--
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................


Patch Set 18:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/8834/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 18
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Wed, 02 Jun 2021 17:51:06 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.

Hello Quanlong Huang, Vihang Karajgaonkar, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/17298

to look at the new patch set (#9).

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................

IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

For transactional tables, catalogd already guarantees consitent table
metadata reads based on the writeIdList passed in the request. For
non transactional tables, the reads are eventually consistent as in
event processor thread in the background, processes HMS events for the
table and updates its metadata.
In this patch, to ensure strong consistency guarantees for external
tables,we invalidate the table metadata from cache if HMS DDL apis
like alter/drop table/partition are accessed from catalogd's metastore
server. As a result of which, any subsequent get table request fetches
the table from HMS and loads it in cache. This ensures that any
get_table/get_partition requests after DDL operations on same table
return updated table metadata. This behavior has a performance penalty
since metadata loading in cache takes time specially for large tables.
The change is behind catalogd server's flag:
invalidate_hms_cache_on_ddls which is enabled by default. The flag
needs to be turned off in case of a performance bottleneck.

Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
---
M be/src/catalog/catalog-server.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M tests/custom_cluster/test_metastore_service.py
6 files changed, 519 insertions(+), 46 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/98/17298/9
-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 9
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................


Patch Set 9:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/17298/9/be/src/util/backend-gflag-util.cc
File be/src/util/backend-gflag-util.cc:

http://gerrit.cloudera.org:8080/#/c/17298/9/be/src/util/backend-gflag-util.cc@310
PS9, Line 310: >>>>>>> 6b42204e2... IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
line too long (92 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 9
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Mon, 24 May 2021 23:57:51 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................


Patch Set 15:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/8800/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 15
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Wed, 26 May 2021 23:42:05 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................


Patch Set 15:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/17298/15/tests/custom_cluster/test_metastore_service.py
File tests/custom_cluster/test_metastore_service.py:

http://gerrit.cloudera.org:8080/#/c/17298/15/tests/custom_cluster/test_metastore_service.py@443
PS15, Line 443: e
flake8: E501 line too long (96 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/17298/15/tests/custom_cluster/test_metastore_service.py@444
PS15, Line 444: c
flake8: E501 line too long (93 > 90 characters)



-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 15
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Wed, 26 May 2021 23:21:22 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................


Patch Set 16:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/17298/16/tests/custom_cluster/test_metastore_service.py
File tests/custom_cluster/test_metastore_service.py:

http://gerrit.cloudera.org:8080/#/c/17298/16/tests/custom_cluster/test_metastore_service.py@443
PS16, Line 443: e
flake8: E501 line too long (96 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/17298/16/tests/custom_cluster/test_metastore_service.py@444
PS16, Line 444: c
flake8: E501 line too long (93 > 90 characters)



-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 16
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Tue, 01 Jun 2021 18:55:42 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.

Sourabh Goyal has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................


Patch Set 8:

(4 comments)

http://gerrit.cloudera.org:8080/#/c/17298/5//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/17298/5//COMMIT_MSG@9
PS5, Line 9: For transactional tables, catalogd already guarantees consitent table
           : metadata reads
> nit, Can you please reformat this commit msg to 72 line width as per the co
Ack


http://gerrit.cloudera.org:8080/#/c/17298/7/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
File fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java:

http://gerrit.cloudera.org:8080/#/c/17298/7/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@2966
PS7, Line 2966: n
> I found that removing the table directly from catalog_ doesn't take the met
Sure.


http://gerrit.cloudera.org:8080/#/c/17298/7/tests/custom_cluster/test_metastore_service.py
File tests/custom_cluster/test_metastore_service.py:

http://gerrit.cloudera.org:8080/#/c/17298/7/tests/custom_cluster/test_metastore_service.py@439
PS7, Line 439: invalid
> nit, s/removed/invalidated
Ack


http://gerrit.cloudera.org:8080/#/c/17298/7/tests/custom_cluster/test_metastore_service.py@442
PS7, Line 442: removed
> nit, s/removed/invalidated
For drop case, we remove (and not invalidate) from the cache.



-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 8
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Mon, 26 Apr 2021 19:43:26 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.

Hello Quanlong Huang, Vihang Karajgaonkar, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/17298

to look at the new patch set (#13).

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................

IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

For transactional tables, catalogd already guarantees consitent table
metadata reads based on the writeIdList passed in the request. For
non transactional tables, the reads are eventually consistent as in
event processor thread in the background, processes HMS events for the
table and updates its metadata.
In this patch, to ensure strong consistency guarantees for external
tables,we invalidate the table metadata from cache if HMS DDL apis
like alter/drop table/partition are accessed from catalogd's metastore
server. As a result of which, any subsequent get table request fetches
the table from HMS and loads it in cache. This ensures that any
get_table/get_partition requests after DDL operations on same table
return updated table metadata. This behavior has a performance penalty
since metadata loading in cache takes time specially for large tables.
The change is behind catalogd server's flag:
invalidate_hms_cache_on_ddls which is enabled by default. The flag
needs to be turned off in case of a performance bottleneck.

Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
---
M be/src/catalog/catalog-server.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M tests/custom_cluster/test_metastore_service.py
6 files changed, 512 insertions(+), 46 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/98/17298/13
-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 13
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.

Hello Quanlong Huang, Vihang Karajgaonkar, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/17298

to look at the new patch set (#12).

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................

IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

For transactional tables, catalogd already guarantees consitent table
metadata reads based on the writeIdList passed in the request. For
non transactional tables, the reads are eventually consistent as in
event processor thread in the background, processes HMS events for the
table and updates its metadata.
In this patch, to ensure strong consistency guarantees for external
tables,we invalidate the table metadata from cache if HMS DDL apis
like alter/drop table/partition are accessed from catalogd's metastore
server. As a result of which, any subsequent get table request fetches
the table from HMS and loads it in cache. This ensures that any
get_table/get_partition requests after DDL operations on same table
return updated table metadata. This behavior has a performance penalty
since metadata loading in cache takes time specially for large tables.
The change is behind catalogd server's flag:
invalidate_hms_cache_on_ddls which is enabled by default. The flag
needs to be turned off in case of a performance bottleneck.

Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
---
M be/src/catalog/catalog-server.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M tests/custom_cluster/test_metastore_service.py
6 files changed, 511 insertions(+), 46 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/98/17298/12
-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 12
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.

Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................


Patch Set 21:

> Patch Set 21:
> 
> The failure does not seem to be related to the patch. 
> > Patch Set 21: Verified-1
> > 
> > Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7192/

I think we can resove it by a rebase.


-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 21
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Mon, 07 Jun 2021 12:44:11 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................


Patch Set 17:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/17298/17/tests/custom_cluster/test_metastore_service.py
File tests/custom_cluster/test_metastore_service.py:

http://gerrit.cloudera.org:8080/#/c/17298/17/tests/custom_cluster/test_metastore_service.py@443
PS17, Line 443: e
flake8: E501 line too long (96 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/17298/17/tests/custom_cluster/test_metastore_service.py@444
PS17, Line 444: c
flake8: E501 line too long (93 > 90 characters)



-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 17
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Tue, 01 Jun 2021 23:57:01 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................


Patch Set 22: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 22
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Wed, 09 Jun 2021 00:10:13 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table cache for hms ddl apis which modify tables and partitions.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table cache for hms ddl apis which modify tables and partitions.
......................................................................


Patch Set 4:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/8583/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 4
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Wed, 14 Apr 2021 23:38:50 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................


Patch Set 19:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/17298/19/tests/custom_cluster/test_metastore_service.py
File tests/custom_cluster/test_metastore_service.py:

http://gerrit.cloudera.org:8080/#/c/17298/19/tests/custom_cluster/test_metastore_service.py@696
PS19, Line 696: ,
flake8: E231 missing whitespace after ','



-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 19
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Wed, 02 Jun 2021 19:22:25 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................


Patch Set 22: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7202/


-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 22
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Tue, 08 Jun 2021 01:24:56 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table cache for hms ddl apis which modify tables and partitions.

Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.

Hello Quanlong Huang, Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/17298

to look at the new patch set (#4).

Change subject: IMPALA-10648: Invalidate catalogd table cache for hms ddl apis which modify tables and partitions.
......................................................................

IMPALA-10648: Invalidate catalogd table cache for hms ddl apis which modify tables and partitions.

Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
---
M be/src/catalog/catalog-server.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M tests/custom_cluster/test_metastore_service.py
6 files changed, 510 insertions(+), 44 deletions(-)

git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/98/17298/4
--
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 4
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................


Patch Set 13:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/8783/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 13
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Tue, 25 May 2021 01:05:36 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................


Patch Set 10:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/17298/10/be/src/util/backend-gflag-util.cc
File be/src/util/backend-gflag-util.cc:

http://gerrit.cloudera.org:8080/#/c/17298/10/be/src/util/backend-gflag-util.cc@310
PS10, Line 310: >>>>>>> 6b42204e2... IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
line too long (92 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 10
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Tue, 25 May 2021 00:12:59 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.

Sourabh Goyal has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................


Patch Set 23:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/17298/23/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
File fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java:

http://gerrit.cloudera.org:8080/#/c/17298/23/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@3024
PS23, Line 3024:       LOG.debug("Couldn't rename " + tableInfo);
> Does this need to throw an error?
Good catch! Yes we should throw an error here? Will MetaException be fine?  
 I have a follow up patch planned for drop table issue. I can address this comment in that patch



-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 23
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Wed, 09 Jun 2021 17:38:56 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table cache for hms ddl apis which modify tables and partitions.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table cache for hms ddl apis which modify tables and partitions.
......................................................................


Patch Set 2:

(50 comments)

http://gerrit.cloudera.org:8080/#/c/17298/2/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
File fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java:

http://gerrit.cloudera.org:8080/#/c/17298/2/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@348
PS2, Line 348:     LOG.info("Invalidate catalogd cache for DDLs on non transactional tables is set to {}",
line too long (91 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/2/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@658
PS2, Line 658:       removeNonTransactionalTableIfExists(dbname, tblname, "drop_table_with_environment_context");
line too long (98 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/2/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@843
PS2, Line 843:       invalidateNonTransactionalTableIfExists(newTable.getDbName(), newTable.getTableName(),
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/2/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@890
PS2, Line 890:       Partition addedPartition = client.getHiveClient().getThriftClient().add_partition(partition);
line too long (99 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/2/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@891
PS2, Line 891:       invalidateNonTransactionalTableIfExists(partition.getDbName(), partition.getTableName(),
line too long (94 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/2/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@904
PS2, Line 904:       invalidateNonTransactionalTableIfExists(partition.getDbName(), partition.getTableName(),
line too long (94 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/2/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@914
PS2, Line 914:       int numPartitionsAdded = client.getHiveClient().getThriftClient().add_partitions(partitionList);
line too long (102 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/2/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@916
PS2, Line 916:         invalidateNonTransactionalTableIfExists(partition.getDbName(), partition.getTableName(),
line too long (96 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/2/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@930
PS2, Line 930:         invalidateNonTransactionalTableIfExists(partitionSpec.getDbName(), partitionSpec.getTableName(),
line too long (104 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/2/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@955
PS2, Line 955:       invalidateNonTransactionalTableIfExists(addPartitionsRequest.getDbName(), addPartitionsRequest.getTblName(),
line too long (114 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/2/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@968
PS2, Line 968:               .append_partition_with_environment_context(dbname, tblname, partVals, environmentContext);
line too long (104 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/2/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@994
PS2, Line 994:               .append_partition_by_name_with_environment_context(dbname, tblname, partName,
line too long (91 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/2/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@1021
PS2, Line 1021:               .drop_partition_with_environment_context(dbname, tblname, partNames, deleteData,
line too long (94 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/2/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@1034
PS2, Line 1034:       boolean partitionsDropped =  client.getHiveClient().getThriftClient().drop_partition_by_name(dbname,
line too long (106 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/2/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@1049
PS2, Line 1049:               .drop_partition_by_name_with_environment_context(dbName, tableName, partName,
line too long (91 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/2/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@1064
PS2, Line 1064:       invalidateNonTransactionalTableIfExists(dropPartitionsRequest.getDbName(), dropPartitionsRequest.getTblName(),
line too long (116 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/2/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@1102
PS2, Line 1102:               .exchange_partitions(partitionSpecs, sourceDb, sourceTable, destDb, destinationTableName);
line too long (104 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/2/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@1457
PS2, Line 1457:               .alter_partitions_with_environment_context(dbName, tblName, list, environmentContext);
line too long (100 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/2/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@1458
PS2, Line 1458:       invalidateNonTransactionalTableIfExists(dbName, tblName, "alter_partitions_with_environment_context");
line too long (108 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/2/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@1483
PS2, Line 1483:       invalidateNonTransactionalTableIfExists(dbName, tblName, "alter_partition_with_environment_context");
line too long (107 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/2/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@2846
PS2, Line 2846:   private void invalidateNonTransactionalTableIfExists(String  dbNameWithCatalog, String tableName, String apiName)
line too long (115 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/2/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@2861
PS2, Line 2861:                       "when parsing dbName {} to invalidate/remove table from cache with error message: {}",
line too long (108 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/2/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@2867
PS2, Line 2867:       LOG.debug("Not invalidating non transactional table {}.{} because db {} does not exist in catalogd cache",
line too long (112 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/2/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@2872
PS2, Line 2872:       LOG.debug("Not invalidating non transactional table {}.{} because it does not exist in catalogd's db cache",
line too long (114 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/2/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@2881
PS2, Line 2881:       LOG.debug("Table {} is transactional. Not removing it from catalogd cache", catalogTbl.getFullName());
line too long (108 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/2/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@2890
PS2, Line 2890:       LOG.info("Invalidated non transactional table {} from catalogd cache due to metastore api: {}",
line too long (101 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/2/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@2901
PS2, Line 2901:   private void removeNonTransactionalTableIfExists(String dbNameWithCatalog, String tableName, String apiName)
line too long (110 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/2/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@2916
PS2, Line 2916:                       "when parsing dbName {} to invalidate/remove table from cache with error message: {}",
line too long (108 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/2/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@2927
PS2, Line 2927:       LOG.debug("Not removing table {}.{} because it does not exist in catalogd's db cache",
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/2/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@2934
PS2, Line 2934:       LOG.debug("Table {} is transactional. Not removing it from catalogd cache", catalogTbl.getFullName());
line too long (108 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/2/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@2942
PS2, Line 2942:             catalog_.removeTableIfExists(catalogTbl.getMetaStoreTable(), tableFound, tableMatched);
line too long (99 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/2/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@2944
PS2, Line 2944:       LOG.info("Removed non transactional table {} from catalogd cache due to HMS api: {}",
line too long (91 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/2/tests/custom_cluster/test_metastore_service.py
File tests/custom_cluster/test_metastore_service.py:

http://gerrit.cloudera.org:8080/#/c/17298/2/tests/custom_cluster/test_metastore_service.py@433
PS2, Line 433: h
flake8: E501 line too long (94 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/17298/2/tests/custom_cluster/test_metastore_service.py@436
PS2, Line 436: o
flake8: E501 line too long (102 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/17298/2/tests/custom_cluster/test_metastore_service.py@437
PS2, Line 437:  
flake8: E501 line too long (95 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/17298/2/tests/custom_cluster/test_metastore_service.py@438
PS2, Line 438: c
flake8: E501 line too long (95 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/17298/2/tests/custom_cluster/test_metastore_service.py@441
PS2, Line 441: c
flake8: E501 line too long (93 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/17298/2/tests/custom_cluster/test_metastore_service.py@461
PS2, Line 461: a
flake8: E501 line too long (112 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/17298/2/tests/custom_cluster/test_metastore_service.py@462
PS2, Line 462: a
flake8: E501 line too long (112 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/17298/2/tests/custom_cluster/test_metastore_service.py@466
PS2, Line 466: e
flake8: E501 line too long (92 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/17298/2/tests/custom_cluster/test_metastore_service.py@474
PS2, Line 474: ,
flake8: E501 line too long (95 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/17298/2/tests/custom_cluster/test_metastore_service.py@486
PS2, Line 486:  
flake8: E501 line too long (96 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/17298/2/tests/custom_cluster/test_metastore_service.py@489
PS2, Line 489: t
flake8: E501 line too long (92 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/17298/2/tests/custom_cluster/test_metastore_service.py@512
PS2, Line 512:  
flake8: E501 line too long (96 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/17298/2/tests/custom_cluster/test_metastore_service.py@519
PS2, Line 519: e
flake8: E501 line too long (100 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/17298/2/tests/custom_cluster/test_metastore_service.py@526
PS2, Line 526: q
flake8: E501 line too long (96 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/17298/2/tests/custom_cluster/test_metastore_service.py@536
PS2, Line 536:  
flake8: E251 unexpected spaces around keyword / parameter equals


http://gerrit.cloudera.org:8080/#/c/17298/2/tests/custom_cluster/test_metastore_service.py@536
PS2, Line 536:  
flake8: E251 unexpected spaces around keyword / parameter equals


http://gerrit.cloudera.org:8080/#/c/17298/2/tests/custom_cluster/test_metastore_service.py@537
PS2, Line 537:  
flake8: E251 unexpected spaces around keyword / parameter equals


http://gerrit.cloudera.org:8080/#/c/17298/2/tests/custom_cluster/test_metastore_service.py@537
PS2, Line 537:  
flake8: E251 unexpected spaces around keyword / parameter equals



-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 2
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 09 Apr 2021 21:47:40 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table cache for hms ddl apis which modify tables and partitions.

Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.

Hello Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/17298

to look at the new patch set (#2).

Change subject: IMPALA-10648: Invalidate catalogd table cache for hms ddl apis which modify tables and partitions.
......................................................................

IMPALA-10648: Invalidate catalogd table cache for hms ddl apis which modify tables and partitions.

git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/98/17298/2
--
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table cache for hms ddl apis which modify tables and partitions.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table cache for hms ddl apis which modify tables and partitions.
......................................................................


Patch Set 3:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/8557/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 3
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 12 Apr 2021 22:29:34 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.

Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................


Patch Set 8:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/17298/8/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
File fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java:

http://gerrit.cloudera.org:8080/#/c/17298/8/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@843
PS8, Line 843: newTable.getTableName()
> Good catch ! 
Yes, that makes sense to me. I think we should also do it for other variations of the alter table APIs in this class.



-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 8
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Tue, 25 May 2021 18:07:28 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table cache for HMS DDL apis which modify tables and partitions.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table cache for HMS DDL apis which modify tables and partitions.
......................................................................


Patch Set 7:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/8615/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 7
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Wed, 21 Apr 2021 02:48:24 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table cache for HMS DDL apis which modify tables and partitions.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table cache for HMS DDL apis which modify tables and partitions.
......................................................................


Patch Set 6:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/8614/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 6
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Wed, 21 Apr 2021 01:02:34 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.

Hello Quanlong Huang, Vihang Karajgaonkar, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/17298

to look at the new patch set (#10).

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................

IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

For transactional tables, catalogd already guarantees consitent table
metadata reads based on the writeIdList passed in the request. For
non transactional tables, the reads are eventually consistent as in
event processor thread in the background, processes HMS events for the
table and updates its metadata.
In this patch, to ensure strong consistency guarantees for external
tables,we invalidate the table metadata from cache if HMS DDL apis
like alter/drop table/partition are accessed from catalogd's metastore
server. As a result of which, any subsequent get table request fetches
the table from HMS and loads it in cache. This ensures that any
get_table/get_partition requests after DDL operations on same table
return updated table metadata. This behavior has a performance penalty
since metadata loading in cache takes time specially for large tables.
The change is behind catalogd server's flag:
invalidate_hms_cache_on_ddls which is enabled by default. The flag
needs to be turned off in case of a performance bottleneck.

Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
---
M be/src/catalog/catalog-server.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M tests/custom_cluster/test_metastore_service.py
6 files changed, 519 insertions(+), 46 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/98/17298/10
-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 10
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.

Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................


Patch Set 23: Code-Review+2

(1 comment)

http://gerrit.cloudera.org:8080/#/c/17298/23/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
File fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java:

http://gerrit.cloudera.org:8080/#/c/17298/23/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@3024
PS23, Line 3024:       LOG.debug("Couldn't rename " + tableInfo);
Does this need to throw an error?

I know you have had quite a few iterations of this patch, so I am okay doing this as a followup.



-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 23
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Wed, 09 Jun 2021 17:00:47 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table cache for hms ddl apis which modify tables and partitions.

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.

Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table cache for hms ddl apis which modify tables and partitions.
......................................................................


Patch Set 5:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/17298/3//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/17298/3//COMMIT_MSG@9
PS3, Line 9: For non transactional tables
> Could you explain why we don't do this for transactional tables in the comm
Could you address on this? Maybe you just miss it due to too many code style comments :)


http://gerrit.cloudera.org:8080/#/c/17298/3//COMMIT_MSG@13
PS3, Line 13: (since table loading in cache takes time) but ensures consistency. This change is behind catalogd
> nit: please adjust the commit message body to fit into at-most 72 chars per
Could you address on this? Maybe you just miss it due to too many code style comments :)



-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 5
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Mon, 19 Apr 2021 08:20:18 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.

Hello Quanlong Huang, Vihang Karajgaonkar, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/17298

to look at the new patch set (#21).

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................

IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

For transactional tables, catalogd already guarantees consitent table
metadata reads based on the writeIdList passed in the request. For
non transactional tables, the reads are eventually consistent as in
event processor thread in the background, processes HMS events for the
table and updates its metadata.
In this patch, to ensure strong consistency guarantees for external
tables,we invalidate the table metadata from cache if HMS DDL apis
like alter/drop table/partition are accessed from catalogd's metastore
server. As a result of which, any subsequent get table request fetches
the table from HMS and loads it in cache. This ensures that any
get_table/get_partition requests after DDL operations on same table
return updated table metadata. This behavior has a performance penalty
since metadata loading in cache takes time specially for large tables.
The change is behind catalogd server's flag:
invalidate_hms_cache_on_ddls which is enabled by default. The flag
needs to be turned off in case of a performance bottleneck.

Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
---
M be/src/catalog/catalog-server.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M tests/custom_cluster/test_metastore_service.py
6 files changed, 713 insertions(+), 46 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/98/17298/21
-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 21
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table cache for hms ddl apis which modify tables and partitions.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table cache for hms ddl apis which modify tables and partitions.
......................................................................


Patch Set 1:

Build Failed 

https://jenkins.impala.io/job/gerrit-code-review-checks/8534/ : Initial code review checks failed. See linked job for details on the failure.


-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 1
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 09 Apr 2021 21:47:44 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.

Hello Quanlong Huang, Vihang Karajgaonkar, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/17298

to look at the new patch set (#20).

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................

IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

For transactional tables, catalogd already guarantees consitent table
metadata reads based on the writeIdList passed in the request. For
non transactional tables, the reads are eventually consistent as in
event processor thread in the background, processes HMS events for the
table and updates its metadata.
In this patch, to ensure strong consistency guarantees for external
tables,we invalidate the table metadata from cache if HMS DDL apis
like alter/drop table/partition are accessed from catalogd's metastore
server. As a result of which, any subsequent get table request fetches
the table from HMS and loads it in cache. This ensures that any
get_table/get_partition requests after DDL operations on same table
return updated table metadata. This behavior has a performance penalty
since metadata loading in cache takes time specially for large tables.
The change is behind catalogd server's flag:
invalidate_hms_cache_on_ddls which is enabled by default. The flag
needs to be turned off in case of a performance bottleneck.

Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
---
M be/src/catalog/catalog-server.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M tests/custom_cluster/test_metastore_service.py
6 files changed, 708 insertions(+), 46 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/98/17298/20
-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 20
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table cache for HMS DDL apis which modify tables and partitions.

Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.

Sourabh Goyal has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table cache for HMS DDL apis which modify tables and partitions.
......................................................................


Patch Set 6:

(2 comments)

@Quanlong: I have addressed all your review comments. Please take a look again.

http://gerrit.cloudera.org:8080/#/c/17298/3//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/17298/3//COMMIT_MSG@9
PS3, Line 9: 
> Could you address on this? Maybe you just miss it due to too many code styl
Ack


http://gerrit.cloudera.org:8080/#/c/17298/3//COMMIT_MSG@13
PS3, Line 13: event processor thread in the background, processes HMS events for the
> Could you address on this? Maybe you just miss it due to too many code styl
Ack



-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 6
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Wed, 21 Apr 2021 00:43:51 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.

Sourabh Goyal has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................


Patch Set 17:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/17298/17/tests/custom_cluster/test_metastore_service.py
File tests/custom_cluster/test_metastore_service.py:

http://gerrit.cloudera.org:8080/#/c/17298/17/tests/custom_cluster/test_metastore_service.py@443
PS17, Line 443: e
> flake8: E501 line too long (96 > 90 characters)
Ack


http://gerrit.cloudera.org:8080/#/c/17298/17/tests/custom_cluster/test_metastore_service.py@444
PS17, Line 444: c
> flake8: E501 line too long (93 > 90 characters)
Ack



-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 17
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Wed, 02 Jun 2021 17:28:13 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................


Patch Set 18: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7189/


-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 18
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Wed, 02 Jun 2021 23:26:00 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table cache for hms ddl apis which modify tables and partitions.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table cache for hms ddl apis which modify tables and partitions.
......................................................................


Patch Set 3:

(50 comments)

http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
File fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java:

http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@348
PS3, Line 348:     LOG.info("Invalidate catalogd cache for DDLs on non transactional tables is set to {}",
line too long (91 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@658
PS3, Line 658:       removeNonTransactionalTableIfExists(dbname, tblname, "drop_table_with_environment_context");
line too long (98 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@843
PS3, Line 843:       invalidateNonTransactionalTableIfExists(newTable.getDbName(), newTable.getTableName(),
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@890
PS3, Line 890:       Partition addedPartition = client.getHiveClient().getThriftClient().add_partition(partition);
line too long (99 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@891
PS3, Line 891:       invalidateNonTransactionalTableIfExists(partition.getDbName(), partition.getTableName(),
line too long (94 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@904
PS3, Line 904:       invalidateNonTransactionalTableIfExists(partition.getDbName(), partition.getTableName(),
line too long (94 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@914
PS3, Line 914:       int numPartitionsAdded = client.getHiveClient().getThriftClient().add_partitions(partitionList);
line too long (102 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@916
PS3, Line 916:         invalidateNonTransactionalTableIfExists(partition.getDbName(), partition.getTableName(),
line too long (96 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@930
PS3, Line 930:         invalidateNonTransactionalTableIfExists(partitionSpec.getDbName(), partitionSpec.getTableName(),
line too long (104 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@955
PS3, Line 955:       invalidateNonTransactionalTableIfExists(addPartitionsRequest.getDbName(), addPartitionsRequest.getTblName(),
line too long (114 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@968
PS3, Line 968:               .append_partition_with_environment_context(dbname, tblname, partVals, environmentContext);
line too long (104 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@994
PS3, Line 994:               .append_partition_by_name_with_environment_context(dbname, tblname, partName,
line too long (91 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@1021
PS3, Line 1021:               .drop_partition_with_environment_context(dbname, tblname, partNames, deleteData,
line too long (94 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@1034
PS3, Line 1034:       boolean partitionsDropped =  client.getHiveClient().getThriftClient().drop_partition_by_name(dbname,
line too long (106 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@1049
PS3, Line 1049:               .drop_partition_by_name_with_environment_context(dbName, tableName, partName,
line too long (91 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@1064
PS3, Line 1064:       invalidateNonTransactionalTableIfExists(dropPartitionsRequest.getDbName(), dropPartitionsRequest.getTblName(),
line too long (116 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@1102
PS3, Line 1102:               .exchange_partitions(partitionSpecs, sourceDb, sourceTable, destDb, destinationTableName);
line too long (104 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@1457
PS3, Line 1457:               .alter_partitions_with_environment_context(dbName, tblName, list, environmentContext);
line too long (100 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@1458
PS3, Line 1458:       invalidateNonTransactionalTableIfExists(dbName, tblName, "alter_partitions_with_environment_context");
line too long (108 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@1483
PS3, Line 1483:       invalidateNonTransactionalTableIfExists(dbName, tblName, "alter_partition_with_environment_context");
line too long (107 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@2846
PS3, Line 2846:   private void invalidateNonTransactionalTableIfExists(String  dbNameWithCatalog, String tableName, String apiName)
line too long (115 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@2861
PS3, Line 2861:                       "when parsing dbName {} to invalidate/remove table from cache with error message: {}",
line too long (108 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@2867
PS3, Line 2867:       LOG.debug("Not invalidating non transactional table {}.{} because db {} does not exist in catalogd cache",
line too long (112 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@2872
PS3, Line 2872:       LOG.debug("Not invalidating non transactional table {}.{} because it does not exist in catalogd's db cache",
line too long (114 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@2881
PS3, Line 2881:       LOG.debug("Table {} is transactional. Not removing it from catalogd cache", catalogTbl.getFullName());
line too long (108 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@2890
PS3, Line 2890:       LOG.info("Invalidated non transactional table {} from catalogd cache due to metastore api: {}",
line too long (101 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@2901
PS3, Line 2901:   private void removeNonTransactionalTableIfExists(String dbNameWithCatalog, String tableName, String apiName)
line too long (110 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@2916
PS3, Line 2916:                       "when parsing dbName {} to invalidate/remove table from cache with error message: {}",
line too long (108 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@2927
PS3, Line 2927:       LOG.debug("Not removing table {}.{} because it does not exist in catalogd's db cache",
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@2934
PS3, Line 2934:       LOG.debug("Table {} is transactional. Not removing it from catalogd cache", catalogTbl.getFullName());
line too long (108 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@2942
PS3, Line 2942:             catalog_.removeTableIfExists(catalogTbl.getMetaStoreTable(), tableFound, tableMatched);
line too long (99 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@2944
PS3, Line 2944:       LOG.info("Removed non transactional table {} from catalogd cache due to HMS api: {}",
line too long (91 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/3/tests/custom_cluster/test_metastore_service.py
File tests/custom_cluster/test_metastore_service.py:

http://gerrit.cloudera.org:8080/#/c/17298/3/tests/custom_cluster/test_metastore_service.py@433
PS3, Line 433: h
flake8: E501 line too long (94 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/17298/3/tests/custom_cluster/test_metastore_service.py@436
PS3, Line 436: o
flake8: E501 line too long (102 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/17298/3/tests/custom_cluster/test_metastore_service.py@437
PS3, Line 437:  
flake8: E501 line too long (95 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/17298/3/tests/custom_cluster/test_metastore_service.py@438
PS3, Line 438: c
flake8: E501 line too long (95 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/17298/3/tests/custom_cluster/test_metastore_service.py@441
PS3, Line 441: c
flake8: E501 line too long (93 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/17298/3/tests/custom_cluster/test_metastore_service.py@461
PS3, Line 461: a
flake8: E501 line too long (112 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/17298/3/tests/custom_cluster/test_metastore_service.py@462
PS3, Line 462: a
flake8: E501 line too long (112 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/17298/3/tests/custom_cluster/test_metastore_service.py@466
PS3, Line 466: e
flake8: E501 line too long (92 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/17298/3/tests/custom_cluster/test_metastore_service.py@474
PS3, Line 474: ,
flake8: E501 line too long (95 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/17298/3/tests/custom_cluster/test_metastore_service.py@486
PS3, Line 486:  
flake8: E501 line too long (96 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/17298/3/tests/custom_cluster/test_metastore_service.py@489
PS3, Line 489: t
flake8: E501 line too long (92 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/17298/3/tests/custom_cluster/test_metastore_service.py@512
PS3, Line 512:  
flake8: E501 line too long (96 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/17298/3/tests/custom_cluster/test_metastore_service.py@519
PS3, Line 519: e
flake8: E501 line too long (100 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/17298/3/tests/custom_cluster/test_metastore_service.py@526
PS3, Line 526: q
flake8: E501 line too long (96 > 90 characters)


http://gerrit.cloudera.org:8080/#/c/17298/3/tests/custom_cluster/test_metastore_service.py@536
PS3, Line 536:  
flake8: E251 unexpected spaces around keyword / parameter equals


http://gerrit.cloudera.org:8080/#/c/17298/3/tests/custom_cluster/test_metastore_service.py@536
PS3, Line 536:  
flake8: E251 unexpected spaces around keyword / parameter equals


http://gerrit.cloudera.org:8080/#/c/17298/3/tests/custom_cluster/test_metastore_service.py@537
PS3, Line 537:  
flake8: E251 unexpected spaces around keyword / parameter equals


http://gerrit.cloudera.org:8080/#/c/17298/3/tests/custom_cluster/test_metastore_service.py@537
PS3, Line 537:  
flake8: E251 unexpected spaces around keyword / parameter equals



-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 3
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 12 Apr 2021 22:10:58 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................


Patch Set 20:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/8837/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 20
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Thu, 03 Jun 2021 00:34:56 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................


Patch Set 9:

Build Failed 

https://jenkins.impala.io/job/gerrit-code-review-checks/8779/ : Initial code review checks failed. See linked job for details on the failure.


-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 9
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Tue, 25 May 2021 00:04:33 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................


Patch Set 15: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7183/


-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 15
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Tue, 01 Jun 2021 23:39:28 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.

Hello Quanlong Huang, Vihang Karajgaonkar, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/17298

to look at the new patch set (#15).

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................

IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

For transactional tables, catalogd already guarantees consitent table
metadata reads based on the writeIdList passed in the request. For
non transactional tables, the reads are eventually consistent as in
event processor thread in the background, processes HMS events for the
table and updates its metadata.
In this patch, to ensure strong consistency guarantees for external
tables,we invalidate the table metadata from cache if HMS DDL apis
like alter/drop table/partition are accessed from catalogd's metastore
server. As a result of which, any subsequent get table request fetches
the table from HMS and loads it in cache. This ensures that any
get_table/get_partition requests after DDL operations on same table
return updated table metadata. This behavior has a performance penalty
since metadata loading in cache takes time specially for large tables.
The change is behind catalogd server's flag:
invalidate_hms_cache_on_ddls which is enabled by default. The flag
needs to be turned off in case of a performance bottleneck.

Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
---
M be/src/catalog/catalog-server.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M tests/custom_cluster/test_metastore_service.py
6 files changed, 597 insertions(+), 46 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/98/17298/15
-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 15
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................


Patch Set 23: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 23
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Wed, 09 Jun 2021 00:25:47 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.

Sourabh Goyal has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................


Patch Set 13:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/17298/8/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
File fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java:

http://gerrit.cloudera.org:8080/#/c/17298/8/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@843
PS8, Line 843: e
> I just realize that ALTER TABLE RENAME TO is also handled here. So I think 
Good catch ! 

For alter_table cases, we can instead call renameTable method in CatalogServiceCatalog. The signature of the method is:  

// Renames a table. Equivalent to an atomic drop + add of the table.
public Pair<Table, Table> renameTable(
      TTableName oldTableName, TTableName newTableName)

Thoughts? 

cc - @Vihang


http://gerrit.cloudera.org:8080/#/c/17298/8/tests/custom_cluster/test_metastore_service.py
File tests/custom_cluster/test_metastore_service.py:

http://gerrit.cloudera.org:8080/#/c/17298/8/tests/custom_cluster/test_metastore_service.py@449
PS8, Line 449:         """
> Could you add a test for renaming the table name, and verify that the old t
Sure will add a test for rename table.



-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 13
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Tue, 25 May 2021 17:27:36 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.

Hello Quanlong Huang, Vihang Karajgaonkar, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/17298

to look at the new patch set (#11).

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................

IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

For transactional tables, catalogd already guarantees consitent table
metadata reads based on the writeIdList passed in the request. For
non transactional tables, the reads are eventually consistent as in
event processor thread in the background, processes HMS events for the
table and updates its metadata.
In this patch, to ensure strong consistency guarantees for external
tables,we invalidate the table metadata from cache if HMS DDL apis
like alter/drop table/partition are accessed from catalogd's metastore
server. As a result of which, any subsequent get table request fetches
the table from HMS and loads it in cache. This ensures that any
get_table/get_partition requests after DDL operations on same table
return updated table metadata. This behavior has a performance penalty
since metadata loading in cache takes time specially for large tables.
The change is behind catalogd server's flag:
invalidate_hms_cache_on_ddls which is enabled by default. The flag
needs to be turned off in case of a performance bottleneck.

Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
---
M be/src/catalog/catalog-server.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M tests/custom_cluster/test_metastore_service.py
6 files changed, 510 insertions(+), 46 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/98/17298/11
-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 11
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................


Patch Set 19: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7190/


-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 19
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Thu, 03 Jun 2021 01:45:08 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.

Hello Quanlong Huang, Vihang Karajgaonkar, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/17298

to look at the new patch set (#19).

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................

IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

For transactional tables, catalogd already guarantees consitent table
metadata reads based on the writeIdList passed in the request. For
non transactional tables, the reads are eventually consistent as in
event processor thread in the background, processes HMS events for the
table and updates its metadata.
In this patch, to ensure strong consistency guarantees for external
tables,we invalidate the table metadata from cache if HMS DDL apis
like alter/drop table/partition are accessed from catalogd's metastore
server. As a result of which, any subsequent get table request fetches
the table from HMS and loads it in cache. This ensures that any
get_table/get_partition requests after DDL operations on same table
return updated table metadata. This behavior has a performance penalty
since metadata loading in cache takes time specially for large tables.
The change is behind catalogd server's flag:
invalidate_hms_cache_on_ddls which is enabled by default. The flag
needs to be turned off in case of a performance bottleneck.

Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
---
M be/src/catalog/catalog-server.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M tests/custom_cluster/test_metastore_service.py
6 files changed, 706 insertions(+), 46 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/98/17298/19
-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 19
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................


Patch Set 21:

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7197/


-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 21
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Fri, 04 Jun 2021 03:40:56 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.

Sourabh Goyal has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................


Patch Set 21:

The failure does not seem to be related to the patch. 
> Patch Set 21: Verified-1
> 
> Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7192/


-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 21
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Thu, 03 Jun 2021 21:48:07 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table cache for HMS DDL apis which modify tables and partitions.

Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.

Hello Quanlong Huang, Vihang Karajgaonkar, Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/17298

to look at the new patch set (#7).

Change subject: IMPALA-10648: Invalidate catalogd table cache for HMS DDL apis which modify tables and partitions.
......................................................................

IMPALA-10648: Invalidate catalogd table cache for HMS DDL apis
which modify tables and partitions.

Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
---
M be/src/catalog/catalog-server.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M tests/custom_cluster/test_metastore_service.py
6 files changed, 514 insertions(+), 46 deletions(-)

git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/98/17298/7
--
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 7
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.

Hello Quanlong Huang, Vihang Karajgaonkar, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/17298

to look at the new patch set (#17).

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................

IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

For transactional tables, catalogd already guarantees consitent table
metadata reads based on the writeIdList passed in the request. For
non transactional tables, the reads are eventually consistent as in
event processor thread in the background, processes HMS events for the
table and updates its metadata.
In this patch, to ensure strong consistency guarantees for external
tables,we invalidate the table metadata from cache if HMS DDL apis
like alter/drop table/partition are accessed from catalogd's metastore
server. As a result of which, any subsequent get table request fetches
the table from HMS and loads it in cache. This ensures that any
get_table/get_partition requests after DDL operations on same table
return updated table metadata. This behavior has a performance penalty
since metadata loading in cache takes time specially for large tables.
The change is behind catalogd server's flag:
invalidate_hms_cache_on_ddls which is enabled by default. The flag
needs to be turned off in case of a performance bottleneck.

Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
---
M be/src/catalog/catalog-server.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M tests/custom_cluster/test_metastore_service.py
6 files changed, 604 insertions(+), 46 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/98/17298/17
-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 17
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................


Patch Set 21:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/8840/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 21
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Thu, 03 Jun 2021 06:56:47 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.

Hello Quanlong Huang, Vihang Karajgaonkar, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/17298

to look at the new patch set (#14).

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................

IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

For transactional tables, catalogd already guarantees consitent table
metadata reads based on the writeIdList passed in the request. For
non transactional tables, the reads are eventually consistent as in
event processor thread in the background, processes HMS events for the
table and updates its metadata.
In this patch, to ensure strong consistency guarantees for external
tables,we invalidate the table metadata from cache if HMS DDL apis
like alter/drop table/partition are accessed from catalogd's metastore
server. As a result of which, any subsequent get table request fetches
the table from HMS and loads it in cache. This ensures that any
get_table/get_partition requests after DDL operations on same table
return updated table metadata. This behavior has a performance penalty
since metadata loading in cache takes time specially for large tables.
The change is behind catalogd server's flag:
invalidate_hms_cache_on_ddls which is enabled by default. The flag
needs to be turned off in case of a performance bottleneck.

Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
---
M be/src/catalog/catalog-server.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M tests/custom_cluster/test_metastore_service.py
6 files changed, 596 insertions(+), 46 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/98/17298/14
-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 14
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.

Hello Quanlong Huang, Vihang Karajgaonkar, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/17298

to look at the new patch set (#16).

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................

IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

For transactional tables, catalogd already guarantees consitent table
metadata reads based on the writeIdList passed in the request. For
non transactional tables, the reads are eventually consistent as in
event processor thread in the background, processes HMS events for the
table and updates its metadata.
In this patch, to ensure strong consistency guarantees for external
tables,we invalidate the table metadata from cache if HMS DDL apis
like alter/drop table/partition are accessed from catalogd's metastore
server. As a result of which, any subsequent get table request fetches
the table from HMS and loads it in cache. This ensures that any
get_table/get_partition requests after DDL operations on same table
return updated table metadata. This behavior has a performance penalty
since metadata loading in cache takes time specially for large tables.
The change is behind catalogd server's flag:
invalidate_hms_cache_on_ddls which is enabled by default. The flag
needs to be turned off in case of a performance bottleneck.

Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
---
M be/src/catalog/catalog-server.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M tests/custom_cluster/test_metastore_service.py
6 files changed, 604 insertions(+), 46 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/98/17298/16
-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 16
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................


Patch Set 14:

Build Failed 

https://jenkins.impala.io/job/gerrit-code-review-checks/8799/ : Initial code review checks failed. See linked job for details on the failure.


-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 14
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Wed, 26 May 2021 22:35:17 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................


Patch Set 21:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7197/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 21
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Thu, 03 Jun 2021 21:50:32 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................


Patch Set 11:

Build Failed 

https://jenkins.impala.io/job/gerrit-code-review-checks/8781/ : Initial code review checks failed. See linked job for details on the failure.


-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 11
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Tue, 25 May 2021 00:24:38 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................


Patch Set 20:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/17298/20/tests/custom_cluster/test_metastore_service.py
File tests/custom_cluster/test_metastore_service.py:

http://gerrit.cloudera.org:8080/#/c/17298/20/tests/custom_cluster/test_metastore_service.py@698
PS20, Line 698: ,
flake8: E231 missing whitespace after ','



-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 20
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Thu, 03 Jun 2021 00:15:17 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table cache for hms ddl apis which modify tables and partitions.

Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.

Sourabh Goyal has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table cache for hms ddl apis which modify tables and partitions.
......................................................................


Patch Set 4:

(26 comments)

http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
File fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java:

http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@141
PS3, Line 141: import org.apache.hadoop.hive.metastore.api.LockRequest;
> Is this used somewhere?
No. Will remove it. Thanks for pointing it out.


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@222
PS3, Line 222: UnlockRequest;
> Is this used somewhere?
Will remove it.


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@347
PS3, Line 347:     "is 
> nit: We use 4 spaces indention for multi-line statement. However, many chan
Ack


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@348
PS3, Line 348:   }
> line too long (91 > 90)
Ack


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@349
PS3, Line 349: 
> nit: 4 spaces indention
Ack


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@890
PS3, Line 890:           client.getHiveClient().getThriftClient().add_partition(partition);
> line too long (99 > 90)
Ack


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@891
PS3, Line 891:       invalidateNonTransactionalTableIfExists(partition.getDbName(),
> line too long (94 > 90)
Ack


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@904
PS3, Line 904:       invalidateNonTransactionalTableIfExists(partition.getDbName(),
> line too long (94 > 90)
Ack


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@914
PS3, Line 914:     try (MetaStoreClient client = catalog_.getMetaStoreClient()) {
> line too long (102 > 90)
Ack


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@915
PS3, Line 915: 
> Is it possible that these partitions belong to different tables?
Very likely the partitions belong to the same table. I will confirm and make the change accordingly. 
Thanks for pointing it out.


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@929
PS3, Line 929: int numPartitionsAdded =  client.getHiveCl
> Same question as above
Will check this one too.


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@930
PS3, Line 930:               .getThriftClient().add_partitions_pspec(list);
> line too long (104 > 90)
Ack


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@955
PS3, Line 955:       AddPartitionsResult result =  client.getHiveClient().getThriftClient()
> line too long (114 > 90)
Ack


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@968
PS3, Line 968:     try (MetaStoreClient client = catalog_.getMetaStoreClient()) {
> line too long (104 > 90)
Ack


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@994
PS3, Line 994:       throws InvalidObjectException, AlreadyExistsException, MetaException, TException {
> line too long (91 > 90)
Ack


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@1021
PS3, Line 1021:       throws NoSuchObjectException, MetaException, TException {
> line too long (94 > 90)
Ack


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@1034
PS3, Line 1034:       boolean deleteData)
> line too long (106 > 90)
Ack


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@1049
PS3, Line 1049:       String partName, boolean deleteData, EnvironmentContext envContext)
> line too long (91 > 90)
Ack


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@1086
PS3, Line 1086: String d
> nit: 4 spaces indention
Ack


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@1087
PS3, Line 1087: throws TException 
> nit: 8 spaces indention and move the comma above
Ack


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@1102
PS3, Line 1102:       String destinationTableName)
> line too long (104 > 90)
Ack


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@1457
PS3, Line 1457:   }
> line too long (100 > 90)
Ack


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@1458
PS3, Line 1458: 
> line too long (108 > 90)
Ack


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@2846
PS3, Line 2846:    * if hms ddl apis are accessed from catalogd's metastore server.
> line too long (115 > 90)
Ack


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@2849
PS3, Line 2849:  o
> nit: In our code style, we need one space after "if". Many changes in this 
Thanks for pointing it out. 
Will fix it.


http://gerrit.cloudera.org:8080/#/c/17298/3/tests/custom_cluster/test_metastore_service.py
File tests/custom_cluster/test_metastore_service.py:

http://gerrit.cloudera.org:8080/#/c/17298/3/tests/custom_cluster/test_metastore_service.py@424
PS3, Line 424:     )
> nit: remove this blank line
Ack



-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 4
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Comment-Date: Wed, 14 Apr 2021 23:16:27 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................


Patch Set 16:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/8825/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 16
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Tue, 01 Jun 2021 19:17:32 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................


Patch Set 10:

Build Failed 

https://jenkins.impala.io/job/gerrit-code-review-checks/8780/ : Initial code review checks failed. See linked job for details on the failure.


-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 10
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Tue, 25 May 2021 00:22:21 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table cache for hms ddl apis which modify tables and partitions.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table cache for hms ddl apis which modify tables and partitions.
......................................................................


Patch Set 5:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/17298/5/tests/custom_cluster/test_metastore_service.py
File tests/custom_cluster/test_metastore_service.py:

http://gerrit.cloudera.org:8080/#/c/17298/5/tests/custom_cluster/test_metastore_service.py@565
PS5, Line 565: q
flake8: E501 line too long (96 > 90 characters)



-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 5
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Thu, 15 Apr 2021 00:21:48 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................


Patch Set 21:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7192/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 21
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Thu, 03 Jun 2021 06:36:50 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table cache for hms ddl apis which modify tables and partitions.

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.

Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table cache for hms ddl apis which modify tables and partitions.
......................................................................


Patch Set 3:

(13 comments)

The solution looks good to me. Please adjust the code style to match existing codes. Thanks!

http://gerrit.cloudera.org:8080/#/c/17298/3//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/17298/3//COMMIT_MSG@9
PS3, Line 9: For non transactional tables
Could you explain why we don't do this for transactional tables in the commit message?


http://gerrit.cloudera.org:8080/#/c/17298/3//COMMIT_MSG@13
PS3, Line 13: (since table loading in cache takes time) but ensures consistency. This change is behind catalogd
nit: please adjust the commit message body to fit into at-most 72 chars per line.


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
File fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java:

http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@141
PS3, Line 141: import org.apache.hadoop.hive.metastore.api.InvalidPartitionException;
Is this used somewhere?


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@222
PS3, Line 222: UnknownPartitionException
Is this used somewhere?


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@347
PS3, Line 347:         
nit: We use 4 spaces indention for multi-line statement. However, many changes in this patch use 8 spaces. Could you configure your IDE to adjust this?


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@349
PS3, Line 349:         
nit: 4 spaces indention


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@915
PS3, Line 915: partitionList
Is it possible that these partitions belong to different tables?
* If no, let's call invalidateNonTransactionalTableIfExists() once
* If yes, let's de-duplicate the table names first and then call invalidateNonTransactionalTableIfExists() for each table.


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@929
PS3, Line 929: for (PartitionSpec partitionSpec : list) {
Same question as above


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@1086
PS3, Line 1086:         
nit: 4 spaces indention


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@1087
PS3, Line 1087:                 , 
nit: 8 spaces indention and move the comma above


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@2849
PS3, Line 2849: if
nit: In our code style, we need one space after "if". Many changes in this patch don't have it. Please adjust them. Thanks!


http://gerrit.cloudera.org:8080/#/c/17298/3/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@2903
PS3, Line 2903:     // return immediately if flag invalidateCacheOnDDLs_ is false
              :     if(!invalidateCacheOnDDLs_) {
              :       LOG.debug("Not removing table {}.{} from catalogd cache because " +
              :                       "invalidateCacheOnDDLs_ flag is set to {} ", dbNameWithCatalog,
              :               tableName, invalidateCacheOnDDLs_);
              :       return;
              :     }
              :     // Parse db name. Throw error if parsing fails.
              :     String dbName = dbNameWithCatalog;
              :     try {
              :       dbName = MetaStoreUtils.parseDbName(dbNameWithCatalog, serverConf_)[1];
              :     } catch (MetaException ex) {
              :       LOG.error("Successfully executed HMS api: {} but encountered error " +
              :                       "when parsing dbName {} to invalidate/remove table from cache with error message: {}",
              :               apiName, dbNameWithCatalog, ex.getMessage());
              :       throw ex;
              :     }
              :     Db db = catalog_.getDb(dbName);
              :     if(db == null) {
              :       LOG.debug("Not removing table {}.{} because db {} does not exist in catalogd cache",
              :               dbName, tableName, dbName);
              :       return;
              :     }
              :     if(!db.containsTable(tableName)) {
              :       LOG.debug("Not removing table {}.{} because it does not exist in catalogd's db cache",
              :               dbName, tableName);
              :       return;
              :     }
              :     org.apache.impala.catalog.Table catalogTbl = db.getTable(tableName);
              :     Map<String, String> tblProperties = catalogTbl.getMetaStoreTable().getParameters();
              :     if(tblProperties == null || MetaStoreUtils.isTransactionalTable(tblProperties)) {
              :       LOG.debug("Table {} is transactional. Not removing it from catalogd cache", catalogTbl.getFullName());
              :       return;
              :     }
These codes are similar to those in the above method. Can we extract them into a method?


http://gerrit.cloudera.org:8080/#/c/17298/3/tests/custom_cluster/test_metastore_service.py
File tests/custom_cluster/test_metastore_service.py:

http://gerrit.cloudera.org:8080/#/c/17298/3/tests/custom_cluster/test_metastore_service.py@424
PS3, Line 424: 
nit: remove this blank line



-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 3
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 14 Apr 2021 06:36:36 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................


Patch Set 14:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/17298/14/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
File fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java:

http://gerrit.cloudera.org:8080/#/c/17298/14/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@2978
PS14, Line 2978:   private void renameNonTransactionalTableIfExists(String oldDbNameWithCatalog, String oldTableName,
line too long (100 > 90)


http://gerrit.cloudera.org:8080/#/c/17298/14/fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java@2979
PS14, Line 2979:                                                    String newDbNameWithCatalog, String newTableName,
line too long (100 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 14
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Wed, 26 May 2021 22:25:08 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.

Sourabh Goyal has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................


Patch Set 23:

@Quanlong @Vihang: The latest test run is successful. Please take a look again and share your feedback.


-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 23
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Wed, 09 Jun 2021 01:11:00 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................


Patch Set 12:

Build Failed 

https://jenkins.impala.io/job/gerrit-code-review-checks/8782/ : Initial code review checks failed. See linked job for details on the failure.


-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 12
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Tue, 25 May 2021 00:27:03 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.

Hello Quanlong Huang, Vihang Karajgaonkar, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/17298

to look at the new patch set (#8).

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................

IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

For transactional tables, catalogd already guarantees consitent table
metadata reads based on the writeIdList passed in the request. For
non transactional tables, the reads are eventually consistent as in
event processor thread in the background, processes HMS events for the
table and updates its metadata.
In this patch, to ensure strong consistency guarantees for external
tables,we invalidate the table metadata from cache if HMS DDL apis
like alter/drop table/partition are accessed from catalogd's metastore
server. As a result of which, any subsequent get table request fetches
the table from HMS and loads it in cache. This ensures that any
get_table/get_partition requests after DDL operations on same table
return updated table metadata. This behavior has a performance penalty
since metadata loading in cache takes time specially for large tables.
The change is behind catalogd server's flag:
invalidate_hms_cache_on_ddls which is enabled by default. The flag
needs to be turned off in case of a performance bottleneck.

Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
---
M be/src/catalog/catalog-server.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M tests/custom_cluster/test_metastore_service.py
6 files changed, 517 insertions(+), 46 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/98/17298/8
-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 8
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................


Patch Set 19:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7190/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 19
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Wed, 02 Jun 2021 19:44:07 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table cache for hms ddl apis which modify tables and partitions.

Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.

Hello Quanlong Huang, Vihang Karajgaonkar, Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/17298

to look at the new patch set (#5).

Change subject: IMPALA-10648: Invalidate catalogd table cache for hms ddl apis which modify tables and partitions.
......................................................................

IMPALA-10648: Invalidate catalogd table cache for hms ddl apis which modify tables and partitions.

Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
---
M be/src/catalog/catalog-server.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/metastore/MetastoreServiceHandler.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M tests/custom_cluster/test_metastore_service.py
6 files changed, 508 insertions(+), 44 deletions(-)

git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/98/17298/5
--
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 5
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table cache for HMS DDL apis which modify tables and partitions.

Posted by "Sourabh Goyal (Code Review)" <ge...@cloudera.org>.

Sourabh Goyal has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table cache for HMS DDL apis which modify tables and partitions.
......................................................................


Patch Set 6:

> Patch Set 5:
> 
> (2 comments)

Thanks for the reminder. I have addressed these in the patch set 6


-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 6
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Wed, 21 Apr 2021 00:44:30 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table cache for hms ddl apis which modify tables and partitions.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table cache for hms ddl apis which modify tables and partitions.
......................................................................


Patch Set 2:

Build Failed 

https://jenkins.impala.io/job/gerrit-code-review-checks/8535/ : Initial code review checks failed. See linked job for details on the failure.


-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 2
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 09 Apr 2021 21:55:02 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................


Patch Set 19:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/8836/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 19
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Wed, 02 Jun 2021 19:42:33 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................


Patch Set 17: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7187/


-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 17
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Wed, 02 Jun 2021 07:13:02 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17298 )

Change subject: IMPALA-10648: Invalidate catalogd table metadata cache for HMS DDL apis
......................................................................


Patch Set 18:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7189/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/17298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idb9cc22ebfb51948433e4d57f4705ce201acaf98
Gerrit-Change-Number: 17298
Gerrit-PatchSet: 18
Gerrit-Owner: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sourabh Goyal <so...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Wed, 02 Jun 2021 17:30:56 +0000
Gerrit-HasComments: No