You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Quanlong Huang (Code Review)" <ge...@cloudera.org> on 2020/09/10 03:43:52 UTC

[Impala-ASF-CR] [WIP] IMPALA-9936: Only send invalidations in DDL responses to LocalCatalog coordinators

Quanlong Huang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16435


Change subject: [WIP] IMPALA-9936: Only send invalidations in DDL responses to LocalCatalog coordinators
......................................................................

[WIP] IMPALA-9936: Only send invalidations in DDL responses to LocalCatalog coordinators

Catalogd RPC response contains the updated catalog objects in a full
form. However, LocalCatalog coordinators just need the object names for
invalidations. It's a waste of space to send the full catalog objects to
LocalCatalog coordinators. On the other hand, there is a risk of OOM due
to hitting the Java array limit when serializing a table that has a huge
metadata footprint.

This patch refactor the catalogd RPC responses to only send back
invalidations in need. To distinguish between legacy and LocalCatalog
coordinators, a new field, want_minimal_response, is introduced in
TCatalogServiceRequestHeader which is the header for most of the
Catalogd RPC requests (e.g. TDdlExecRequest, TUpdateCatalogRequest and
TResetMetadataRequest). LocalCatalog coordinators will set this field to
true. When adding updated catalog objects to the response, catalogd will
add invalidations which only contain the object names (e.g. db name,
table name). Note that function objects are small so are ignored in this
optimization.

Tests:
 - Run test_ddl.py in both legacy catalog mode and local catalog mode.
 - TODO: figure out how to add test coverage for this

Change-Id: Id45827295ddee3eb6e98a11c55f582b2aebe5f38
---
M be/src/exec/catalog-op-executor.cc
M be/src/service/client-request-state.cc
M common/thrift/CatalogService.thrift
M fe/src/main/java/org/apache/impala/catalog/CatalogObject.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/Db.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/test/java/org/apache/impala/catalog/CatalogTest.java
10 files changed, 206 insertions(+), 138 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/35/16435/1
-- 
To view, visit http://gerrit.cloudera.org:8080/16435
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Id45827295ddee3eb6e98a11c55f582b2aebe5f38
Gerrit-Change-Number: 16435
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>

[Impala-ASF-CR] IMPALA-9936: Only send invalidations in DDL responses to LocalCatalog coordinators

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/16435 )

Change subject: IMPALA-9936: Only send invalidations in DDL responses to LocalCatalog coordinators
......................................................................


Patch Set 5: Code-Review+2

(1 comment)

Thank Tim's quick review! Carry on Tim's +2.

http://gerrit.cloudera.org:8080/#/c/16435/4/be/src/exec/catalog-op-executor.cc
File be/src/exec/catalog-op-executor.cc:

http://gerrit.cloudera.org:8080/#/c/16435/4/be/src/exec/catalog-op-executor.cc@72
PS4, Line 72: VerifyMinimalResponse
> nit: VerifyMinimalResponse
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/16435
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id45827295ddee3eb6e98a11c55f582b2aebe5f38
Gerrit-Change-Number: 16435
Gerrit-PatchSet: 5
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Sat, 05 Dec 2020 03:17:19 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-9936: Only send invalidations in DDL responses to LocalCatalog coordinators

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16435 )

Change subject: IMPALA-9936: Only send invalidations in DDL responses to LocalCatalog coordinators
......................................................................


Patch Set 4:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6728/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/16435
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id45827295ddee3eb6e98a11c55f582b2aebe5f38
Gerrit-Change-Number: 16435
Gerrit-PatchSet: 4
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 04 Dec 2020 05:41:17 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9936: Only send invalidations in DDL responses to LocalCatalog coordinators

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16435 )

Change subject: IMPALA-9936: Only send invalidations in DDL responses to LocalCatalog coordinators
......................................................................


Patch Set 5: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/16435
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id45827295ddee3eb6e98a11c55f582b2aebe5f38
Gerrit-Change-Number: 16435
Gerrit-PatchSet: 5
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Sat, 05 Dec 2020 08:52:39 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9936: Only send invalidations in DDL responses to LocalCatalog coordinators

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16435 )

Change subject: IMPALA-9936: Only send invalidations in DDL responses to LocalCatalog coordinators
......................................................................


Patch Set 3: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6726/


-- 
To view, visit http://gerrit.cloudera.org:8080/16435
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id45827295ddee3eb6e98a11c55f582b2aebe5f38
Gerrit-Change-Number: 16435
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 03 Dec 2020 14:20:37 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9936: Only send invalidations in DDL responses to LocalCatalog coordinators

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/16435

to look at the new patch set (#3).

Change subject: IMPALA-9936: Only send invalidations in DDL responses to LocalCatalog coordinators
......................................................................

IMPALA-9936: Only send invalidations in DDL responses to LocalCatalog coordinators

Catalogd RPC response contains the updated catalog objects in a full
form. For instance, a RPC for adding a new partition to an HdfsTable
will return the whole HdfsTable object(metadata) containing all the
partitions. This is required by legacy coordinators where the whole
HdfsTable object is used to replace the stale object(metadata snapshot).
However, LocalCatalog coordinators just need the object names for
invalidations. It's a waste of space to send the full catalog objects to
LocalCatalog coordinators. On the other hand, there is a risk of OOM due
to hitting the Java array limit when serializing a table that has a huge
metadata footprint.

This patch refactors the catalogd RPC responses to only send back
invalidations in need. To distinguish between legacy and LocalCatalog
coordinators, a new field, want_minimal_response, is introduced in
TCatalogServiceRequestHeader which is the header for most of the
Catalogd RPC requests (e.g. TDdlExecRequest, TUpdateCatalogRequest and
TResetMetadataRequest). LocalCatalog coordinators will set this field to
true. When adding updated catalog objects to the response, catalogd will
add invalidations which only contain the object names (e.g. db name,
table name). Note that function objects are small so are ignored in this
optimization.

Tests:
 - Add DCHECKs in catalog-op-executor.cc to verify the catalog objects
   recieved by LocalCatalog coordinators are in minimal mode.
 - Run test_ddl.py in both legacy catalog mode and local catalog mode.

Change-Id: Id45827295ddee3eb6e98a11c55f582b2aebe5f38
---
M be/src/exec/catalog-op-executor.cc
M be/src/service/client-request-state.cc
M common/thrift/CatalogService.thrift
M fe/src/main/java/org/apache/impala/catalog/CatalogObject.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/Db.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/test/java/org/apache/impala/catalog/CatalogTest.java
10 files changed, 237 insertions(+), 139 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/35/16435/3
-- 
To view, visit http://gerrit.cloudera.org:8080/16435
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id45827295ddee3eb6e98a11c55f582b2aebe5f38
Gerrit-Change-Number: 16435
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] IMPALA-9936: Only send invalidations in DDL responses to LocalCatalog coordinators

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Hello Vihang Karajgaonkar, Tim Armstrong, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/16435

to look at the new patch set (#5).

Change subject: IMPALA-9936: Only send invalidations in DDL responses to LocalCatalog coordinators
......................................................................

IMPALA-9936: Only send invalidations in DDL responses to LocalCatalog coordinators

Catalogd RPC response contains the updated catalog objects in a full
form. For instance, a RPC for adding a new partition to an HdfsTable
will return the whole HdfsTable object(metadata) containing all the
partitions. This is required by legacy coordinators where the whole
HdfsTable object is used to replace the stale object(metadata snapshot).
However, LocalCatalog coordinators just need the object names for
invalidations. It's a waste of space to send the full catalog objects to
LocalCatalog coordinators. On the other hand, there is a risk of OOM due
to hitting the Java array limit when serializing a table that has a huge
metadata footprint.

This patch refactors the catalogd RPC responses to only send back
invalidations in need. To distinguish between legacy and LocalCatalog
coordinators, a new field, want_minimal_response, is introduced in
TCatalogServiceRequestHeader which is the header for most of the
Catalogd RPC requests (e.g. TDdlExecRequest, TUpdateCatalogRequest and
TResetMetadataRequest). LocalCatalog coordinators will set this field to
true. When adding updated catalog objects to the response, catalogd will
add invalidations which only contain the object names (e.g. db name,
table name). Note that function objects are small so are ignored in this
optimization.

Tests:
 - Add DCHECKs in catalog-op-executor.cc to verify the catalog objects
   recieved by LocalCatalog coordinators are in minimal mode.
 - Run test_ddl.py in both legacy catalog mode and local catalog mode.

Change-Id: Id45827295ddee3eb6e98a11c55f582b2aebe5f38
---
M be/src/exec/catalog-op-executor.cc
M be/src/service/client-request-state.cc
M common/thrift/CatalogService.thrift
M fe/src/main/java/org/apache/impala/catalog/CatalogObject.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/Db.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/test/java/org/apache/impala/catalog/CatalogTest.java
10 files changed, 239 insertions(+), 139 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/35/16435/5
-- 
To view, visit http://gerrit.cloudera.org:8080/16435
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id45827295ddee3eb6e98a11c55f582b2aebe5f38
Gerrit-Change-Number: 16435
Gerrit-PatchSet: 5
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>

[Impala-ASF-CR] IMPALA-9936: Only send invalidations in DDL responses to LocalCatalog coordinators

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16435 )

Change subject: IMPALA-9936: Only send invalidations in DDL responses to LocalCatalog coordinators
......................................................................


Patch Set 2:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/16435/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:

http://gerrit.cloudera.org:8080/#/c/16435/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3413
PS2, Line 3413:       response.result.addToRemoved_catalog_objects(result.first.toMinimalTCatalogObject());
line too long (91 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/16435
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id45827295ddee3eb6e98a11c55f582b2aebe5f38
Gerrit-Change-Number: 16435
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 03 Dec 2020 08:27:20 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-9936: Only send invalidations in DDL responses to LocalCatalog coordinators

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/16435

to look at the new patch set (#2).

Change subject: IMPALA-9936: Only send invalidations in DDL responses to LocalCatalog coordinators
......................................................................

IMPALA-9936: Only send invalidations in DDL responses to LocalCatalog coordinators

Catalogd RPC response contains the updated catalog objects in a full
form. For instance, a RPC for adding a new partition to an HdfsTable
will return the whole HdfsTable object(metadata) containing all the
partitions. This is required by legacy coordinators where the whole
HdfsTable object is used to replace the stale object(metadata snapshot).
However, LocalCatalog coordinators just need the object names for
invalidations. It's a waste of space to send the full catalog objects to
LocalCatalog coordinators. On the other hand, there is a risk of OOM due
to hitting the Java array limit when serializing a table that has a huge
metadata footprint.

This patch refactors the catalogd RPC responses to only send back
invalidations in need. To distinguish between legacy and LocalCatalog
coordinators, a new field, want_minimal_response, is introduced in
TCatalogServiceRequestHeader which is the header for most of the
Catalogd RPC requests (e.g. TDdlExecRequest, TUpdateCatalogRequest and
TResetMetadataRequest). LocalCatalog coordinators will set this field to
true. When adding updated catalog objects to the response, catalogd will
add invalidations which only contain the object names (e.g. db name,
table name). Note that function objects are small so are ignored in this
optimization.

Tests:
 - Add DCHECKs in catalog-op-executor.cc to verify the catalog objects
   recieved by LocalCatalog coordinators are in minimal mode.
 - Run test_ddl.py in both legacy catalog mode and local catalog mode.

Change-Id: Id45827295ddee3eb6e98a11c55f582b2aebe5f38
---
M be/src/exec/catalog-op-executor.cc
M be/src/service/client-request-state.cc
M common/thrift/CatalogService.thrift
M fe/src/main/java/org/apache/impala/catalog/CatalogObject.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/Db.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/test/java/org/apache/impala/catalog/CatalogTest.java
10 files changed, 236 insertions(+), 139 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/35/16435/2
-- 
To view, visit http://gerrit.cloudera.org:8080/16435
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id45827295ddee3eb6e98a11c55f582b2aebe5f38
Gerrit-Change-Number: 16435
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] IMPALA-9936: Only send invalidations in DDL responses to LocalCatalog coordinators

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16435 )

Change subject: IMPALA-9936: Only send invalidations in DDL responses to LocalCatalog coordinators
......................................................................


Patch Set 5:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/7782/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/16435
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id45827295ddee3eb6e98a11c55f582b2aebe5f38
Gerrit-Change-Number: 16435
Gerrit-PatchSet: 5
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Sat, 05 Dec 2020 03:27:40 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9936: Only send invalidations in DDL responses to LocalCatalog coordinators

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16435 )

Change subject: IMPALA-9936: Only send invalidations in DDL responses to LocalCatalog coordinators
......................................................................

IMPALA-9936: Only send invalidations in DDL responses to LocalCatalog coordinators

Catalogd RPC response contains the updated catalog objects in a full
form. For instance, a RPC for adding a new partition to an HdfsTable
will return the whole HdfsTable object(metadata) containing all the
partitions. This is required by legacy coordinators where the whole
HdfsTable object is used to replace the stale object(metadata snapshot).
However, LocalCatalog coordinators just need the object names for
invalidations. It's a waste of space to send the full catalog objects to
LocalCatalog coordinators. On the other hand, there is a risk of OOM due
to hitting the Java array limit when serializing a table that has a huge
metadata footprint.

This patch refactors the catalogd RPC responses to only send back
invalidations in need. To distinguish between legacy and LocalCatalog
coordinators, a new field, want_minimal_response, is introduced in
TCatalogServiceRequestHeader which is the header for most of the
Catalogd RPC requests (e.g. TDdlExecRequest, TUpdateCatalogRequest and
TResetMetadataRequest). LocalCatalog coordinators will set this field to
true. When adding updated catalog objects to the response, catalogd will
add invalidations which only contain the object names (e.g. db name,
table name). Note that function objects are small so are ignored in this
optimization.

Tests:
 - Add DCHECKs in catalog-op-executor.cc to verify the catalog objects
   recieved by LocalCatalog coordinators are in minimal mode.
 - Run test_ddl.py in both legacy catalog mode and local catalog mode.

Change-Id: Id45827295ddee3eb6e98a11c55f582b2aebe5f38
Reviewed-on: http://gerrit.cloudera.org:8080/16435
Reviewed-by: Quanlong Huang <hu...@gmail.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/exec/catalog-op-executor.cc
M be/src/service/client-request-state.cc
M common/thrift/CatalogService.thrift
M fe/src/main/java/org/apache/impala/catalog/CatalogObject.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/Db.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/test/java/org/apache/impala/catalog/CatalogTest.java
10 files changed, 239 insertions(+), 139 deletions(-)

Approvals:
  Quanlong Huang: Looks good to me, approved
  Impala Public Jenkins: Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/16435
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Id45827295ddee3eb6e98a11c55f582b2aebe5f38
Gerrit-Change-Number: 16435
Gerrit-PatchSet: 6
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>

[Impala-ASF-CR] [WIP] IMPALA-9936: Only send invalidations in DDL responses to LocalCatalog coordinators

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16435 )

Change subject: [WIP] IMPALA-9936: Only send invalidations in DDL responses to LocalCatalog coordinators
......................................................................


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/16435/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:

http://gerrit.cloudera.org:8080/#/c/16435/1/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3221
PS1, Line 3221:       response.result.addToRemoved_catalog_objects(result.first.toMinimalTCatalogObject());
line too long (91 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/16435
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id45827295ddee3eb6e98a11c55f582b2aebe5f38
Gerrit-Change-Number: 16435
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 10 Sep 2020 03:44:39 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-9936: Only send invalidations in DDL responses to LocalCatalog coordinators

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/16435 )

Change subject: IMPALA-9936: Only send invalidations in DDL responses to LocalCatalog coordinators
......................................................................


Patch Set 3:

> Patch Set 3: Verified-1
> 
> Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6726/

The DCHECK added in PS3 capture that LOAD DATA DDL still receive full table object from catalogd. Fixed it in PS4.


-- 
To view, visit http://gerrit.cloudera.org:8080/16435
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id45827295ddee3eb6e98a11c55f582b2aebe5f38
Gerrit-Change-Number: 16435
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 04 Dec 2020 01:37:34 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9936: Only send invalidations in DDL responses to LocalCatalog coordinators

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16435 )

Change subject: IMPALA-9936: Only send invalidations in DDL responses to LocalCatalog coordinators
......................................................................


Patch Set 4:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/7771/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/16435
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id45827295ddee3eb6e98a11c55f582b2aebe5f38
Gerrit-Change-Number: 16435
Gerrit-PatchSet: 4
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 04 Dec 2020 01:59:16 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9936: Only send invalidations in DDL responses to LocalCatalog coordinators

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16435 )

Change subject: IMPALA-9936: Only send invalidations in DDL responses to LocalCatalog coordinators
......................................................................


Patch Set 3:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6726/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/16435
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id45827295ddee3eb6e98a11c55f582b2aebe5f38
Gerrit-Change-Number: 16435
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 03 Dec 2020 08:44:33 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9936: Only send invalidations in DDL responses to LocalCatalog coordinators

Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16435 )

Change subject: IMPALA-9936: Only send invalidations in DDL responses to LocalCatalog coordinators
......................................................................


Patch Set 4: Code-Review+2

(1 comment)

This makes sense to me.

http://gerrit.cloudera.org:8080/#/c/16435/4/be/src/exec/catalog-op-executor.cc
File be/src/exec/catalog-op-executor.cc:

http://gerrit.cloudera.org:8080/#/c/16435/4/be/src/exec/catalog-op-executor.cc@72
PS4, Line 72: verifyMinimalResponse
nit: VerifyMinimalResponse



-- 
To view, visit http://gerrit.cloudera.org:8080/16435
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id45827295ddee3eb6e98a11c55f582b2aebe5f38
Gerrit-Change-Number: 16435
Gerrit-PatchSet: 4
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Sat, 05 Dec 2020 01:48:21 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] [WIP] IMPALA-9936: Only send invalidations in DDL responses to LocalCatalog coordinators

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16435 )

Change subject: [WIP] IMPALA-9936: Only send invalidations in DDL responses to LocalCatalog coordinators
......................................................................


Patch Set 1:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/7148/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/16435
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id45827295ddee3eb6e98a11c55f582b2aebe5f38
Gerrit-Change-Number: 16435
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 10 Sep 2020 04:06:19 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9936: Only send invalidations in DDL responses to LocalCatalog coordinators

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/16435

to look at the new patch set (#4).

Change subject: IMPALA-9936: Only send invalidations in DDL responses to LocalCatalog coordinators
......................................................................

IMPALA-9936: Only send invalidations in DDL responses to LocalCatalog coordinators

Catalogd RPC response contains the updated catalog objects in a full
form. For instance, a RPC for adding a new partition to an HdfsTable
will return the whole HdfsTable object(metadata) containing all the
partitions. This is required by legacy coordinators where the whole
HdfsTable object is used to replace the stale object(metadata snapshot).
However, LocalCatalog coordinators just need the object names for
invalidations. It's a waste of space to send the full catalog objects to
LocalCatalog coordinators. On the other hand, there is a risk of OOM due
to hitting the Java array limit when serializing a table that has a huge
metadata footprint.

This patch refactors the catalogd RPC responses to only send back
invalidations in need. To distinguish between legacy and LocalCatalog
coordinators, a new field, want_minimal_response, is introduced in
TCatalogServiceRequestHeader which is the header for most of the
Catalogd RPC requests (e.g. TDdlExecRequest, TUpdateCatalogRequest and
TResetMetadataRequest). LocalCatalog coordinators will set this field to
true. When adding updated catalog objects to the response, catalogd will
add invalidations which only contain the object names (e.g. db name,
table name). Note that function objects are small so are ignored in this
optimization.

Tests:
 - Add DCHECKs in catalog-op-executor.cc to verify the catalog objects
   recieved by LocalCatalog coordinators are in minimal mode.
 - Run test_ddl.py in both legacy catalog mode and local catalog mode.

Change-Id: Id45827295ddee3eb6e98a11c55f582b2aebe5f38
---
M be/src/exec/catalog-op-executor.cc
M be/src/service/client-request-state.cc
M common/thrift/CatalogService.thrift
M fe/src/main/java/org/apache/impala/catalog/CatalogObject.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/Db.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/test/java/org/apache/impala/catalog/CatalogTest.java
10 files changed, 239 insertions(+), 139 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/35/16435/4
-- 
To view, visit http://gerrit.cloudera.org:8080/16435
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id45827295ddee3eb6e98a11c55f582b2aebe5f38
Gerrit-Change-Number: 16435
Gerrit-PatchSet: 4
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] IMPALA-9936: Only send invalidations in DDL responses to LocalCatalog coordinators

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16435 )

Change subject: IMPALA-9936: Only send invalidations in DDL responses to LocalCatalog coordinators
......................................................................


Patch Set 2:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/7766/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/16435
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id45827295ddee3eb6e98a11c55f582b2aebe5f38
Gerrit-Change-Number: 16435
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 03 Dec 2020 08:48:46 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9936: Only send invalidations in DDL responses to LocalCatalog coordinators

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16435 )

Change subject: IMPALA-9936: Only send invalidations in DDL responses to LocalCatalog coordinators
......................................................................


Patch Set 4: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/16435
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id45827295ddee3eb6e98a11c55f582b2aebe5f38
Gerrit-Change-Number: 16435
Gerrit-PatchSet: 4
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 04 Dec 2020 11:12:49 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9936: Only send invalidations in DDL responses to LocalCatalog coordinators

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16435 )

Change subject: IMPALA-9936: Only send invalidations in DDL responses to LocalCatalog coordinators
......................................................................


Patch Set 5:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6730/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/16435
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id45827295ddee3eb6e98a11c55f582b2aebe5f38
Gerrit-Change-Number: 16435
Gerrit-PatchSet: 5
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Comment-Date: Sat, 05 Dec 2020 03:18:09 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-9936: Only send invalidations in DDL responses to LocalCatalog coordinators

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16435 )

Change subject: IMPALA-9936: Only send invalidations in DDL responses to LocalCatalog coordinators
......................................................................


Patch Set 3:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/7767/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/16435
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id45827295ddee3eb6e98a11c55f582b2aebe5f38
Gerrit-Change-Number: 16435
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 03 Dec 2020 09:05:56 +0000
Gerrit-HasComments: No