You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "liuyao (Code Review)" <ge...@cloudera.org> on 2021/07/03 11:05:22 UTC

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync 

liuyao has uploaded this change for review. ( http://gerrit.cloudera.org:8080/17645


Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................

IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

The change of catalogServiceId did not trigger a full topic update
request.

During the execution of DDL or DML, restarting Catalogd will cause
catalogServiceId to change. The DDL will execute successfully, and
then directly update the local cache (not through Statestored), modify
the local catalog with the new catalogServiceId, trigger an exception,
print the exception information, but did not submit full topic update
request. The version of Catalogd is lower than Impalad, so the metadata
information synchronized from Catalogd will be lost.

Under normal circumstances, if the catalogServiceId changes, Impalad
will submit full topic update request, Impalad will receive
TUpdateCatalogCacheRequest, and is_delta is false, that is full update.
Impalad will reset the local catalog cache, and then update, and the
version of DatalogD is greater than Impalad.

Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
---
M be/src/service/impala-server.cc
1 file changed, 1 insertion(+), 4 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/17645/1
-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 1
Gerrit-Owner: liuyao <li...@sensorsdata.cn>

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................


Patch Set 7:

> Patch Set 7:
> 
> 16:50:03 ERROR: cannot verify jenkins.impala.io's certificate, issued by ‘CN=Go Daddy Secure Certificate Authority - G2,OU=http://certs.godaddy.com/repository/,O=GoDaddy.com\\, Inc.,L=Scottsdale,ST=Arizona,C=US’:
> 16:50:03   Issued certificate has expired.
> 16:50:03 To connect to jenkins.impala.io insecurely, use `--no-check-certificate'.
> 16:50:03 + ssh -p 29418 impala-public-jenkins@gerrit.cloudera.org gerrit review --verified -1 --message '"Build' failed: 'https://jenkins.impala.io/job/gerrit-verify-dryrun/7319/"' --project Impala-ASF 17645,7
> 16:50:05 + exit 1
> 16:50:05 Build step 'Execute shell' marked build as failure

These can be ignored. The actual failure is
02:28:16 parallel-all-tests #9056 started.
08:50:03 parallel-all-tests #9056 completed. Result was FAILURE

Then you can trace to the actual failed job:
https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/4470/

They are Iceberg related failures. Seems unrelated:

 metadata.test_show_create_table.TestShowCreateTable.test_show_create_table[protocol: beeswax | table_format: text/none]
query_test.test_iceberg.TestIcebergTable.test_catalogs[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: parquet/none]


-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 7
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Wed, 21 Jul 2021 23:56:46 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................


Patch Set 7: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7310/


-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 7
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Sat, 17 Jul 2021 08:57:11 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................


Patch Set 1: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 1
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sat, 03 Jul 2021 17:08:42 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................


Patch Set 13: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 13
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Mon, 26 Jul 2021 16:25:16 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "liuyao (Code Review)" <ge...@cloudera.org>.
liuyao has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................


Patch Set 20:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/17645/16/tests/custom_cluster/test_restart_services.py
File tests/custom_cluster/test_restart_services.py:

http://gerrit.cloudera.org:8080/#/c/17645/16/tests/custom_cluster/test_restart_services.py@239
PS16, Line 239: sleep(self.UPDATE_FREQUENCY_S * 2)
> While thinking about comments for this sleep, I realize this actually revea
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 20
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Thu, 19 Aug 2021 09:23:08 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "liuyao (Code Review)" <ge...@cloudera.org>.
liuyao has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................


Patch Set 4:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/17645/4/tests/custom_cluster/test_restart_services.py
File tests/custom_cluster/test_restart_services.py:

http://gerrit.cloudera.org:8080/#/c/17645/4/tests/custom_cluster/test_restart_services.py@189
PS4, Line 189:     # Wait enough time to synchronize the metadata
             :     sleep(20)
> We should not depend on sleep. This is what I mean in the previous
 > comment: https://gerrit.cloudera.org/c/17645/1//COMMIT_MSG#17
 > 
 > When AlterTable finishes, the metadata of this table should be
 > up-to-date. The current patch can't satify this.

I think of three solutions:
Solution 1. Directly trigger the full topic update, but you have to wait until the full update is completed before you can update the local cache, because the local cache has a higher version. But I did not find the api to trigger the full update, ImpalaServer::CatalogUpdateCallback is not applicable
Solution 2. When restart catalogs, delay DDL execution time. After the ddl is executed, determine whether the catalog service id of the returned result is the same as the local one. If it is different, wait for the local catalog service id to change, and then return success
Solution 3. When call exec_env_->frontend()->UpdateCatalogCache, if the catalog server id is different, the catalog object will be updated directly without comparing the versions



-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 4
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Fri, 16 Jul 2021 03:20:31 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................


Patch Set 5:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/9100/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 5
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Fri, 16 Jul 2021 05:13:07 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "liuyao (Code Review)" <ge...@cloudera.org>.
liuyao has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................


Patch Set 10:

(4 comments)

http://gerrit.cloudera.org:8080/#/c/17645/7/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:

http://gerrit.cloudera.org:8080/#/c/17645/7/be/src/service/impala-server.cc@a2142
PS7, Line 2142: 
> Can we keep this? If we wait for the next statestore update, this won't cau
Done


http://gerrit.cloudera.org:8080/#/c/17645/7/be/src/service/impala-server.cc@2144
PS7, Line 2144:     TUniqueId cur_service_id;
> Could you add a log before the while-loop? E.g.
Done


http://gerrit.cloudera.org:8080/#/c/17645/7/be/src/service/impala-server.cc@2150
PS7, Line 2150:             << ".ID in response:" << catalog_service_id << ". Catalogd may be restarted."
> I think this captures the case that catalogd restart again and we get anoth
The previous catalog service ID was not recorded, so I printed expected catalog service ID


http://gerrit.cloudera.org:8080/#/c/17645/7/tests/custom_cluster/test_restart_services.py
File tests/custom_cluster/test_restart_services.py:

http://gerrit.cloudera.org:8080/#/c/17645/7/tests/custom_cluster/test_restart_services.py@169
PS7, Line 169:   def test_restart_catalogd(self):
> This is already a good test case. Can we add another one to cover the case 
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 10
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Thu, 22 Jul 2021 10:11:24 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-5476: Fix catalogd restart brings stale metadata

Posted by "liuyao (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Aman Sinha, Thomas Tauber-Marshall, Vihang Karajgaonkar, Zoltan Borok-Nagy, Tim Armstrong, Wenzhe Zhou, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/17645

to look at the new patch set (#26).

Change subject: IMPALA-5476: Fix catalogd restart brings stale metadata
......................................................................

IMPALA-5476: Fix catalogd restart brings stale metadata

ImpaladCatalog#updateCatalog() doesn't trigger a full topic update
request when detecting catalogServiceId changes. It just updates the
local catalogServiceId and throws an exception to abort applying the
DDL/DML results. This causes a problem when catalogd is restarted and
the DDL/DML is executed on the restarted instance. In this case, only
the local catalogServiceId is updated to the latest. The local catalog
remains stale. Then when dealing with the following updates from
statestore, the catalogServiceId always matches, so updates will be
applied without exceptions. However, the catalog objects usually won't
be updated since they have higher versions (from the old catalogd
instance) than those in the update. This brings the local catalog out
of sync until the catalog version of the new catalogd grows larger
enough.

Note that in dealing with the catalog updates from statestore, if the
catalogServiceId unmatches, impalad will request a full topic update.
See more in ImpalaServer::CatalogUpdateCallback().

This patch fixes this issue by checking the catalogServiceId before
invoking UpdateCatalogCache() of FE. If catalogServiceId doesn't match
the one in the DDL/DML result, wait until it changes. The following
update from statestore will change it and unblocks the DDL/DML thread.

Testing

add several tests in
tests/custom_cluster/test_restart_services.py

Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
---
M be/src/service/impala-server.cc
M tests/custom_cluster/test_restart_services.py
2 files changed, 150 insertions(+), 13 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/17645/26
-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 26
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>

[Impala-ASF-CR] IMPALA-5476: Fix catalogd restart brings stale metadata

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix catalogd restart brings stale metadata
......................................................................


Patch Set 27: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 27
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Fri, 20 Aug 2021 10:42:33 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "liuyao (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Aman Sinha, Thomas Tauber-Marshall, Vihang Karajgaonkar, Zoltan Borok-Nagy, Tim Armstrong, Wenzhe Zhou, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/17645

to look at the new patch set (#19).

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................

IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

The change of catalogServiceId did not trigger a full topic update
request.

During the execution of DDL or DML, restarting Catalogd will cause
catalogServiceId to change. The DDL will execute successfully, and
then directly update the local cache (not through Statestored), modify
the local catalog with the new catalogServiceId, trigger an exception,
print the exception information, but did not submit full topic update
request. The version of Catalogd is lower than Impalad, so the metadata
information synchronized from Catalogd will be lost.

Under normal circumstances, if the catalogServiceId changes, Impalad
will submit full topic update request, Impalad will receive
TUpdateCatalogCacheRequest, and is_delta is false, that is full update.
Impalad will reset the local catalog cache, and then update, and the
version of CatalogD is greater than Impalad.

Testing

add TestRestart#test_restart_catalogd in
ests/custom_cluster/test_restart_services.py

Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
---
M be/src/service/impala-server.cc
M tests/custom_cluster/test_restart_services.py
2 files changed, 186 insertions(+), 13 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/17645/19
-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 19
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>

[Impala-ASF-CR] IMPALA-5476: Fix catalogd restart brings stale metadata

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix catalogd restart brings stale metadata
......................................................................


Patch Set 25:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/17645/25/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:

http://gerrit.cloudera.org:8080/#/c/17645/25/be/src/service/impala-server.cc@2146
PS25, Line 2146:     
line has trailing whitespace



-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 25
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Fri, 20 Aug 2021 09:14:04 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "liuyao (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Aman Sinha, Thomas Tauber-Marshall, Vihang Karajgaonkar, Zoltan Borok-Nagy, Tim Armstrong, Wenzhe Zhou, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/17645

to look at the new patch set (#2).

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................

IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

The change of catalogServiceId did not trigger a full topic update
request.

During the execution of DDL or DML, restarting Catalogd will cause
catalogServiceId to change. The DDL will execute successfully, and
then directly update the local cache (not through Statestored), modify
the local catalog with the new catalogServiceId, trigger an exception,
print the exception information, but did not submit full topic update
request. The version of Catalogd is lower than Impalad, so the metadata
information synchronized from Catalogd will be lost.

Under normal circumstances, if the catalogServiceId changes, Impalad
will submit full topic update request, Impalad will receive
TUpdateCatalogCacheRequest, and is_delta is false, that is full update.
Impalad will reset the local catalog cache, and then update, and the
version of CatalogD is greater than Impalad.

Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
---
M be/src/service/impala-server.cc
1 file changed, 1 insertion(+), 4 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/17645/2
-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 2
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "liuyao (Code Review)" <ge...@cloudera.org>.
liuyao has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................


Patch Set 16:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/17645/10/tests/custom_cluster/test_restart_services.py
File tests/custom_cluster/test_restart_services.py:

http://gerrit.cloudera.org:8080/#/c/17645/10/tests/custom_cluster/test_restart_services.py@208
PS10, Line 208:         LOG.info(str(e))
              :       if i == 5:
              :         self.cluster.catalogd.restart()
              : 
> statestored will send topic update request to catalogd once it's registered
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 16
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Fri, 06 Aug 2021 10:56:05 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "liuyao (Code Review)" <ge...@cloudera.org>.
liuyao has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................


Patch Set 17:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/17645/16/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:

http://gerrit.cloudera.org:8080/#/c/17645/16/be/src/service/impala-server.cc@2129
PS16, Line 2129:     TUniqueId cur_service_id;
               :     {
               :       unique_lock<mutex> ver_lock(catalog_version_lock_);
               :       cur_service_id = catalog_update_info_.catalog_service_id;
               :       if (catalog_update_info_.catalog_service_id != catalog_service_id) {
               :         LOG(INFO) << "Catalog service ID mismatch. Current ID: "
               :             << PrintId(cur_service_id) << ". ID in response: "
               :             << PrintId(catalog_service_id) << ". Catalogd may be restarted. Waitin
> nit: Let's move these into the if-clause at line 2155.
Done


http://gerrit.cloudera.org:8080/#/c/17645/16/tests/custom_cluster/test_restart_services.py
File tests/custom_cluster/test_restart_services.py:

http://gerrit.cloudera.org:8080/#/c/17645/16/tests/custom_cluster/test_restart_services.py@217
PS16, Line 217: 10
> nit: Can we lower this down to 5?
Done. down to 10


http://gerrit.cloudera.org:8080/#/c/17645/16/tests/custom_cluster/test_restart_services.py@239
PS16, Line 239: sleep(self.UPDATE_FREQUENCY_S * 2)
> While thinking about comments for this sleep, I realize this actually revea
If the service id is monotonically increasing, then this problem is better solved. But the service id is not, maybe we can treat it as a todo



-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 17
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Mon, 16 Aug 2021 03:19:11 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "liuyao (Code Review)" <ge...@cloudera.org>.
liuyao has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................


Patch Set 4:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/17645/4/tests/custom_cluster/test_restart_services.py
File tests/custom_cluster/test_restart_services.py:

http://gerrit.cloudera.org:8080/#/c/17645/4/tests/custom_cluster/test_restart_services.py@189
PS4, Line 189:     # Wait enough time to synchronize the metadata
             :     sleep(20)
> > We should not depend on sleep. This is what I mean in the previous
I prefer Solution 2, this kind of modification is smaller and safer



-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 4
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Fri, 16 Jul 2021 03:36:37 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................


Patch Set 23: Code-Review+2

(7 comments)

The patch is in good shape. Let's make the descriptions and comments clear!

http://gerrit.cloudera.org:8080/#/c/17645/23//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/17645/23//COMMIT_MSG@9
PS23, Line 9: The change of catalogServiceId did not trigger a full topic update
            : request.
            : 
            : During the execution of DDL or DML, restarting Catalogd will cause
            : catalogServiceId to change. The DDL will execute successfully, and
            : then directly update the local cache (not through Statestored), modify
            : the local catalog with the new catalogServiceId, trigger an exception,
            : print the exception information, but did not submit full topic update
            : request. The version of Catalogd is lower than Impalad, so the metadata
            : information synchronized from Catalogd will be lost.
nit: ImpaladCatalog#updateCatalog() doesn't trigger a full topic update request when detecting catalogServiceId changes. It just updates the local catalogServiceId and throws an exception to abort applying the DDL/DML results. This causes a problem when catalogd is restarted and the DDL/DML is executed on the restarted instance. In this case, only the local catalogServiceId is updated to the latest. The local catalog remains stale. Then when dealing with the following updates from statestore, the catalogServiceId always matches, so updates will be applied without exceptions. However, the catalog objects usually won't be updated since they have higher versions (from the old catalogd instance) than those in the update. This brings the local catalog out of sync until the catalog version of the new catalogd grows larger enough.


http://gerrit.cloudera.org:8080/#/c/17645/23//COMMIT_MSG@20
PS23, Line 20: Under normal circumstances, if the catalogServiceId changes, Impalad
             : will submit full topic update request, Impalad will receive
             : TUpdateCatalogCacheRequest, and is_delta is false, that is full update.
             : Impalad will reset the local catalog cache, and then update, and the
             : version of CatalogD is greater than Impalad.
nit: Note that in dealing with the catalog updates from statestore, if the catalogServiceId unmatches, impalad will request a full topic update. See more in ImpalaServer::CatalogUpdateCallback().


http://gerrit.cloudera.org:8080/#/c/17645/23//COMMIT_MSG@25
PS23, Line 25: 
Let's add some description about how we fix the bug:

This patch fixes this issue by checking the catalogServiceId before invoking UpdateCatalogCache() of FE. If catalogServiceId doesn't match the one in the DDL/DML result, wait until it changes. The following update from statestore will change it and unblocks the DDL/DML thread.


http://gerrit.cloudera.org:8080/#/c/17645/23//COMMIT_MSG@28
PS23, Line 28: TestRestart#test_restart_catalogd
nit: several tests


http://gerrit.cloudera.org:8080/#/c/17645/23//COMMIT_MSG@29
PS23, Line 29: ests
nit: tests


http://gerrit.cloudera.org:8080/#/c/17645/23/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:

http://gerrit.cloudera.org:8080/#/c/17645/23/be/src/service/impala-server.cc@2146
PS23, Line 2146:     // TODO Restart catalogd twice. When restarting for the second time, ddl executes
               :     // successfully, but we may receive the service id of the first restart, which will
               :     // cause the local cache to not be updated in time
nit: could you move these into the else-block at line 2164 and change them to

// We can't apply updates on another service id, because the local catalog is still inconsistent with the catalogd that executes the DDL. Catalogd may be restarted more than once inside a statestore update cycle. 'cur_service_id' could belong to 1) a stale update from the previous restarted catalogd, or 2) a newer update from next restarted catalogd. We are good to ignore the DDL result at the second case. However, in the first case clients may see stale catalog until the expected catalog topic update comes. TODO: handle the first case in IMPALA-10875.


http://gerrit.cloudera.org:8080/#/c/17645/23/tests/custom_cluster/test_restart_services.py
File tests/custom_cluster/test_restart_services.py:

http://gerrit.cloudera.org:8080/#/c/17645/23/tests/custom_cluster/test_restart_services.py@230
PS23, Line 230: execute_query_async
Currently execute_query() and execute_query_async() are the same for DDLs. But could you change this to execute_query()? This may cause test failure if one day we fix IMPALA-718. In that case, execute_query_async will return immediately and thread.join() at line 236 doesn't guard anything.



-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 23
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Fri, 20 Aug 2021 07:57:50 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-5476: Fix catalogd restart brings stale metadata

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix catalogd restart brings stale metadata
......................................................................


Patch Set 24: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 24
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Fri, 20 Aug 2021 15:21:05 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................


Patch Set 9:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/9140/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 9
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Thu, 22 Jul 2021 10:30:12 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................


Patch Set 13:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7346/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 13
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Mon, 26 Jul 2021 10:19:35 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................


Patch Set 16:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/17645/16/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:

http://gerrit.cloudera.org:8080/#/c/17645/16/be/src/service/impala-server.cc@2129
PS16, Line 2129:     CatalogUpdateResultIterator callback_ctx(catalog_update_result);
               :     TUpdateCatalogCacheRequest update_req;
               :     update_req.__set_is_delta(true);
               :     update_req.__set_native_iterator_ptr(reinterpret_cast<int64_t>(&callback_ctx));
               :     // The catalog version is updated in WaitForCatalogUpdate below. So we need a
               :     // standalone field in the request to update the service ID without touching the
               :     // catalog version.
               :     update_req.__set_catalog_service_id(catalog_update_result.catalog_service_id);
nit: Let's move these into the if-clause at line 2155.


http://gerrit.cloudera.org:8080/#/c/17645/16/tests/custom_cluster/test_restart_services.py
File tests/custom_cluster/test_restart_services.py:

http://gerrit.cloudera.org:8080/#/c/17645/16/tests/custom_cluster/test_restart_services.py@217
PS16, Line 217: 20
nit: Can we lower this down to 5?


http://gerrit.cloudera.org:8080/#/c/17645/16/tests/custom_cluster/test_restart_services.py@239
PS16, Line 239: sleep(self.UPDATE_FREQUENCY_S * 2)
While thinking about comments for this sleep, I realize this actually reveals a bug of our approach.

I add a log in CatalogServiceCatalog's constructor to print the service id. Let's say the 3 catalogd instances are using service id s0, s1 and s2. The WARNING log we want is

 Ignoring catalog update result of catalog service ID: s1. The expected catalog service ID: s1. Current catalog service ID: s2. Catalogd may be restarted more than once.

But what I saw is

 Ignoring catalog update result of catalog service ID: s2. The expected catalog service ID: s2. Current catalog service ID: s1. Catalogd may be restarted more than once.

This means the catalog of the first restarted catalogd is still propagated to the impalad, and we are ignoring the ddl update from the second restarted catalogd. This will lead to stale catalog until the next statestore update comes. The following test can reveal the trouble:

  UPDATE_FREQUENCY_S = 20

  @pytest.mark.execute_serially
  @CustomClusterTestSuite.with_args(
    statestored_args="--statestore_update_frequency_ms={frequency_ms}"
    .format(frequency_ms=(UPDATE_FREQUENCY_S * 1000)))
  def test_restart_catalogd_twice(self):
    self.execute_query_expect_success(self.client, "drop table if exists join_aa")
    self.execute_query_expect_success(self.client, "create table join_aa(id int)")
    # Make the catalog object version grow large enough
    self.execute_query_expect_success(self.client, "invalidate metadata")

    # No need to care whether the dll is executed successfully, it is just to make
    # the local catalog catche of impalad out of sync
    for i in range(0, 10):
      try:
        query = "alter table join_aa add columns (age" + str(i) + " int)"
        self.execute_query_async(query)
      except Exception, e:
        LOG.info(str(e))
    self.cluster.catalogd.restart()
    sleep(self.UPDATE_FREQUENCY_S * 2)
    self.cluster.catalogd.restart()

    self.execute_query_expect_success(self.client, "drop table join_aa")
    # Should not see stale metadata on 'join_aa'
    result = self.execute_query_expect_success(self.client, "show tables")
    assert 'join_aa' not in result.data

Do you have any ideas on fixing this?



-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 16
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Mon, 09 Aug 2021 01:59:11 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "liuyao (Code Review)" <ge...@cloudera.org>.
liuyao has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................


Patch Set 1:

When updating the local cache, catalogServiceId should not be changed here, because the full update will not be triggered here


-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 1
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Tue, 06 Jul 2021 11:24:36 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................


Patch Set 6: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7308/


-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 6
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Fri, 16 Jul 2021 18:41:53 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "liuyao (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Aman Sinha, Thomas Tauber-Marshall, Vihang Karajgaonkar, Zoltan Borok-Nagy, Tim Armstrong, Wenzhe Zhou, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/17645

to look at the new patch set (#16).

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................

IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

The change of catalogServiceId did not trigger a full topic update
request.

During the execution of DDL or DML, restarting Catalogd will cause
catalogServiceId to change. The DDL will execute successfully, and
then directly update the local cache (not through Statestored), modify
the local catalog with the new catalogServiceId, trigger an exception,
print the exception information, but did not submit full topic update
request. The version of Catalogd is lower than Impalad, so the metadata
information synchronized from Catalogd will be lost.

Under normal circumstances, if the catalogServiceId changes, Impalad
will submit full topic update request, Impalad will receive
TUpdateCatalogCacheRequest, and is_delta is false, that is full update.
Impalad will reset the local catalog cache, and then update, and the
version of CatalogD is greater than Impalad.

Testing

add TestRestart#test_restart_catalogd in
ests/custom_cluster/test_restart_services.py

Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
---
M be/src/service/impala-server.cc
M tests/custom_cluster/test_restart_services.py
2 files changed, 139 insertions(+), 5 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/17645/16
-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 16
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "liuyao (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Aman Sinha, Thomas Tauber-Marshall, Vihang Karajgaonkar, Zoltan Borok-Nagy, Tim Armstrong, Wenzhe Zhou, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/17645

to look at the new patch set (#21).

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................

IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

The change of catalogServiceId did not trigger a full topic update
request.

During the execution of DDL or DML, restarting Catalogd will cause
catalogServiceId to change. The DDL will execute successfully, and
then directly update the local cache (not through Statestored), modify
the local catalog with the new catalogServiceId, trigger an exception,
print the exception information, but did not submit full topic update
request. The version of Catalogd is lower than Impalad, so the metadata
information synchronized from Catalogd will be lost.

Under normal circumstances, if the catalogServiceId changes, Impalad
will submit full topic update request, Impalad will receive
TUpdateCatalogCacheRequest, and is_delta is false, that is full update.
Impalad will reset the local catalog cache, and then update, and the
version of CatalogD is greater than Impalad.

Testing

add TestRestart#test_restart_catalogd in
ests/custom_cluster/test_restart_services.py

Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
---
M be/src/service/impala-server.cc
M tests/custom_cluster/test_restart_services.py
2 files changed, 143 insertions(+), 13 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/17645/21
-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 21
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................


Patch Set 7:

(5 comments)

The current solution looks good to me. Could you also add test cases for the LocalCatalog mode? You can use anotation like this to enable it:

  @CustomClusterTestSuite.with_args(
      impalad_args="--use_local_catalog=true",
      catalogd_args="--catalog_topic_mode=minimal")

http://gerrit.cloudera.org:8080/#/c/17645/7/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:

http://gerrit.cloudera.org:8080/#/c/17645/7/be/src/service/impala-server.cc@a2142
PS7, Line 2142: 
Can we keep this? If we wait for the next statestore update, this won't cause trouble now.

In my understanding, this is the only difference we can use to distinguish DDL updates from statestore updates.


http://gerrit.cloudera.org:8080/#/c/17645/7/be/src/service/impala-server.cc@2144
PS7, Line 2144:         while (cur_service_id == catalog_update_info_.catalog_service_id) {
Could you add a log before the while-loop? E.g.

 Catalog service id mismatch. Current id: xxx. Id in response: yyy. Catalogd may be restarted. Waiting for new catalog update from statestore.


http://gerrit.cloudera.org:8080/#/c/17645/7/be/src/service/impala-server.cc@2150
PS7, Line 2150:     if (cur_service_id == catalog_service_id) {
I think this captures the case that catalogd restart again and we get another catalog_service_id from statestore update. Could you add a warning log in the else-clause? E.g.

 Ignoring catalog update result of catalog service id: xxx. The previous catalog service id is yyy. Current catalog service id is zzz. Catalogd may be restarted more than once.


http://gerrit.cloudera.org:8080/#/c/17645/4/tests/custom_cluster/test_restart_services.py
File tests/custom_cluster/test_restart_services.py:

http://gerrit.cloudera.org:8080/#/c/17645/4/tests/custom_cluster/test_restart_services.py@189
PS4, Line 189:     self.execute_query_expect_success(self.client, "select name from join_aa")
             :     self.exec
> I prefer Solution 2, this kind of modification is smaller and safer
Yeah, agree on these.


http://gerrit.cloudera.org:8080/#/c/17645/7/tests/custom_cluster/test_restart_services.py
File tests/custom_cluster/test_restart_services.py:

http://gerrit.cloudera.org:8080/#/c/17645/7/tests/custom_cluster/test_restart_services.py@169
PS7, Line 169:   def test_restart_catalogd(self):
This is already a good test case. Can we add another one to cover the case that catalogd is restarted twice?



-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 7
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Thu, 22 Jul 2021 00:54:20 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................


Patch Set 5:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/17645/5/tests/custom_cluster/test_restart_services.py
File tests/custom_cluster/test_restart_services.py:

http://gerrit.cloudera.org:8080/#/c/17645/5/tests/custom_cluster/test_restart_services.py@177
PS5, Line 177: ,
flake8: E231 missing whitespace after ','



-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 5
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Fri, 16 Jul 2021 04:46:55 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................


Patch Set 21:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/9331/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 21
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Fri, 20 Aug 2021 03:25:25 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................


Patch Set 1:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7273/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 1
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sat, 03 Jul 2021 11:05:54 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "liuyao (Code Review)" <ge...@cloudera.org>.
liuyao has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................


Patch Set 2:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/17645/1//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/17645/1//COMMIT_MSG@16
PS1, Line 16: but did not submit full topic update
            : request
> Shouldn't we fix this instead? The new catalog objects in the DDL
 > response come from the new catalogd instance. Will they be able to
 > update the local catalog cache? Or are they just got ignored due to
 > versions too small?

we should fix this bug.The new catalog objects in the DDL
response come from the new catalogd instance, they just got ignored due to versions too small

 > 
 > I think updating the local catalog cache immediately using the DDL
 > response aims to let the coordinator get latest catalog
 > immediately. After this patch, the coordinator will need to wait
 > for the next statestore topic update to get the latest catalog,
 > which doesn't match the existing functionality. Did I miss
 > anything?

This patch do not change the previous process. After the ddl is successfully executed, impalad will immediately modify the local catalog cache instead of waiting for the statestore notification.
When the ddl is executed successfully, it is reasonable for impalad to modify the local cache, but it is redundant to modify the service id every time, and it will cause the object version to be very high, causing the local catalog cache to be out of sync


http://gerrit.cloudera.org:8080/#/c/17645/1//COMMIT_MSG@24
PS1, Line 24: CatalogD
> CatalogD?
Done


http://gerrit.cloudera.org:8080/#/c/17645/1/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:

http://gerrit.cloudera.org:8080/#/c/17645/1/be/src/service/impala-server.cc@a2133
PS1, Line 2133: 
              : 
              : 
              : 
> Is a better solution to set the catalogServiceId_ only if firstRun is true 
Modifying the server id here may be redundant. Is it necessary to do so?



-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 2
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Tue, 13 Jul 2021 03:30:42 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "liuyao (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Aman Sinha, Thomas Tauber-Marshall, Vihang Karajgaonkar, Zoltan Borok-Nagy, Tim Armstrong, Wenzhe Zhou, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/17645

to look at the new patch set (#23).

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................

IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

The change of catalogServiceId did not trigger a full topic update
request.

During the execution of DDL or DML, restarting Catalogd will cause
catalogServiceId to change. The DDL will execute successfully, and
then directly update the local cache (not through Statestored), modify
the local catalog with the new catalogServiceId, trigger an exception,
print the exception information, but did not submit full topic update
request. The version of Catalogd is lower than Impalad, so the metadata
information synchronized from Catalogd will be lost.

Under normal circumstances, if the catalogServiceId changes, Impalad
will submit full topic update request, Impalad will receive
TUpdateCatalogCacheRequest, and is_delta is false, that is full update.
Impalad will reset the local catalog cache, and then update, and the
version of CatalogD is greater than Impalad.

Testing

add TestRestart#test_restart_catalogd in
ests/custom_cluster/test_restart_services.py

Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
---
M be/src/service/impala-server.cc
M tests/custom_cluster/test_restart_services.py
2 files changed, 144 insertions(+), 13 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/17645/23
-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 23
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................


Patch Set 7:

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7319/


-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 7
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Tue, 20 Jul 2021 08:50:04 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "liuyao (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Aman Sinha, Thomas Tauber-Marshall, Vihang Karajgaonkar, Zoltan Borok-Nagy, Tim Armstrong, Wenzhe Zhou, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/17645

to look at the new patch set (#15).

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................

IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

The change of catalogServiceId did not trigger a full topic update
request.

During the execution of DDL or DML, restarting Catalogd will cause
catalogServiceId to change. The DDL will execute successfully, and
then directly update the local cache (not through Statestored), modify
the local catalog with the new catalogServiceId, trigger an exception,
print the exception information, but did not submit full topic update
request. The version of Catalogd is lower than Impalad, so the metadata
information synchronized from Catalogd will be lost.

Under normal circumstances, if the catalogServiceId changes, Impalad
will submit full topic update request, Impalad will receive
TUpdateCatalogCacheRequest, and is_delta is false, that is full update.
Impalad will reset the local catalog cache, and then update, and the
version of CatalogD is greater than Impalad.

Testing

add TestRestart#test_restart_catalogd in
ests/custom_cluster/test_restart_services.py

Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
---
M be/src/service/impala-server.cc
M tests/custom_cluster/test_restart_services.py
2 files changed, 138 insertions(+), 5 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/17645/15
-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 15
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "liuyao (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Aman Sinha, Thomas Tauber-Marshall, Vihang Karajgaonkar, Zoltan Borok-Nagy, Tim Armstrong, Wenzhe Zhou, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/17645

to look at the new patch set (#24).

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................

IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

The change of catalogServiceId did not trigger a full topic update
request.

ImpaladCatalog#updateCatalog() doesn't trigger a full topic update
request when detecting catalogServiceId changes. It just updates the
local catalogServiceId and throws an exception to abort applying the
DDL/DML results. This causes a problem when catalogd is restarted and
the DDL/DML is executed on the restarted instance. In this case, only
the local catalogServiceId is updated to the latest. The local catalog
remains stale. Then when dealing with the following updates from
statestore, the catalogServiceId always matches, so updates will be
applied without exceptions. However, the catalog objects usually won't
be updated since they have higher versions (from the old catalogd
instance) than those in the update. This brings the local catalog out
of sync until the catalog version of the new catalogd grows larger
enough.

Note that in dealing with the catalog updates from statestore, if the
catalogServiceId unmatches, impalad will request a full topic update.
See more in ImpalaServer::CatalogUpdateCallback().

This patch fixes this issue by checking the catalogServiceId before
invoking UpdateCatalogCache() of FE. If catalogServiceId doesn't match
the one in the DDL/DML result, wait until it changes. The following
update from statestore will change it and unblocks the DDL/DML thread.

Testing

add several tests in
tests/custom_cluster/test_restart_services.py

Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
---
M be/src/service/impala-server.cc
M tests/custom_cluster/test_restart_services.py
2 files changed, 152 insertions(+), 13 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/17645/24
-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 24
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "liuyao (Code Review)" <ge...@cloudera.org>.
liuyao has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................


Patch Set 3:

add unit test


-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 3
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Tue, 13 Jul 2021 11:52:02 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................


Patch Set 1:

(2 comments)

Thanks a lot for reproducing and fixing the issue!

However, I think we don't have test coverage on this case, unfortunately. Can we add a test somewhere, e.g. tests/custom_cluster/test_restart_services.py, to cover the bug?

http://gerrit.cloudera.org:8080/#/c/17645/1//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/17645/1//COMMIT_MSG@16
PS1, Line 16: but did not submit full topic update
            : request
Shouldn't we fix this instead? The new catalog objects in the DDL response come from the new catalogd instance. Will they be able to update the local catalog cache? Or are they just got ignored due to versions too small?

I think updating the local catalog cache immediately using the DDL response aims to let the coordinator get latest catalog immediately. After this patch, the coordinator will need to wait for the next statestore topic update to get the latest catalog, which doesn't match the existing functionality. Did I miss anything?


http://gerrit.cloudera.org:8080/#/c/17645/1//COMMIT_MSG@24
PS1, Line 24: DatalogD
CatalogD?



-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 1
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Mon, 12 Jul 2021 11:32:28 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................


Patch Set 23:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/9333/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 23
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Fri, 20 Aug 2021 03:49:05 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................


Patch Set 6:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7308/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 6
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Fri, 16 Jul 2021 12:42:29 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................


Patch Set 10:

(8 comments)

The patch is pretty good now. Just have some minor comments.

http://gerrit.cloudera.org:8080/#/c/17645/10/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:

http://gerrit.cloudera.org:8080/#/c/17645/10/be/src/service/impala-server.cc@2149
PS10, Line 2149: :
nit: add a space after ":"


http://gerrit.cloudera.org:8080/#/c/17645/10/be/src/service/impala-server.cc@2149
PS10, Line 2149: cur_service_id
We need PrintId(cur_service_id) to get human-readable id. Same for other id loggings.


http://gerrit.cloudera.org:8080/#/c/17645/10/be/src/service/impala-server.cc@2150
PS10, Line 2150: .ID in response:
nit: add a space after "." and ":"


http://gerrit.cloudera.org:8080/#/c/17645/10/be/src/service/impala-server.cc@2167
PS10, Line 2167: :
nit: add a space after each ":"


http://gerrit.cloudera.org:8080/#/c/17645/10/be/src/service/impala-server.cc@2168
PS10, Line 2168: ". The expected catalog service ID:"
               :           << catalog_service_id <<
nit: we can remove this if it's logging the same id.


http://gerrit.cloudera.org:8080/#/c/17645/10/tests/custom_cluster/test_restart_services.py
File tests/custom_cluster/test_restart_services.py:

http://gerrit.cloudera.org:8080/#/c/17645/10/tests/custom_cluster/test_restart_services.py@186
PS10, Line 186:     self.execute_query_expect_success(self.client,
              :         "alter table join_aa add columns (name string)")
Could you add another test with this being executed with sync_ddl=true? Just want to make sure the logics of WaitForCatalogUpdateTopicPropagation works normally.


http://gerrit.cloudera.org:8080/#/c/17645/10/tests/custom_cluster/test_restart_services.py@208
PS10, Line 208:       if i == 5:
              :         self.cluster.catalogd.restart()
              :       if i == 6:
              :         self.cluster.catalogd.restart()
I ran this test several times but I'm not able to see the WARNING log of "Ignoring catalog update result of catalog service ID". Could you verify that this cover that branch? The log file of custom cluster test is /tmp/impalad.INFO.

Maybe we just need to merge this two if-clause into one. So catalogd is restarted twice quickly.


http://gerrit.cloudera.org:8080/#/c/17645/10/tests/custom_cluster/test_restart_services.py@223
PS10, Line 223: test_restart_local_catalog
nit: rename to "test_restart_catalogd_with_local_catalog"



-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 10
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Mon, 26 Jul 2021 03:00:57 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "liuyao (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Aman Sinha, Thomas Tauber-Marshall, Vihang Karajgaonkar, Zoltan Borok-Nagy, Tim Armstrong, Wenzhe Zhou, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/17645

to look at the new patch set (#20).

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................

IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

The change of catalogServiceId did not trigger a full topic update
request.

During the execution of DDL or DML, restarting Catalogd will cause
catalogServiceId to change. The DDL will execute successfully, and
then directly update the local cache (not through Statestored), modify
the local catalog with the new catalogServiceId, trigger an exception,
print the exception information, but did not submit full topic update
request. The version of Catalogd is lower than Impalad, so the metadata
information synchronized from Catalogd will be lost.

Under normal circumstances, if the catalogServiceId changes, Impalad
will submit full topic update request, Impalad will receive
TUpdateCatalogCacheRequest, and is_delta is false, that is full update.
Impalad will reset the local catalog cache, and then update, and the
version of CatalogD is greater than Impalad.

Testing

add TestRestart#test_restart_catalogd in
ests/custom_cluster/test_restart_services.py

Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
---
M be/src/service/impala-server.cc
M tests/custom_cluster/test_restart_services.py
2 files changed, 186 insertions(+), 13 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/17645/20
-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 20
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "liuyao (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Aman Sinha, Thomas Tauber-Marshall, Vihang Karajgaonkar, Zoltan Borok-Nagy, Tim Armstrong, Wenzhe Zhou, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/17645

to look at the new patch set (#4).

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................

IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

The change of catalogServiceId did not trigger a full topic update
request.

During the execution of DDL or DML, restarting Catalogd will cause
catalogServiceId to change. The DDL will execute successfully, and
then directly update the local cache (not through Statestored), modify
the local catalog with the new catalogServiceId, trigger an exception,
print the exception information, but did not submit full topic update
request. The version of Catalogd is lower than Impalad, so the metadata
information synchronized from Catalogd will be lost.

Under normal circumstances, if the catalogServiceId changes, Impalad
will submit full topic update request, Impalad will receive
TUpdateCatalogCacheRequest, and is_delta is false, that is full update.
Impalad will reset the local catalog cache, and then update, and the
version of CatalogD is greater than Impalad.

Testing

add TestRestart#test_restart_catalogd in
ests/custom_cluster/test_restart_services.py

Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
---
M be/src/service/impala-server.cc
M tests/custom_cluster/test_restart_services.py
2 files changed, 29 insertions(+), 4 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/17645/4
-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 4
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "Vihang Karajgaonkar (Code Review)" <ge...@cloudera.org>.
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/17645/1/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:

http://gerrit.cloudera.org:8080/#/c/17645/1/be/src/service/impala-server.cc@a2133
PS1, Line 2133: 
              : 
              : 
              : 
Is a better solution to set the catalogServiceId_ only if firstRun is true here?
https://github.com/apache/impala/blob/4d01d8eb517770628f77406d83f484f02d4d21db/fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java#L172



-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 1
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Mon, 12 Jul 2021 22:49:52 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................


Patch Set 4:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/17645/4/tests/custom_cluster/test_restart_services.py
File tests/custom_cluster/test_restart_services.py:

http://gerrit.cloudera.org:8080/#/c/17645/4/tests/custom_cluster/test_restart_services.py@189
PS4, Line 189:     # Wait enough time to synchronize the metadata
             :     sleep(20)
We should not depend on sleep. This is what I mean in the previous comment: https://gerrit.cloudera.org/c/17645/1//COMMIT_MSG#17

When AlterTable finishes, the metadata of this table should be up-to-date. The current patch can't satify this.



-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 4
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Fri, 16 Jul 2021 02:52:19 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................


Patch Set 3:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/17645/3/tests/custom_cluster/test_restart_services.py
File tests/custom_cluster/test_restart_services.py:

http://gerrit.cloudera.org:8080/#/c/17645/3/tests/custom_cluster/test_restart_services.py@165
PS3, Line 165:     
flake8: W293 blank line contains whitespace


http://gerrit.cloudera.org:8080/#/c/17645/3/tests/custom_cluster/test_restart_services.py@165
PS3, Line 165:     
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/17645/3/tests/custom_cluster/test_restart_services.py@177
PS3, Line 177: ,
flake8: E231 missing whitespace after ','



-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 3
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Tue, 13 Jul 2021 11:51:46 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................


Patch Set 7:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7319/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 7
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Tue, 20 Jul 2021 02:28:15 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "liuyao (Code Review)" <ge...@cloudera.org>.
liuyao has removed Tim Armstrong from this change.  ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................


Removed reviewer Tim Armstrong.
-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: deleteReviewer
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 1
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................


Patch Set 6:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7309/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 6
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Sat, 17 Jul 2021 02:24:46 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "liuyao (Code Review)" <ge...@cloudera.org>.
liuyao has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................


Patch Set 13:

(6 comments)

http://gerrit.cloudera.org:8080/#/c/17645/10/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:

http://gerrit.cloudera.org:8080/#/c/17645/10/be/src/service/impala-server.cc@2150
PS10, Line 2150: g_version_update
> nit: add a space after "." and ":"
Done


http://gerrit.cloudera.org:8080/#/c/17645/10/be/src/service/impala-server.cc@2167
PS10, Line 2167: 
> nit: add a space after each ":"
Done


http://gerrit.cloudera.org:8080/#/c/17645/10/be/src/service/impala-server.cc@2168
PS10, Line 2168: ocess the catalog update that covers the effects
               :     // (catalog objects) of this o
> nit: we can remove this if it's logging the same id.
Done


http://gerrit.cloudera.org:8080/#/c/17645/10/tests/custom_cluster/test_restart_services.py
File tests/custom_cluster/test_restart_services.py:

http://gerrit.cloudera.org:8080/#/c/17645/10/tests/custom_cluster/test_restart_services.py@186
PS10, Line 186:     self.execute_query_expect_success(self.client,
              :         "alter table join_aa add columns (name string)")
> Could you add another test with this being executed with sync_ddl=true? Jus
Done


http://gerrit.cloudera.org:8080/#/c/17645/10/tests/custom_cluster/test_restart_services.py@208
PS10, Line 208:         LOG.info(str(e))
              :       if i == 5:
              :         self.cluster.catalogd.restart()
              : 
> I ran this test several times but I'm not able to see the WARNING log of "I
I tried many times, but the WARNING  log was not printed. It may be because of the following reason: after catalogd restarts successfully for the second time, catalogd will not immediately notify statestored to clear catalog topic entries. After waiting for "statestore_update_frequency_ms" milliseconds, catalogd receives the update request from statestored, and then catalogd notifies statestored to clear the catalog topic entries. Therefore, impalad has enough time to complete the first full update, which is triggered by the first restart of catalogd.


http://gerrit.cloudera.org:8080/#/c/17645/10/tests/custom_cluster/test_restart_services.py@223
PS10, Line 223: Make the catalog object ve
> nit: rename to "test_restart_catalogd_with_local_catalog"
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 13
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Mon, 26 Jul 2021 06:05:00 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................


Patch Set 12:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/9162/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 12
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Mon, 26 Jul 2021 06:01:26 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................


Patch Set 3:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/9084/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 3
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Tue, 13 Jul 2021 12:12:40 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................


Patch Set 11:

(4 comments)

http://gerrit.cloudera.org:8080/#/c/17645/11/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:

http://gerrit.cloudera.org:8080/#/c/17645/11/be/src/service/impala-server.cc@2143
PS11, Line 2143:         LOG(INFO) << "Catalog service ID mismatch. Current ID: " << PrintId(cur_service_id)
line too long (91 > 90)


http://gerrit.cloudera.org:8080/#/c/17645/11/tests/custom_cluster/test_restart_services.py
File tests/custom_cluster/test_restart_services.py:

http://gerrit.cloudera.org:8080/#/c/17645/11/tests/custom_cluster/test_restart_services.py@190
PS11, Line 190:   
flake8: W293 blank line contains whitespace


http://gerrit.cloudera.org:8080/#/c/17645/11/tests/custom_cluster/test_restart_services.py@190
PS11, Line 190:   
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/17645/11/tests/custom_cluster/test_restart_services.py@199
PS11, Line 199:  
flake8: E203 whitespace before ':'



-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 11
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Mon, 26 Jul 2021 05:31:23 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................


Patch Set 8:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/17645/8/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:

http://gerrit.cloudera.org:8080/#/c/17645/8/be/src/service/impala-server.cc@2163
PS8, Line 2163:           << catalog_service_id << ". Current catalog service ID:" 
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/17645/8/tests/custom_cluster/test_restart_services.py
File tests/custom_cluster/test_restart_services.py:

http://gerrit.cloudera.org:8080/#/c/17645/8/tests/custom_cluster/test_restart_services.py@190
PS8, Line 190:   
flake8: W293 blank line contains whitespace


http://gerrit.cloudera.org:8080/#/c/17645/8/tests/custom_cluster/test_restart_services.py@190
PS8, Line 190:   
line has trailing whitespace



-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 8
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Thu, 22 Jul 2021 09:54:53 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-5476: Fix catalogd restart brings stale metadata

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix catalogd restart brings stale metadata
......................................................................


Patch Set 26: Code-Review+2

Thanks for the contribution and the patience on addressing all the comments!


-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 26
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Fri, 20 Aug 2021 10:41:23 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................


Patch Set 24:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7407/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 24
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Fri, 20 Aug 2021 09:02:31 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "liuyao (Code Review)" <ge...@cloudera.org>.
liuyao has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................


Patch Set 7:

16:50:03 ERROR: cannot verify jenkins.impala.io's certificate, issued by ‘CN=Go Daddy Secure Certificate Authority - G2,OU=http://certs.godaddy.com/repository/,O=GoDaddy.com\\, Inc.,L=Scottsdale,ST=Arizona,C=US’:
16:50:03   Issued certificate has expired.
16:50:03 To connect to jenkins.impala.io insecurely, use `--no-check-certificate'.
16:50:03 + ssh -p 29418 impala-public-jenkins@gerrit.cloudera.org gerrit review --verified -1 --message '"Build' failed: 'https://jenkins.impala.io/job/gerrit-verify-dryrun/7319/"' --project Impala-ASF 17645,7
16:50:05 + exit 1
16:50:05 Build step 'Execute shell' marked build as failure


-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 7
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Wed, 21 Jul 2021 09:59:01 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................


Patch Set 22:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/9332/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 22
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Fri, 20 Aug 2021 03:32:24 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "liuyao (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Aman Sinha, Thomas Tauber-Marshall, Vihang Karajgaonkar, Zoltan Borok-Nagy, Tim Armstrong, Wenzhe Zhou, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/17645

to look at the new patch set (#5).

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................

IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

The change of catalogServiceId did not trigger a full topic update
request.

During the execution of DDL or DML, restarting Catalogd will cause
catalogServiceId to change. The DDL will execute successfully, and
then directly update the local cache (not through Statestored), modify
the local catalog with the new catalogServiceId, trigger an exception,
print the exception information, but did not submit full topic update
request. The version of Catalogd is lower than Impalad, so the metadata
information synchronized from Catalogd will be lost.

Under normal circumstances, if the catalogServiceId changes, Impalad
will submit full topic update request, Impalad will receive
TUpdateCatalogCacheRequest, and is_delta is false, that is full update.
Impalad will reset the local catalog cache, and then update, and the
version of CatalogD is greater than Impalad.

Testing

add TestRestart#test_restart_catalogd in
ests/custom_cluster/test_restart_services.py

Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
---
M be/src/service/impala-server.cc
M tests/custom_cluster/test_restart_services.py
2 files changed, 44 insertions(+), 9 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/17645/5
-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 5
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................


Patch Set 15:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/9251/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 15
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 06 Aug 2021 10:55:41 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "liuyao (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Aman Sinha, Thomas Tauber-Marshall, Vihang Karajgaonkar, Zoltan Borok-Nagy, Tim Armstrong, Wenzhe Zhou, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/17645

to look at the new patch set (#3).

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................

IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

The change of catalogServiceId did not trigger a full topic update
request.

During the execution of DDL or DML, restarting Catalogd will cause
catalogServiceId to change. The DDL will execute successfully, and
then directly update the local cache (not through Statestored), modify
the local catalog with the new catalogServiceId, trigger an exception,
print the exception information, but did not submit full topic update
request. The version of Catalogd is lower than Impalad, so the metadata
information synchronized from Catalogd will be lost.

Under normal circumstances, if the catalogServiceId changes, Impalad
will submit full topic update request, Impalad will receive
TUpdateCatalogCacheRequest, and is_delta is false, that is full update.
Impalad will reset the local catalog cache, and then update, and the
version of CatalogD is greater than Impalad.

Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
---
M be/src/service/impala-server.cc
M tests/custom_cluster/test_restart_services.py
2 files changed, 29 insertions(+), 4 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/17645/3
-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 3
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................


Patch Set 20:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/9324/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 20
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Thu, 19 Aug 2021 09:44:12 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................


Patch Set 17:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/9301/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 17
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Mon, 16 Aug 2021 03:38:48 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "liuyao (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Aman Sinha, Thomas Tauber-Marshall, Vihang Karajgaonkar, Zoltan Borok-Nagy, Tim Armstrong, Wenzhe Zhou, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/17645

to look at the new patch set (#22).

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................

IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

The change of catalogServiceId did not trigger a full topic update
request.

During the execution of DDL or DML, restarting Catalogd will cause
catalogServiceId to change. The DDL will execute successfully, and
then directly update the local cache (not through Statestored), modify
the local catalog with the new catalogServiceId, trigger an exception,
print the exception information, but did not submit full topic update
request. The version of Catalogd is lower than Impalad, so the metadata
information synchronized from Catalogd will be lost.

Under normal circumstances, if the catalogServiceId changes, Impalad
will submit full topic update request, Impalad will receive
TUpdateCatalogCacheRequest, and is_delta is false, that is full update.
Impalad will reset the local catalog cache, and then update, and the
version of CatalogD is greater than Impalad.

Testing

add TestRestart#test_restart_catalogd in
ests/custom_cluster/test_restart_services.py

Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
---
M be/src/service/impala-server.cc
M tests/custom_cluster/test_restart_services.py
2 files changed, 144 insertions(+), 13 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/17645/22
-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 22
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>

[Impala-ASF-CR] IMPALA-5476: Fix catalogd restart brings stale metadata

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix catalogd restart brings stale metadata
......................................................................


Patch Set 25:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/9335/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 25
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Fri, 20 Aug 2021 09:37:22 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................


Patch Set 16:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/9252/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 16
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Fri, 06 Aug 2021 11:02:21 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "liuyao (Code Review)" <ge...@cloudera.org>.
liuyao has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................


Patch Set 24:

(7 comments)

http://gerrit.cloudera.org:8080/#/c/17645/23//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/17645/23//COMMIT_MSG@9
PS23, Line 9: The change of catalogServiceId did not trigger a full topic update
            : request.
            : 
            : ImpaladCatalog#updateCatalog() doesn't trigger a full topic update
            : request when detecting catalogServiceId changes. It just updates the
            : local catalogServiceId and throws an exception to abort applying the
            : DDL/DML results. This causes a problem when catalogd is restarted and
            : the DDL/DML is executed on the restarted instance. In this case, only
            : the local catalogServiceId is updated to the latest. The local catalog
            : remains stale. Then when dealing with the following 
> nit: ImpaladCatalog#updateCatalog() doesn't trigger a full topic update req
Done


http://gerrit.cloudera.org:8080/#/c/17645/23//COMMIT_MSG@20
PS23, Line 20: applied without exceptions. However, the catalog objects usually won't
             : be updated since they have higher versions (from the old catalogd
             : instance) than those in the update. This brings the local catalog out
             : of sync until the catalog version of the new catalogd grows larger
             : enough.
> nit: Note that in dealing with the catalog updates from statestore, if the 
Done


http://gerrit.cloudera.org:8080/#/c/17645/23//COMMIT_MSG@25
PS23, Line 25: 
> Let's add some description about how we fix the bug:
Done


http://gerrit.cloudera.org:8080/#/c/17645/23//COMMIT_MSG@28
PS23, Line 28: more in ImpalaServer::CatalogUpda
> nit: several tests
Done


http://gerrit.cloudera.org:8080/#/c/17645/23//COMMIT_MSG@29
PS23, Line 29: 
> nit: tests
Done


http://gerrit.cloudera.org:8080/#/c/17645/23/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:

http://gerrit.cloudera.org:8080/#/c/17645/23/be/src/service/impala-server.cc@2146
PS23, Line 2146:     // TODO Restart catalogd twice. When restarting for the second time, ddl executes
               :     // successfully, but we may receive the service id of the first restart, which will
               :     // cause the local cache to not be updated in time
> nit: could you move these into the else-block at line 2164 and change them 
Done


http://gerrit.cloudera.org:8080/#/c/17645/23/tests/custom_cluster/test_restart_services.py
File tests/custom_cluster/test_restart_services.py:

http://gerrit.cloudera.org:8080/#/c/17645/23/tests/custom_cluster/test_restart_services.py@230
PS23, Line 230: execute_query(query
> Currently execute_query() and execute_query_async() are the same for DDLs. 
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 24
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Fri, 20 Aug 2021 08:50:13 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................


Patch Set 12:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/17645/12/tests/custom_cluster/test_restart_services.py
File tests/custom_cluster/test_restart_services.py:

http://gerrit.cloudera.org:8080/#/c/17645/12/tests/custom_cluster/test_restart_services.py@199
PS12, Line 199:  
flake8: E203 whitespace before ':'



-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 12
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Mon, 26 Jul 2021 05:39:57 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................


Patch Set 14:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/17645/10/tests/custom_cluster/test_restart_services.py
File tests/custom_cluster/test_restart_services.py:

http://gerrit.cloudera.org:8080/#/c/17645/10/tests/custom_cluster/test_restart_services.py@208
PS10, Line 208:         LOG.info(str(e))
              :       if i == 5:
              :         self.cluster.catalogd.restart()
              : 
> I tried many times, but the WARNING  log was not printed. It may be because
statestored will send topic update request to catalogd once it's registered. I ran the test and realized that we restart catalogd too quickly, i.e. catalogd hasn't been ready after restart but we restart it again. So impalad is actually talking to the second restarted catalogd.

The problem is restart() returns too early. Connected to statestored doesn't mean catalogd is ready: https://github.com/apache/impala/blob/ca17e307ab3abb2c95c27b3ba749adf6bf16efc7/tests/common/impala_cluster.py#L569-L571

We probably need to add a metric like "catalog-server.ready" to indicate the start is finish.



-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 14
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 05 Aug 2021 03:28:10 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "liuyao (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Aman Sinha, Thomas Tauber-Marshall, Vihang Karajgaonkar, Zoltan Borok-Nagy, Tim Armstrong, Wenzhe Zhou, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/17645

to look at the new patch set (#11).

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................

IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

The change of catalogServiceId did not trigger a full topic update
request.

During the execution of DDL or DML, restarting Catalogd will cause
catalogServiceId to change. The DDL will execute successfully, and
then directly update the local cache (not through Statestored), modify
the local catalog with the new catalogServiceId, trigger an exception,
print the exception information, but did not submit full topic update
request. The version of Catalogd is lower than Impalad, so the metadata
information synchronized from Catalogd will be lost.

Under normal circumstances, if the catalogServiceId changes, Impalad
will submit full topic update request, Impalad will receive
TUpdateCatalogCacheRequest, and is_delta is false, that is full update.
Impalad will reset the local catalog cache, and then update, and the
version of CatalogD is greater than Impalad.

Testing

add TestRestart#test_restart_catalogd in
ests/custom_cluster/test_restart_services.py

Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
---
M be/src/service/impala-server.cc
M tests/custom_cluster/test_restart_services.py
2 files changed, 134 insertions(+), 5 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/17645/11
-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 11
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................


Patch Set 21:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/17645/21/tests/custom_cluster/test_restart_services.py
File tests/custom_cluster/test_restart_services.py:

http://gerrit.cloudera.org:8080/#/c/17645/21/tests/custom_cluster/test_restart_services.py@228
PS21, Line 228: d
flake8: E306 expected 1 blank line before a nested definition, found 0



-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 21
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Fri, 20 Aug 2021 03:01:53 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-5476: Fix catalogd restart brings stale metadata

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix catalogd restart brings stale metadata
......................................................................


Patch Set 27:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7408/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 27
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Fri, 20 Aug 2021 10:42:34 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "liuyao (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Aman Sinha, Thomas Tauber-Marshall, Vihang Karajgaonkar, Zoltan Borok-Nagy, Tim Armstrong, Wenzhe Zhou, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/17645

to look at the new patch set (#9).

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................

IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

The change of catalogServiceId did not trigger a full topic update
request.

During the execution of DDL or DML, restarting Catalogd will cause
catalogServiceId to change. The DDL will execute successfully, and
then directly update the local cache (not through Statestored), modify
the local catalog with the new catalogServiceId, trigger an exception,
print the exception information, but did not submit full topic update
request. The version of Catalogd is lower than Impalad, so the metadata
information synchronized from Catalogd will be lost.

Under normal circumstances, if the catalogServiceId changes, Impalad
will submit full topic update request, Impalad will receive
TUpdateCatalogCacheRequest, and is_delta is false, that is full update.
Impalad will reset the local catalog cache, and then update, and the
version of CatalogD is greater than Impalad.

Testing

add TestRestart#test_restart_catalogd in
ests/custom_cluster/test_restart_services.py

Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
---
M be/src/service/impala-server.cc
M tests/custom_cluster/test_restart_services.py
2 files changed, 109 insertions(+), 5 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/17645/9
-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 9
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................


Patch Set 19:

(5 comments)

http://gerrit.cloudera.org:8080/#/c/17645/19/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:

http://gerrit.cloudera.org:8080/#/c/17645/19/be/src/service/impala-server.cc@2148
PS19, Line 2148:             && FLAGS_catalog_update_max_wait_time_s > 
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/17645/19/tests/custom_cluster/test_restart_services.py
File tests/custom_cluster/test_restart_services.py:

http://gerrit.cloudera.org:8080/#/c/17645/19/tests/custom_cluster/test_restart_services.py@246
PS19, Line 246:   
flake8: W293 blank line contains whitespace


http://gerrit.cloudera.org:8080/#/c/17645/19/tests/custom_cluster/test_restart_services.py@246
PS19, Line 246:   
line has trailing whitespace


http://gerrit.cloudera.org:8080/#/c/17645/19/tests/custom_cluster/test_restart_services.py@248
PS19, Line 248: U
flake8: E303 too many blank lines (2)


http://gerrit.cloudera.org:8080/#/c/17645/19/tests/custom_cluster/test_restart_services.py@249
PS19, Line 249: @
flake8: E301 expected 1 blank line, found 0



-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 19
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Thu, 19 Aug 2021 09:17:54 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "liuyao (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Aman Sinha, Thomas Tauber-Marshall, Vihang Karajgaonkar, Zoltan Borok-Nagy, Tim Armstrong, Wenzhe Zhou, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/17645

to look at the new patch set (#12).

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................

IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

The change of catalogServiceId did not trigger a full topic update
request.

During the execution of DDL or DML, restarting Catalogd will cause
catalogServiceId to change. The DDL will execute successfully, and
then directly update the local cache (not through Statestored), modify
the local catalog with the new catalogServiceId, trigger an exception,
print the exception information, but did not submit full topic update
request. The version of Catalogd is lower than Impalad, so the metadata
information synchronized from Catalogd will be lost.

Under normal circumstances, if the catalogServiceId changes, Impalad
will submit full topic update request, Impalad will receive
TUpdateCatalogCacheRequest, and is_delta is false, that is full update.
Impalad will reset the local catalog cache, and then update, and the
version of CatalogD is greater than Impalad.

Testing

add TestRestart#test_restart_catalogd in
ests/custom_cluster/test_restart_services.py

Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
---
M be/src/service/impala-server.cc
M tests/custom_cluster/test_restart_services.py
2 files changed, 135 insertions(+), 5 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/17645/12
-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 12
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "liuyao (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Aman Sinha, Thomas Tauber-Marshall, Vihang Karajgaonkar, Zoltan Borok-Nagy, Tim Armstrong, Wenzhe Zhou, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/17645

to look at the new patch set (#13).

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................

IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

The change of catalogServiceId did not trigger a full topic update
request.

During the execution of DDL or DML, restarting Catalogd will cause
catalogServiceId to change. The DDL will execute successfully, and
then directly update the local cache (not through Statestored), modify
the local catalog with the new catalogServiceId, trigger an exception,
print the exception information, but did not submit full topic update
request. The version of Catalogd is lower than Impalad, so the metadata
information synchronized from Catalogd will be lost.

Under normal circumstances, if the catalogServiceId changes, Impalad
will submit full topic update request, Impalad will receive
TUpdateCatalogCacheRequest, and is_delta is false, that is full update.
Impalad will reset the local catalog cache, and then update, and the
version of CatalogD is greater than Impalad.

Testing

add TestRestart#test_restart_catalogd in
ests/custom_cluster/test_restart_services.py

Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
---
M be/src/service/impala-server.cc
M tests/custom_cluster/test_restart_services.py
2 files changed, 135 insertions(+), 5 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/17645/13
-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 13
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>

[Impala-ASF-CR] IMPALA-5476: Fix catalogd restart brings stale metadata

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix catalogd restart brings stale metadata
......................................................................


Patch Set 27: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 27
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Fri, 20 Aug 2021 17:09:07 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................


Patch Set 6: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 6
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Sat, 17 Jul 2021 08:30:55 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "liuyao (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Aman Sinha, Thomas Tauber-Marshall, Vihang Karajgaonkar, Zoltan Borok-Nagy, Tim Armstrong, Wenzhe Zhou, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/17645

to look at the new patch set (#17).

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................

IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

The change of catalogServiceId did not trigger a full topic update
request.

During the execution of DDL or DML, restarting Catalogd will cause
catalogServiceId to change. The DDL will execute successfully, and
then directly update the local cache (not through Statestored), modify
the local catalog with the new catalogServiceId, trigger an exception,
print the exception information, but did not submit full topic update
request. The version of Catalogd is lower than Impalad, so the metadata
information synchronized from Catalogd will be lost.

Under normal circumstances, if the catalogServiceId changes, Impalad
will submit full topic update request, Impalad will receive
TUpdateCatalogCacheRequest, and is_delta is false, that is full update.
Impalad will reset the local catalog cache, and then update, and the
version of CatalogD is greater than Impalad.

Testing

add TestRestart#test_restart_catalogd in
ests/custom_cluster/test_restart_services.py

Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
---
M be/src/service/impala-server.cc
M tests/custom_cluster/test_restart_services.py
2 files changed, 146 insertions(+), 13 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/17645/17
-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 17
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................


Patch Set 8:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/9139/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 8
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Thu, 22 Jul 2021 10:20:30 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................


Patch Set 19:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/9323/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 19
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Thu, 19 Aug 2021 09:39:27 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................


Patch Set 6:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/9101/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 6
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Fri, 16 Jul 2021 06:51:38 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5476: Fix catalogd restart brings stale metadata

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix catalogd restart brings stale metadata
......................................................................


Patch Set 26:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/9336/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 26
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Fri, 20 Aug 2021 10:59:52 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................


Patch Set 24:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/9334/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 24
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Fri, 20 Aug 2021 09:07:04 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................


Patch Set 11:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/9161/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 11
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Mon, 26 Jul 2021 05:51:28 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................


Patch Set 22:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/17645/22/tests/custom_cluster/test_restart_services.py
File tests/custom_cluster/test_restart_services.py:

http://gerrit.cloudera.org:8080/#/c/17645/22/tests/custom_cluster/test_restart_services.py@228
PS22, Line 228:     
flake8: W293 blank line contains whitespace


http://gerrit.cloudera.org:8080/#/c/17645/22/tests/custom_cluster/test_restart_services.py@228
PS22, Line 228:     
line has trailing whitespace



-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 22
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Fri, 20 Aug 2021 03:09:59 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................


Patch Set 15:

(4 comments)

http://gerrit.cloudera.org:8080/#/c/17645/15/tests/custom_cluster/test_restart_services.py
File tests/custom_cluster/test_restart_services.py:

http://gerrit.cloudera.org:8080/#/c/17645/15/tests/custom_cluster/test_restart_services.py@218
PS15, Line 218: @
flake8: E301 expected 1 blank line, found 0


http://gerrit.cloudera.org:8080/#/c/17645/15/tests/custom_cluster/test_restart_services.py@221
PS15, Line 221:  
flake8: E251 unexpected spaces around keyword / parameter equals


http://gerrit.cloudera.org:8080/#/c/17645/15/tests/custom_cluster/test_restart_services.py@221
PS15, Line 221:  
flake8: E251 unexpected spaces around keyword / parameter equals


http://gerrit.cloudera.org:8080/#/c/17645/15/tests/custom_cluster/test_restart_services.py@221
PS15, Line 221: *
flake8: E226 missing whitespace around arithmetic operator



-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 15
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 06 Aug 2021 10:35:17 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-5476: Fix catalogd restart brings stale metadata

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix catalogd restart brings stale metadata
......................................................................

IMPALA-5476: Fix catalogd restart brings stale metadata

ImpaladCatalog#updateCatalog() doesn't trigger a full topic update
request when detecting catalogServiceId changes. It just updates the
local catalogServiceId and throws an exception to abort applying the
DDL/DML results. This causes a problem when catalogd is restarted and
the DDL/DML is executed on the restarted instance. In this case, only
the local catalogServiceId is updated to the latest. The local catalog
remains stale. Then when dealing with the following updates from
statestore, the catalogServiceId always matches, so updates will be
applied without exceptions. However, the catalog objects usually won't
be updated since they have higher versions (from the old catalogd
instance) than those in the update. This brings the local catalog out
of sync until the catalog version of the new catalogd grows larger
enough.

Note that in dealing with the catalog updates from statestore, if the
catalogServiceId unmatches, impalad will request a full topic update.
See more in ImpalaServer::CatalogUpdateCallback().

This patch fixes this issue by checking the catalogServiceId before
invoking UpdateCatalogCache() of FE. If catalogServiceId doesn't match
the one in the DDL/DML result, wait until it changes. The following
update from statestore will change it and unblocks the DDL/DML thread.

Testing

add several tests in
tests/custom_cluster/test_restart_services.py

Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Reviewed-on: http://gerrit.cloudera.org:8080/17645
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/service/impala-server.cc
M tests/custom_cluster/test_restart_services.py
2 files changed, 150 insertions(+), 13 deletions(-)

Approvals:
  Impala Public Jenkins: Looks good to me, approved; Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 28
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>

[Impala-ASF-CR] IMPALA-5476: Fix catalogd restart brings stale metadata

Posted by "liuyao (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Aman Sinha, Thomas Tauber-Marshall, Vihang Karajgaonkar, Zoltan Borok-Nagy, Tim Armstrong, Wenzhe Zhou, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/17645

to look at the new patch set (#25).

Change subject: IMPALA-5476: Fix catalogd restart brings stale metadata
......................................................................

IMPALA-5476: Fix catalogd restart brings stale metadata

ImpaladCatalog#updateCatalog() doesn't trigger a full topic update
request when detecting catalogServiceId changes. It just updates the
local catalogServiceId and throws an exception to abort applying the
DDL/DML results. This causes a problem when catalogd is restarted and
the DDL/DML is executed on the restarted instance. In this case, only
the local catalogServiceId is updated to the latest. The local catalog
remains stale. Then when dealing with the following updates from
statestore, the catalogServiceId always matches, so updates will be
applied without exceptions. However, the catalog objects usually won't
be updated since they have higher versions (from the old catalogd
instance) than those in the update. This brings the local catalog out
of sync until the catalog version of the new catalogd grows larger
enough.

Note that in dealing with the catalog updates from statestore, if the
catalogServiceId unmatches, impalad will request a full topic update.
See more in ImpalaServer::CatalogUpdateCallback().

This patch fixes this issue by checking the catalogServiceId before
invoking UpdateCatalogCache() of FE. If catalogServiceId doesn't match
the one in the DDL/DML result, wait until it changes. The following
update from statestore will change it and unblocks the DDL/DML thread.

Testing

add several tests in
tests/custom_cluster/test_restart_services.py

Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
---
M be/src/service/impala-server.cc
M tests/custom_cluster/test_restart_services.py
2 files changed, 150 insertions(+), 13 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/17645/25
-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 25
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "liuyao (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Aman Sinha, Thomas Tauber-Marshall, Vihang Karajgaonkar, Zoltan Borok-Nagy, Tim Armstrong, Wenzhe Zhou, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/17645

to look at the new patch set (#8).

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................

IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

The change of catalogServiceId did not trigger a full topic update
request.

During the execution of DDL or DML, restarting Catalogd will cause
catalogServiceId to change. The DDL will execute successfully, and
then directly update the local cache (not through Statestored), modify
the local catalog with the new catalogServiceId, trigger an exception,
print the exception information, but did not submit full topic update
request. The version of Catalogd is lower than Impalad, so the metadata
information synchronized from Catalogd will be lost.

Under normal circumstances, if the catalogServiceId changes, Impalad
will submit full topic update request, Impalad will receive
TUpdateCatalogCacheRequest, and is_delta is false, that is full update.
Impalad will reset the local catalog cache, and then update, and the
version of CatalogD is greater than Impalad.

Testing

add TestRestart#test_restart_catalogd in
ests/custom_cluster/test_restart_services.py

Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
---
M be/src/service/impala-server.cc
M tests/custom_cluster/test_restart_services.py
2 files changed, 109 insertions(+), 5 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/17645/8
-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 8
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................


Patch Set 7:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7310/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 7
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Sat, 17 Jul 2021 02:37:58 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................


Patch Set 1:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/9031/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 1
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sat, 03 Jul 2021 11:27:44 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17645 )

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................


Patch Set 13:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/9163/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 13
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Mon, 26 Jul 2021 06:22:01 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

Posted by "liuyao (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Aman Sinha, Thomas Tauber-Marshall, Vihang Karajgaonkar, Zoltan Borok-Nagy, Tim Armstrong, Wenzhe Zhou, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/17645

to look at the new patch set (#6).

Change subject: IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync
......................................................................

IMPALA-5476: Fix Catalogd restart bring about metadata is out of sync

The change of catalogServiceId did not trigger a full topic update
request.

During the execution of DDL or DML, restarting Catalogd will cause
catalogServiceId to change. The DDL will execute successfully, and
then directly update the local cache (not through Statestored), modify
the local catalog with the new catalogServiceId, trigger an exception,
print the exception information, but did not submit full topic update
request. The version of Catalogd is lower than Impalad, so the metadata
information synchronized from Catalogd will be lost.

Under normal circumstances, if the catalogServiceId changes, Impalad
will submit full topic update request, Impalad will receive
TUpdateCatalogCacheRequest, and is_delta is false, that is full update.
Impalad will reset the local catalog cache, and then update, and the
version of CatalogD is greater than Impalad.

Testing

add TestRestart#test_restart_catalogd in
ests/custom_cluster/test_restart_services.py

Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
---
M be/src/service/impala-server.cc
M tests/custom_cluster/test_restart_services.py
2 files changed, 44 insertions(+), 9 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/17645/6
-- 
To view, visit http://gerrit.cloudera.org:8080/17645
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9fe25f5a2a42fb432e306ef08ae35750c8f3c50c
Gerrit-Change-Number: 17645
Gerrit-PatchSet: 6
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vi...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>