You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org> on 2024/06/05 16:56:45 UTC

[Impala-ASF-CR] IMPALA-13134: DDL hang with SYNC DDL enabled when Catalogd is changed to standby status

Wenzhe Zhou has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21480


Change subject: IMPALA-13134: DDL hang with SYNC_DDL enabled when Catalogd is changed to standby status
......................................................................

IMPALA-13134: DDL hang with SYNC_DDL enabled when Catalogd is changed to standby status

Catalogd waits for SYNC_DDL version when it processes a DDL with
SYNC_DDL enabled. If the status of Catalogd is changed from active to
standby when CatalogServiceCatalog.waitForSyncDdlVersion() is called,
the standby catalogd does not receive catalog topic updates from
statestore, hence catalogd thread waits indefinitely.

This patch fixed the issue by re-generating service id when Catalogd
is changed to standby status and throwing exception if its service id
has been changed when waiting for SYNC_DDL version.

Testing:
 - Added unit-test code for CatalogD HA to run DDL with SYNC_DDL enabled
   and injected delay when waiting SYNC_DLL version, then verify that
   DDL query fails due to catalog failover.
 - Passed test_catalogd_ha.py.

Change-Id: I2dcd628cff3c10d2e7566ba2d9de0b5886a18fc1
---
M be/src/catalog/catalog-server.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/util/DebugUtils.java
M tests/custom_cluster/test_catalogd_ha.py
7 files changed, 73 insertions(+), 0 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/80/21480/2
-- 
To view, visit http://gerrit.cloudera.org:8080/21480
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I2dcd628cff3c10d2e7566ba2d9de0b5886a18fc1
Gerrit-Change-Number: 21480
Gerrit-PatchSet: 2
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>

[Impala-ASF-CR] IMPALA-13134: DDL hang with SYNC DDL enabled when Catalogd is changed to standby status

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21480 )

Change subject: IMPALA-13134: DDL hang with SYNC_DDL enabled when Catalogd is changed to standby status
......................................................................


Patch Set 2: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/21480
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2dcd628cff3c10d2e7566ba2d9de0b5886a18fc1
Gerrit-Change-Number: 21480
Gerrit-PatchSet: 2
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Thu, 06 Jun 2024 01:39:35 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13134: DDL hang with SYNC DDL enabled when Catalogd is changed to standby status

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/21480 )

Change subject: IMPALA-13134: DDL hang with SYNC_DDL enabled when Catalogd is changed to standby status
......................................................................

IMPALA-13134: DDL hang with SYNC_DDL enabled when Catalogd is changed to standby status

Catalogd waits for SYNC_DDL version when it processes a DDL with
SYNC_DDL enabled. If the status of Catalogd is changed from active to
standby when CatalogServiceCatalog.waitForSyncDdlVersion() is called,
the standby catalogd does not receive catalog topic updates from
statestore, hence catalogd thread waits indefinitely.

This patch fixed the issue by re-generating service id when Catalogd
is changed to standby status and throwing exception if its service id
has been changed when waiting for SYNC_DDL version.

Testing:
 - Added unit-test code for CatalogD HA to run DDL with SYNC_DDL enabled
   and injected delay when waiting SYNC_DLL version, then verify that
   DDL query fails due to catalog failover.
 - Passed test_catalogd_ha.py.

Change-Id: I2dcd628cff3c10d2e7566ba2d9de0b5886a18fc1
Reviewed-on: http://gerrit.cloudera.org:8080/21480
Reviewed-by: Riza Suminto <ri...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/catalog/catalog-server.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/util/DebugUtils.java
M tests/custom_cluster/test_catalogd_ha.py
7 files changed, 73 insertions(+), 0 deletions(-)

Approvals:
  Riza Suminto: Looks good to me, approved
  Impala Public Jenkins: Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/21480
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I2dcd628cff3c10d2e7566ba2d9de0b5886a18fc1
Gerrit-Change-Number: 21480
Gerrit-PatchSet: 3
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>

[Impala-ASF-CR] IMPALA-13134: DDL hang with SYNC DDL enabled when Catalogd is changed to standby status

Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21480 )

Change subject: IMPALA-13134: DDL hang with SYNC_DDL enabled when Catalogd is changed to standby status
......................................................................


Patch Set 2:

(2 comments)

Thanks Riza for your feedback.

http://gerrit.cloudera.org:8080/#/c/21480/2/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:

http://gerrit.cloudera.org:8080/#/c/21480/2/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@3515
PS2, Line 3515: BackendConfig.INSTANCE.debugActions()
> Maybe use debug action from ddlRequest.getQuery_options().getDebug_action()
ddlRequest is not available in this function so we have to add additional parameter to pass query_option. But waitForSyncDdlVersion() is also called from CatalogOpExecutor.execResetMetadataImpl() where query_option is not available in TResetMetadataRequest. TResetMetadataRequest has a field 'debug_action', but it seems not been set properly.


http://gerrit.cloudera.org:8080/#/c/21480/2/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@3519
PS2, Line 3519: if (!serviceId.equals(JniCatalog.getServiceId())) {
> Should this check done closer to L3544 (just before topicUpdateLog_.wait(TO
Prefer to make it failed earlier.



-- 
To view, visit http://gerrit.cloudera.org:8080/21480
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2dcd628cff3c10d2e7566ba2d9de0b5886a18fc1
Gerrit-Change-Number: 21480
Gerrit-PatchSet: 2
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Wed, 05 Jun 2024 18:58:11 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-13134: DDL hang with SYNC DDL enabled when Catalogd is changed to standby status

Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/21480 )

Change subject: IMPALA-13134: DDL hang with SYNC_DDL enabled when Catalogd is changed to standby status
......................................................................


Patch Set 2:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/21480/2/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:

http://gerrit.cloudera.org:8080/#/c/21480/2/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@3519
PS2, Line 3519: if (!serviceId.equals(JniCatalog.getServiceId())) {
> Ack.
Yes, it will wait one more attempt.



-- 
To view, visit http://gerrit.cloudera.org:8080/21480
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2dcd628cff3c10d2e7566ba2d9de0b5886a18fc1
Gerrit-Change-Number: 21480
Gerrit-PatchSet: 2
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Wed, 05 Jun 2024 19:52:15 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-13134: DDL hang with SYNC DDL enabled when Catalogd is changed to standby status

Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/21480 )

Change subject: IMPALA-13134: DDL hang with SYNC_DDL enabled when Catalogd is changed to standby status
......................................................................


Patch Set 2: Code-Review+2

(1 comment)

http://gerrit.cloudera.org:8080/#/c/21480/2/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:

http://gerrit.cloudera.org:8080/#/c/21480/2/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@3519
PS2, Line 3519: if (!serviceId.equals(JniCatalog.getServiceId())) {
> Yes, it will wait one more attempt.
Ok, thanks!



-- 
To view, visit http://gerrit.cloudera.org:8080/21480
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2dcd628cff3c10d2e7566ba2d9de0b5886a18fc1
Gerrit-Change-Number: 21480
Gerrit-PatchSet: 2
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Wed, 05 Jun 2024 19:55:18 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-13134: DDL hang with SYNC DDL enabled when Catalogd is changed to standby status

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21480 )

Change subject: IMPALA-13134: DDL hang with SYNC_DDL enabled when Catalogd is changed to standby status
......................................................................


Patch Set 2:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10690/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/21480
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2dcd628cff3c10d2e7566ba2d9de0b5886a18fc1
Gerrit-Change-Number: 21480
Gerrit-PatchSet: 2
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Wed, 05 Jun 2024 20:29:24 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-13134: DDL hang with SYNC DDL enabled when Catalogd is changed to standby status

Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/21480 )

Change subject: IMPALA-13134: DDL hang with SYNC_DDL enabled when Catalogd is changed to standby status
......................................................................


Patch Set 2: Code-Review+1

(2 comments)

http://gerrit.cloudera.org:8080/#/c/21480/2/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:

http://gerrit.cloudera.org:8080/#/c/21480/2/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@3515
PS2, Line 3515: BackendConfig.INSTANCE.debugActions()
> ddlRequest is not available in this function so we have to add additional p
Ack


http://gerrit.cloudera.org:8080/#/c/21480/2/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@3519
PS2, Line 3519: if (!serviceId.equals(JniCatalog.getServiceId())) {
> Prefer to make it failed earlier.
Ack.

Follow up question: if serviceId does change slightly after this check, it will still wait for 1 more attempt for 10s, right?



-- 
To view, visit http://gerrit.cloudera.org:8080/21480
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2dcd628cff3c10d2e7566ba2d9de0b5886a18fc1
Gerrit-Change-Number: 21480
Gerrit-PatchSet: 2
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Wed, 05 Jun 2024 19:44:45 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-13134: DDL hang with SYNC DDL enabled when Catalogd is changed to standby status

Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/21480 )

Change subject: IMPALA-13134: DDL hang with SYNC_DDL enabled when Catalogd is changed to standby status
......................................................................


Patch Set 2:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/21480/2/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:

http://gerrit.cloudera.org:8080/#/c/21480/2/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@3515
PS2, Line 3515: BackendConfig.INSTANCE.debugActions()
Maybe use debug action from ddlRequest.getQuery_options().getDebug_action()?
See CatalogOpExecutor.java L448


http://gerrit.cloudera.org:8080/#/c/21480/2/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@3519
PS2, Line 3519: if (!serviceId.equals(JniCatalog.getServiceId())) {
Should this check done closer to L3544 (just before topicUpdateLog_.wait(TOPIC_UPDATE_WAIT_TIMEOUT_MS)) ?



-- 
To view, visit http://gerrit.cloudera.org:8080/21480
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2dcd628cff3c10d2e7566ba2d9de0b5886a18fc1
Gerrit-Change-Number: 21480
Gerrit-PatchSet: 2
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Wed, 05 Jun 2024 17:39:16 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-13134: DDL hang with SYNC DDL enabled when Catalogd is changed to standby status

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21480 )

Change subject: IMPALA-13134: DDL hang with SYNC_DDL enabled when Catalogd is changed to standby status
......................................................................


Patch Set 2:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/16274/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/21480
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2dcd628cff3c10d2e7566ba2d9de0b5886a18fc1
Gerrit-Change-Number: 21480
Gerrit-PatchSet: 2
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Wed, 05 Jun 2024 17:21:18 +0000
Gerrit-HasComments: No