You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org> on 2023/08/08 16:41:43 UTC

[Impala-ASF-CR] IMPALA-12340: Fix flaky test two catalogd with force active

Wenzhe Zhou has uploaded this change for review. ( http://gerrit.cloudera.org:8080/20323


Change subject: IMPALA-12340: Fix flaky test_two_catalogd_with_force_active
......................................................................

IMPALA-12340: Fix flaky test_two_catalogd_with_force_active

The issue could be re-produced by repeatedly running the test case for
about 50 times in Jenkins or on local machine.
The issue was introduced by IMPALA-12304, for which we tried not to
wake up the thread for update_catalogd RPC if there is no change for
elected active catalogd. Since we cannot hold mutex when calling
SendUpdateCatalogdNotification(), it may cause the notification to the
condition variable not been processed. To fix the issue, revert the
optimization added in IMPALA-12304, and add sleep in
test_two_catalogd_with_force_active.

Testing:
 - Repeatedly ran test_two_catalogd_with_force_active for 8000 times on
   local machine without failure, ran the test for 1000 times in Jenkins
   without failure.
   Repeatedly ran all test cases in test_catalogd_ha.py for 21 hours in
   Jenkins without failure.
 - Passed core test.

Change-Id: Ifc213422a5a3360d07a22046e1f46fdf0be1d2fd
---
M be/src/statestore/statestore.cc
M tests/custom_cluster/test_catalogd_ha.py
2 files changed, 5 insertions(+), 5 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/23/20323/2
-- 
To view, visit http://gerrit.cloudera.org:8080/20323
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ifc213422a5a3360d07a22046e1f46fdf0be1d2fd
Gerrit-Change-Number: 20323
Gerrit-PatchSet: 2
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>

[Impala-ASF-CR] IMPALA-12340: Fix flaky test two catalogd with force active

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20323 )

Change subject: IMPALA-12340: Fix flaky test_two_catalogd_with_force_active
......................................................................

IMPALA-12340: Fix flaky test_two_catalogd_with_force_active

The issue could be re-produced by repeatedly running the test case for
about 50 times in Jenkins or on local machine.
The issue was introduced by IMPALA-12304, for which we tried not to
wake up the thread for update_catalogd RPC if there is no change for
elected active catalogd. Since we cannot hold mutex when calling
SendUpdateCatalogdNotification(), it may cause the notification to the
condition variable not been processed. To fix the issue, revert the
optimization added in IMPALA-12304, and add sleep in
test_two_catalogd_with_force_active.

Testing:
 - Repeatedly ran test_two_catalogd_with_force_active for 8000 times on
   local machine without failure, ran the test for 1000 times in Jenkins
   without failure.
   Repeatedly ran all test cases in test_catalogd_ha.py for 21 hours in
   Jenkins without failure.
 - Passed core test.

Change-Id: Ifc213422a5a3360d07a22046e1f46fdf0be1d2fd
Reviewed-on: http://gerrit.cloudera.org:8080/20323
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/statestore/statestore.cc
M tests/custom_cluster/test_catalogd_ha.py
2 files changed, 5 insertions(+), 5 deletions(-)

Approvals:
  Impala Public Jenkins: Looks good to me, approved; Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/20323
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ifc213422a5a3360d07a22046e1f46fdf0be1d2fd
Gerrit-Change-Number: 20323
Gerrit-PatchSet: 4
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>

[Impala-ASF-CR] IMPALA-12340: Fix flaky test two catalogd with force active

Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/20323 )

Change subject: IMPALA-12340: Fix flaky test_two_catalogd_with_force_active
......................................................................


Patch Set 2: Code-Review+1

Looks OK to me.


-- 
To view, visit http://gerrit.cloudera.org:8080/20323
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifc213422a5a3360d07a22046e1f46fdf0be1d2fd
Gerrit-Change-Number: 20323
Gerrit-PatchSet: 2
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Tue, 08 Aug 2023 17:05:35 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12340: Fix flaky test two catalogd with force active

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20323 )

Change subject: IMPALA-12340: Fix flaky test_two_catalogd_with_force_active
......................................................................


Patch Set 2:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/13723/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/20323
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifc213422a5a3360d07a22046e1f46fdf0be1d2fd
Gerrit-Change-Number: 20323
Gerrit-PatchSet: 2
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Tue, 08 Aug 2023 17:05:35 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12340: Fix flaky test two catalogd with force active

Posted by "Andrew Sherman (Code Review)" <ge...@cloudera.org>.
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/20323 )

Change subject: IMPALA-12340: Fix flaky test_two_catalogd_with_force_active
......................................................................


Patch Set 2: Code-Review+2

LGTM


-- 
To view, visit http://gerrit.cloudera.org:8080/20323
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifc213422a5a3360d07a22046e1f46fdf0be1d2fd
Gerrit-Change-Number: 20323
Gerrit-PatchSet: 2
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Tue, 08 Aug 2023 18:24:56 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12340: Fix flaky test two catalogd with force active

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20323 )

Change subject: IMPALA-12340: Fix flaky test_two_catalogd_with_force_active
......................................................................


Patch Set 3: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/20323
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifc213422a5a3360d07a22046e1f46fdf0be1d2fd
Gerrit-Change-Number: 20323
Gerrit-PatchSet: 3
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Tue, 08 Aug 2023 22:36:09 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12340: Fix flaky test two catalogd with force active

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20323 )

Change subject: IMPALA-12340: Fix flaky test_two_catalogd_with_force_active
......................................................................


Patch Set 3:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9575/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/20323
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifc213422a5a3360d07a22046e1f46fdf0be1d2fd
Gerrit-Change-Number: 20323
Gerrit-PatchSet: 3
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Tue, 08 Aug 2023 18:26:28 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12340: Fix flaky test two catalogd with force active

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20323 )

Change subject: IMPALA-12340: Fix flaky test_two_catalogd_with_force_active
......................................................................


Patch Set 3: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/20323
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifc213422a5a3360d07a22046e1f46fdf0be1d2fd
Gerrit-Change-Number: 20323
Gerrit-PatchSet: 3
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Tue, 08 Aug 2023 18:26:27 +0000
Gerrit-HasComments: No