You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Quanlong Huang (Code Review)" <ge...@cloudera.org> on 2025/07/22 03:53:48 UTC

[Impala-ASF-CR] IMPALA-14227: (Addendum) Add more tests for catalogd HA warm failover

Quanlong Huang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/23206


Change subject: IMPALA-14227: (Addendum) Add more tests for catalogd HA warm failover
......................................................................

IMPALA-14227: (Addendum) Add more tests for catalogd HA warm failover

This adds more tests in test_catalogd_ha.py for warm failover.
Refactored _test_metadata_after_failover to run in the following way:
 - Run DDL/DML in the active catalogd.
 - Kill the active catalogd and wait until the failover finishes.
 - Verify the DDL/DML results in the new active catalogd.
 - Restart the killed catalogd
It accepts two methods in parameters to perform the DDL/DML and the
verifier. In the last step, the killed catalogd is started so we keep
having 2 catalogd and can merge these into a single test by invoking
_test_metadata_after_failover for different method pairs. This saves
some test time.

The following DDL/DML statements are tested:
 - CreateTable
 - AddPartition
 - REFRESH
 - DropPartition
 - INSERT
 - DropTable
After each failover, the table is verified to be warmed up (i.e. loaded).

Also validate flags in startup to make sure enable_insert_events and
enable_reload_events are both set to true when warm failover is enabled,
i.e. --catalogd_ha_reset_metadata_on_failover=false.

Change-Id: I6b20adeb0bd175592b425e521138c41196347600
---
M be/src/common/init.cc
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M tests/custom_cluster/test_catalogd_ha.py
3 files changed, 181 insertions(+), 48 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/06/23206/1
-- 
To view, visit http://gerrit.cloudera.org:8080/23206
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I6b20adeb0bd175592b425e521138c41196347600
Gerrit-Change-Number: 23206
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>

[Impala-ASF-CR] IMPALA-14227: (Addendum) Add more tests for catalogd HA warm failover

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23206 )

Change subject: IMPALA-14227: (Addendum) Add more tests for catalogd HA warm failover
......................................................................


Patch Set 1:

(1 comment)

gerrit-auto-critic failed. You can reproduce it locally using command:

  python3 bin/jenkins/critique-gerrit-review.py --dryrun

To run it, you might need a virtual env with Python3's venv installed.

http://gerrit.cloudera.org:8080/#/c/23206/1/tests/custom_cluster/test_catalogd_ha.py
File tests/custom_cluster/test_catalogd_ha.py:

http://gerrit.cloudera.org:8080/#/c/23206/1/tests/custom_cluster/test_catalogd_ha.py@728
PS1, Line 728: u
flake8: U100 Unused argument 'unique_database'



-- 
To view, visit http://gerrit.cloudera.org:8080/23206
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6b20adeb0bd175592b425e521138c41196347600
Gerrit-Change-Number: 23206
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 22 Jul 2025 03:55:47 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-14227: (Addendum) Add more tests for catalogd HA warm failover

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23206 )

Change subject: IMPALA-14227: (Addendum) Add more tests for catalogd HA warm failover
......................................................................


Patch Set 5:

Build Failed 

https://jenkins.impala.io/job/gerrit-code-review-checks/19984/ : Initial code review checks failed. See linked job for details on the failure.


-- 
To view, visit http://gerrit.cloudera.org:8080/23206
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6b20adeb0bd175592b425e521138c41196347600
Gerrit-Change-Number: 23206
Gerrit-PatchSet: 5
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Tue, 29 Jul 2025 15:30:36 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14227: (Addendum) Add more tests for catalogd HA warm failover

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/23206 )

Change subject: IMPALA-14227: (Addendum) Add more tests for catalogd HA warm failover
......................................................................


Patch Set 5:

> Patch Set 4: Verified-1
> 
> Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/12246/

Some tests failed due to the flag validation on --enable_reload_events


-- 
To view, visit http://gerrit.cloudera.org:8080/23206
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6b20adeb0bd175592b425e521138c41196347600
Gerrit-Change-Number: 23206
Gerrit-PatchSet: 5
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Tue, 29 Jul 2025 15:05:16 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14227: (Addendum) Add more tests for catalogd HA warm failover

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Hello Riza Suminto, Csaba Ringhofer, Wenzhe Zhou, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/23206

to look at the new patch set (#3).

Change subject: IMPALA-14227: (Addendum) Add more tests for catalogd HA warm failover
......................................................................

IMPALA-14227: (Addendum) Add more tests for catalogd HA warm failover

This adds more tests in test_catalogd_ha.py for warm failover.
Refactored _test_metadata_after_failover to run in the following way:
 - Run DDL/DML in the active catalogd.
 - Kill the active catalogd and wait until the failover finishes.
 - Verify the DDL/DML results in the new active catalogd.
 - Restart the killed catalogd
It accepts two methods in parameters to perform the DDL/DML and the
verifier. In the last step, the killed catalogd is started so we keep
having 2 catalogd and can merge these into a single test by invoking
_test_metadata_after_failover for different method pairs. This saves
some test time.

The following DDL/DML statements are tested:
 - CreateTable
 - AddPartition
 - REFRESH
 - DropPartition
 - INSERT
 - DropTable
After each failover, the table is verified to be warmed up (i.e. loaded).

Also validate flags in startup to make sure enable_insert_events and
enable_reload_events are both set to true when warm failover is enabled,
i.e. --catalogd_ha_reset_metadata_on_failover=false.

Change-Id: I6b20adeb0bd175592b425e521138c41196347600
---
M be/src/common/init.cc
M bin/rat_exclude_files.txt
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
A testdata/data/warmup_test_config.txt
M tests/custom_cluster/test_catalogd_ha.py
5 files changed, 183 insertions(+), 48 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/06/23206/3
-- 
To view, visit http://gerrit.cloudera.org:8080/23206
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6b20adeb0bd175592b425e521138c41196347600
Gerrit-Change-Number: 23206
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>

[Impala-ASF-CR] IMPALA-14227: (Addendum) Add more tests for catalogd HA warm failover

Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/23206 )

Change subject: IMPALA-14227: (Addendum) Add more tests for catalogd HA warm failover
......................................................................


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/23206/1/tests/custom_cluster/test_catalogd_ha.py
File tests/custom_cluster/test_catalogd_ha.py:

http://gerrit.cloudera.org:8080/#/c/23206/1/tests/custom_cluster/test_catalogd_ha.py@624
PS1, Line 624: file://%s/testdata/data/warmup_test_config.txt
change file:// to FILESYSTEM_PREFIX ? Does not see warmup_test_config.txt under testdata/data. Is it a new data file?



-- 
To view, visit http://gerrit.cloudera.org:8080/23206
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6b20adeb0bd175592b425e521138c41196347600
Gerrit-Change-Number: 23206
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Wed, 23 Jul 2025 04:43:58 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-14227: (Addendum) Add more tests for catalogd HA warm failover

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/23206 )

Change subject: IMPALA-14227: (Addendum) Add more tests for catalogd HA warm failover
......................................................................


Patch Set 4: Code-Review+2

Carry Wenzhe's +2.


-- 
To view, visit http://gerrit.cloudera.org:8080/23206
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6b20adeb0bd175592b425e521138c41196347600
Gerrit-Change-Number: 23206
Gerrit-PatchSet: 4
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Tue, 29 Jul 2025 07:47:52 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14227: (Addendum) Add more tests for catalogd HA warm failover

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23206 )

Change subject: IMPALA-14227: (Addendum) Add more tests for catalogd HA warm failover
......................................................................


Patch Set 5: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/23206
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6b20adeb0bd175592b425e521138c41196347600
Gerrit-Change-Number: 23206
Gerrit-PatchSet: 5
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Tue, 29 Jul 2025 21:12:05 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14227: (Addendum) Add more tests for catalogd HA warm failover

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23206 )

Change subject: IMPALA-14227: (Addendum) Add more tests for catalogd HA warm failover
......................................................................


Patch Set 4:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/12246/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/23206
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6b20adeb0bd175592b425e521138c41196347600
Gerrit-Change-Number: 23206
Gerrit-PatchSet: 4
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Tue, 29 Jul 2025 07:49:09 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14227: (Addendum) Add more tests for catalogd HA warm failover

Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/23206 )

Change subject: IMPALA-14227: (Addendum) Add more tests for catalogd HA warm failover
......................................................................


Patch Set 3: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/23206
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6b20adeb0bd175592b425e521138c41196347600
Gerrit-Change-Number: 23206
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Wed, 23 Jul 2025 16:06:34 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14227: (Addendum) Add more tests for catalogd HA warm failover

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/23206 )

Change subject: IMPALA-14227: (Addendum) Add more tests for catalogd HA warm failover
......................................................................


Patch Set 2:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/23206/1/tests/custom_cluster/test_catalogd_ha.py
File tests/custom_cluster/test_catalogd_ha.py:

http://gerrit.cloudera.org:8080/#/c/23206/1/tests/custom_cluster/test_catalogd_ha.py@624
PS1, Line 624: file://%s/testdata/data/warmup_test_config.txt
> change file:// to FILESYSTEM_PREFIX ? Does not see warmup_test_config.txt u
Oops, forgot to add this file.



-- 
To view, visit http://gerrit.cloudera.org:8080/23206
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6b20adeb0bd175592b425e521138c41196347600
Gerrit-Change-Number: 23206
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Wed, 23 Jul 2025 07:32:11 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-14227: (Addendum) Add more tests for catalogd HA warm failover

Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/23206 )

Change subject: IMPALA-14227: (Addendum) Add more tests for catalogd HA warm failover
......................................................................


Patch Set 5:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/23206/5/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:

http://gerrit.cloudera.org:8080/#/c/23206/5/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@3728
PS5, Line 3728: getNumCatalogResetStarts()
> Did not find where this function is called.
never mind, found it



-- 
To view, visit http://gerrit.cloudera.org:8080/23206
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6b20adeb0bd175592b425e521138c41196347600
Gerrit-Change-Number: 23206
Gerrit-PatchSet: 5
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Wed, 30 Jul 2025 00:29:04 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-14227: (Addendum) Add more tests for catalogd HA warm failover

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Hello Riza Suminto, Csaba Ringhofer, Wenzhe Zhou, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/23206

to look at the new patch set (#5).

Change subject: IMPALA-14227: (Addendum) Add more tests for catalogd HA warm failover
......................................................................

IMPALA-14227: (Addendum) Add more tests for catalogd HA warm failover

This adds more tests in test_catalogd_ha.py for warm failover.
Refactored _test_metadata_after_failover to run in the following way:
 - Run DDL/DML in the active catalogd.
 - Kill the active catalogd and wait until the failover finishes.
 - Verify the DDL/DML results in the new active catalogd.
 - Restart the killed catalogd
It accepts two methods in parameters to perform the DDL/DML and the
verifier. In the last step, the killed catalogd is started so we keep
having 2 catalogd and can merge these into a single test by invoking
_test_metadata_after_failover for different method pairs. This saves
some test time.

The following DDL/DML statements are tested:
 - CreateTable
 - AddPartition
 - REFRESH
 - DropPartition
 - INSERT
 - DropTable
After each failover, the table is verified to be warmed up (i.e. loaded).

Also validate flags in startup to make sure enable_insert_events and
enable_reload_events are both set to true when warm failover is enabled,
i.e. --catalogd_ha_reset_metadata_on_failover=false.

Change-Id: I6b20adeb0bd175592b425e521138c41196347600
---
M be/src/common/init.cc
M bin/rat_exclude_files.txt
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
A testdata/data/warmup_test_config.txt
M tests/custom_cluster/test_catalogd_ha.py
M tests/custom_cluster/test_ext_data_sources.py
6 files changed, 195 insertions(+), 56 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/06/23206/5
-- 
To view, visit http://gerrit.cloudera.org:8080/23206
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6b20adeb0bd175592b425e521138c41196347600
Gerrit-Change-Number: 23206
Gerrit-PatchSet: 5
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>

[Impala-ASF-CR] IMPALA-14227: (Addendum) Add more tests for catalogd HA warm failover

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23206 )

Change subject: IMPALA-14227: (Addendum) Add more tests for catalogd HA warm failover
......................................................................


Patch Set 1:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/19937/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/23206
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6b20adeb0bd175592b425e521138c41196347600
Gerrit-Change-Number: 23206
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Tue, 22 Jul 2025 04:20:42 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14227: (Addendum) Add more tests for catalogd HA warm failover

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23206 )

Change subject: IMPALA-14227: (Addendum) Add more tests for catalogd HA warm failover
......................................................................


Patch Set 2:

Build Failed 

https://jenkins.impala.io/job/gerrit-code-review-checks/19948/ : Initial code review checks failed. See linked job for details on the failure.


-- 
To view, visit http://gerrit.cloudera.org:8080/23206
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6b20adeb0bd175592b425e521138c41196347600
Gerrit-Change-Number: 23206
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Wed, 23 Jul 2025 07:58:44 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14227: (Addendum) Add more tests for catalogd HA warm failover

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Hello Riza Suminto, Csaba Ringhofer, Wenzhe Zhou, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/23206

to look at the new patch set (#2).

Change subject: IMPALA-14227: (Addendum) Add more tests for catalogd HA warm failover
......................................................................

IMPALA-14227: (Addendum) Add more tests for catalogd HA warm failover

This adds more tests in test_catalogd_ha.py for warm failover.
Refactored _test_metadata_after_failover to run in the following way:
 - Run DDL/DML in the active catalogd.
 - Kill the active catalogd and wait until the failover finishes.
 - Verify the DDL/DML results in the new active catalogd.
 - Restart the killed catalogd
It accepts two methods in parameters to perform the DDL/DML and the
verifier. In the last step, the killed catalogd is started so we keep
having 2 catalogd and can merge these into a single test by invoking
_test_metadata_after_failover for different method pairs. This saves
some test time.

The following DDL/DML statements are tested:
 - CreateTable
 - AddPartition
 - REFRESH
 - DropPartition
 - INSERT
 - DropTable
After each failover, the table is verified to be warmed up (i.e. loaded).

Also validate flags in startup to make sure enable_insert_events and
enable_reload_events are both set to true when warm failover is enabled,
i.e. --catalogd_ha_reset_metadata_on_failover=false.

Change-Id: I6b20adeb0bd175592b425e521138c41196347600
---
M be/src/common/init.cc
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
A testdata/data/warmup_test_config.txt
M tests/custom_cluster/test_catalogd_ha.py
4 files changed, 182 insertions(+), 48 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/06/23206/2
-- 
To view, visit http://gerrit.cloudera.org:8080/23206
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6b20adeb0bd175592b425e521138c41196347600
Gerrit-Change-Number: 23206
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>

[Impala-ASF-CR] IMPALA-14227: (Addendum) Add more tests for catalogd HA warm failover

Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/23206 )

Change subject: IMPALA-14227: (Addendum) Add more tests for catalogd HA warm failover
......................................................................


Patch Set 5:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/23206/5/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:

http://gerrit.cloudera.org:8080/#/c/23206/5/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@3728
PS5, Line 3728: getNumCatalogResetStarts()
Did not find where this function is called.



-- 
To view, visit http://gerrit.cloudera.org:8080/23206
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6b20adeb0bd175592b425e521138c41196347600
Gerrit-Change-Number: 23206
Gerrit-PatchSet: 5
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Wed, 30 Jul 2025 00:21:07 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-14227: (Addendum) Add more tests for catalogd HA warm failover

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23206 )

Change subject: IMPALA-14227: (Addendum) Add more tests for catalogd HA warm failover
......................................................................


Patch Set 5:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/12247/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/23206
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6b20adeb0bd175592b425e521138c41196347600
Gerrit-Change-Number: 23206
Gerrit-PatchSet: 5
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Tue, 29 Jul 2025 15:05:30 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14227: (Addendum) Add more tests for catalogd HA warm failover

Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/23206 )

Change subject: IMPALA-14227: (Addendum) Add more tests for catalogd HA warm failover
......................................................................


Patch Set 5: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/23206
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6b20adeb0bd175592b425e521138c41196347600
Gerrit-Change-Number: 23206
Gerrit-PatchSet: 5
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Wed, 30 Jul 2025 00:29:05 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14227: (Addendum) Add more tests for catalogd HA warm failover

Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/23206 )

Change subject: IMPALA-14227: (Addendum) Add more tests for catalogd HA warm failover
......................................................................

IMPALA-14227: (Addendum) Add more tests for catalogd HA warm failover

This adds more tests in test_catalogd_ha.py for warm failover.
Refactored _test_metadata_after_failover to run in the following way:
 - Run DDL/DML in the active catalogd.
 - Kill the active catalogd and wait until the failover finishes.
 - Verify the DDL/DML results in the new active catalogd.
 - Restart the killed catalogd
It accepts two methods in parameters to perform the DDL/DML and the
verifier. In the last step, the killed catalogd is started so we keep
having 2 catalogd and can merge these into a single test by invoking
_test_metadata_after_failover for different method pairs. This saves
some test time.

The following DDL/DML statements are tested:
 - CreateTable
 - AddPartition
 - REFRESH
 - DropPartition
 - INSERT
 - DropTable
After each failover, the table is verified to be warmed up (i.e. loaded).

Also validate flags in startup to make sure enable_insert_events and
enable_reload_events are both set to true when warm failover is enabled,
i.e. --catalogd_ha_reset_metadata_on_failover=false.

Change-Id: I6b20adeb0bd175592b425e521138c41196347600
Reviewed-on: http://gerrit.cloudera.org:8080/23206
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Wenzhe Zhou <wz...@cloudera.com>
---
M be/src/common/init.cc
M bin/rat_exclude_files.txt
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
A testdata/data/warmup_test_config.txt
M tests/custom_cluster/test_catalogd_ha.py
M tests/custom_cluster/test_ext_data_sources.py
6 files changed, 195 insertions(+), 56 deletions(-)

Approvals:
  Impala Public Jenkins: Verified
  Wenzhe Zhou: Looks good to me, approved

-- 
To view, visit http://gerrit.cloudera.org:8080/23206
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I6b20adeb0bd175592b425e521138c41196347600
Gerrit-Change-Number: 23206
Gerrit-PatchSet: 6
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>

[Impala-ASF-CR] IMPALA-14227: (Addendum) Add more tests for catalogd HA warm failover

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23206 )

Change subject: IMPALA-14227: (Addendum) Add more tests for catalogd HA warm failover
......................................................................


Patch Set 4: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/12246/


-- 
To view, visit http://gerrit.cloudera.org:8080/23206
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6b20adeb0bd175592b425e521138c41196347600
Gerrit-Change-Number: 23206
Gerrit-PatchSet: 4
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Tue, 29 Jul 2025 13:56:33 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14227: (Addendum) Add more tests for catalogd HA warm failover

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23206 )

Change subject: IMPALA-14227: (Addendum) Add more tests for catalogd HA warm failover
......................................................................


Patch Set 3:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/19949/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/23206
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6b20adeb0bd175592b425e521138c41196347600
Gerrit-Change-Number: 23206
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Wed, 23 Jul 2025 10:32:20 +0000
Gerrit-HasComments: No