You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Jason Fehr (Code Review)" <ge...@cloudera.org> on 2026/05/05 17:12:34 UTC
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Jason Fehr has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24275
Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock
......................................................................
IMPALA-14949: Fix Catalogd Startup Deadlock
Reset metadata handling code no longer yields up the write lock on
versionLock_ in CatalogServiceCatalog during the Catalogd startup
initial reset metadata.
When Catalogd runs a global reset, the rebuildDbCache() function
will pause and unlock versionLock_ at intervals defined by the
--reset_metadata_lock_duration_ms startup flag. The purpose of this
unlocking is to ensure a long running invalidate metadata does not
block other metadata operations.
During startup, Catalogd runs a global metadata reset. Other threads
must wait until this initial reset finishes before they can process
operations such as getPartialCatalogObject. If the initial reset
takes longer than the duration in the
--reset_metadata_lock_duration_ms startup flag, it will release its
ownership of versionLock_ followed by immediately re-obtaining the
write lock. In between, other threads take a read lock on
versionLock_ but sleep until the initial reset completes. At that
point, a deadlock situation occurs where the initial reset thread is
waiting on a versionLock_ write lock while other threads that hold
the versionLock_ read lock are waiting on the initial reset thread
to complete.
Manual testing replicated this issue and demonstrated the fix
effectiveness. The replication required adding an arbitrary sleep in
the CatalogServiceCatalog.rebuildDbCache function's while loop.
Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
1 file changed, 2 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/75/24275/1
--
To view, visit http://gerrit.cloudera.org:8080/24275
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404
Gerrit-Change-Number: 24275
Gerrit-PatchSet: 1
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 )
Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock
......................................................................
Patch Set 4: Code-Review+1
(2 comments)
http://gerrit.cloudera.org:8080/#/c/24275/3/tests/custom_cluster/test_catalog_wait.py
File tests/custom_cluster/test_catalog_wait.py:
http://gerrit.cloudera.org:8080/#/c/24275/3/tests/custom_cluster/test_catalog_wait.py@179
PS3, Line 179: def count_completed_queries():
> Replaced with waiting for 100 queries to complete.
Done
http://gerrit.cloudera.org:8080/#/c/24275/3/tests/custom_cluster/test_catalog_wait.py@185
PS3, Line 185: func=count_completed_queries, max_attempts=10, sleep_time_s=2, backoff=1), \
> There needs to be time for queries that need Catalogd interaction to queue
Done
--
To view, visit http://gerrit.cloudera.org:8080/24275
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404
Gerrit-Change-Number: 24275
Gerrit-PatchSet: 4
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Fri, 08 May 2026 18:52:53 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 )
Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock
......................................................................
Patch Set 8:
> Patch Set 8:
>
> Build Failed
>
> https://jenkins.impala.io/job/gerrit-code-review-checks/22380/ : Initial code review checks failed. See linked job for details on the failure.
Build failed because it could not download a jar from the native-toolchain S3 bucket.
--
To view, visit http://gerrit.cloudera.org:8080/24275
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404
Gerrit-Change-Number: 24275
Gerrit-PatchSet: 8
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Mon, 11 May 2026 20:43:50 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-14949: Fix Catalogd Startup Deadlock
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 )
Change subject: [WIP] IMPALA-14949: Fix Catalogd Startup Deadlock
......................................................................
Patch Set 6:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/22368/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24275
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404
Gerrit-Change-Number: 24275
Gerrit-PatchSet: 6
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Fri, 08 May 2026 22:23:50 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Posted by "Yida Wu (Code Review)" <ge...@cloudera.org>.
Yida Wu has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 )
Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock
......................................................................
Patch Set 8: Code-Review+1
(2 comments)
http://gerrit.cloudera.org:8080/#/c/24275/8/tests/custom_cluster/test_catalog_wait.py
File tests/custom_cluster/test_catalog_wait.py:
http://gerrit.cloudera.org:8080/#/c/24275/8/tests/custom_cluster/test_catalog_wait.py@170
PS8, Line 170: repetedly
nit. repeatedly
http://gerrit.cloudera.org:8080/#/c/24275/8/tests/custom_cluster/test_catalog_wait.py@193
PS8, Line 193: CREEATED
nit. CREATED
--
To view, visit http://gerrit.cloudera.org:8080/24275
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404
Gerrit-Change-Number: 24275
Gerrit-PatchSet: 8
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Tue, 12 May 2026 17:46:47 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 )
Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock
......................................................................
Patch Set 2: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/24275
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404
Gerrit-Change-Number: 24275
Gerrit-PatchSet: 2
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Wed, 06 May 2026 23:14:41 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/24275 )
Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock
......................................................................
IMPALA-14949: Fix Catalogd Startup Deadlock
The getOrLoadTable functionality now waits for the initial database
metadata to be loaded for the given database before it takes the
versionLock_ read lock. This order of operations is reversed from its
previous ordering.
There was a deadlock condition during a slow Catalogd startup because
the thread that performs the initial global reset periodically
releases the versionLock_ write lock to allow for metadata operations
on already loaded databases to proceed. If the database was not yet
loaded, the read thread waited until it was loaded holding a read
lock on versionLock_. However, the initial global reset thread could
not proceed with loading any databases because it needed a write lock
on versionLock_.
Manual testing replicated this issue and demonstrated the fix
effectiveness. The replication required adding an arbitrary sleep in
the CatalogServiceCatalog.rebuildDbCache function's while loop.
Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M tests/custom_cluster/test_catalog_wait.py
2 files changed, 111 insertions(+), 11 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/75/24275/3
--
To view, visit http://gerrit.cloudera.org:8080/24275
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404
Gerrit-Change-Number: 24275
Gerrit-PatchSet: 3
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
[Impala-ASF-CR] [WIP] IMPALA-14949: Fix Catalogd Startup Deadlock
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/24275 )
Change subject: [WIP] IMPALA-14949: Fix Catalogd Startup Deadlock
......................................................................
[WIP] IMPALA-14949: Fix Catalogd Startup Deadlock
The getOrLoadTable functionality now waits for the initial database
metadata to be loaded for the given database before it takes the
versionLock_ read lock. This order of operations is reversed from its
previous ordering.
There was a deadlock condition during a slow Catalogd startup because
the thread that performs the initial global reset periodically
releases the versionLock_ write lock to allow for metadata operations
on already loaded databases to proceed. If the database was not yet
loaded, the read thread waited until it was loaded holding a read
lock on versionLock_. However, the initial global reset thread could
not proceed with loading any databases because it needed a write lock
on versionLock_.
Testing accomplished with a new custom cluster test. This new test
was run on a local Impala dev machine without the fix in place to
demonstrate it consistently failed because of the deadlock. It was
also successfully run with the fix in place both on a local Impala
dev machine and in an automated build.
Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M tests/custom_cluster/test_catalog_wait.py
2 files changed, 137 insertions(+), 11 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/75/24275/6
--
To view, visit http://gerrit.cloudera.org:8080/24275
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404
Gerrit-Change-Number: 24275
Gerrit-PatchSet: 6
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 )
Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock
......................................................................
Patch Set 8:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/13457/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/24275
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404
Gerrit-Change-Number: 24275
Gerrit-PatchSet: 8
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Tue, 12 May 2026 17:01:48 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 )
Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock
......................................................................
Patch Set 9:
> Patch Set 9: Verified-1
>
> Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/13458/
Build failed with known error downloading artifacts.
--
To view, visit http://gerrit.cloudera.org:8080/24275
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404
Gerrit-Change-Number: 24275
Gerrit-PatchSet: 9
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Tue, 12 May 2026 23:57:49 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 )
Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock
......................................................................
Patch Set 2:
(1 comment)
The fix looks good to me. Do you plan to add the test?
http://gerrit.cloudera.org:8080/#/c/24275/2/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:
http://gerrit.cloudera.org:8080/#/c/24275/2/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2609
PS2, Line 2609: or if the initial reset is in progress.
nit: this is stale now
--
To view, visit http://gerrit.cloudera.org:8080/24275
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404
Gerrit-Change-Number: 24275
Gerrit-PatchSet: 2
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Thu, 07 May 2026 14:49:24 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/24275 )
Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock
......................................................................
IMPALA-14949: Fix Catalogd Startup Deadlock
The getOrLoadTable functionality now waits for the initial database
metadata to be loaded for the given database before it takes the
versionLock_ read lock. This order of operations is reversed from its
previous ordering.
There was a deadlock condition during a slow Catalogd startup because
the thread that performs the initial global reset periodically
releases the versionLock_ write lock to allow for metadata operations
on already loaded databases to proceed. If the database was not yet
loaded, the read thread waited until it was loaded holding a read
lock on versionLock_. However, the initial global reset thread could
not proceed with loading any databases because it needed a write lock
on versionLock_.
Testing accomplished with a new custom cluster test. This new test
was run on a local Impala dev machine without the fix in place to
demonstrate it consistently failed because of the deadlock. It was
also successfully run with the fix in place both on a local Impala
dev machine and in an automated build.
Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M tests/custom_cluster/test_catalog_wait.py
2 files changed, 137 insertions(+), 11 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/75/24275/5
--
To view, visit http://gerrit.cloudera.org:8080/24275
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404
Gerrit-Change-Number: 24275
Gerrit-PatchSet: 5
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 )
Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock
......................................................................
Patch Set 8:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/22380/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/24275
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404
Gerrit-Change-Number: 24275
Gerrit-PatchSet: 8
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Mon, 11 May 2026 20:11:01 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 )
Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock
......................................................................
Patch Set 9:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/22394/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24275
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404
Gerrit-Change-Number: 24275
Gerrit-PatchSet: 9
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Tue, 12 May 2026 18:53:50 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 )
Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock
......................................................................
Patch Set 2: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/24275
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404
Gerrit-Change-Number: 24275
Gerrit-PatchSet: 2
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Thu, 07 May 2026 14:49:46 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 )
Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock
......................................................................
Patch Set 3:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/22357/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/24275
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404
Gerrit-Change-Number: 24275
Gerrit-PatchSet: 3
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Thu, 07 May 2026 22:01:21 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 )
Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/24275/1/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:
http://gerrit.cloudera.org:8080/#/c/24275/1/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2611
PS1, Line 2611: triggeredInitialReset_
> Yeah, it will be blocked, but the initial refresh should be light since the
The behavior matches syncDdl, which semantically makes sense for startup. However I wonder how it interacts with HA failover.
I think __set_sync_ddl(true) at https://github.com/apache/impala/blob/master/be/src/catalog/catalog-server.cc#L1050 would be semantically equivalent, and I think it would also be used for catalog reset after failover.
--
To view, visit http://gerrit.cloudera.org:8080/24275
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404
Gerrit-Change-Number: 24275
Gerrit-PatchSet: 1
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Tue, 05 May 2026 22:33:11 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 )
Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/24275/1/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:
http://gerrit.cloudera.org:8080/#/c/24275/1/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2611
PS1, Line 2611: triggeredInitialReset_
> That makes sense. I'm ok with this change, but I'd like Quanlong or Sai to
same here.
--
To view, visit http://gerrit.cloudera.org:8080/24275
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404
Gerrit-Change-Number: 24275
Gerrit-PatchSet: 1
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Wed, 06 May 2026 00:10:14 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 )
Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/22341/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24275
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404
Gerrit-Change-Number: 24275
Gerrit-PatchSet: 2
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Wed, 06 May 2026 23:33:41 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 )
Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/22331/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24275
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404
Gerrit-Change-Number: 24275
Gerrit-PatchSet: 1
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Tue, 05 May 2026 17:34:58 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Posted by "Yida Wu (Code Review)" <ge...@cloudera.org>.
Yida Wu has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 )
Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock
......................................................................
Patch Set 1:
I think we probably need a regression test case for this
--
To view, visit http://gerrit.cloudera.org:8080/24275
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404
Gerrit-Change-Number: 24275
Gerrit-PatchSet: 1
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Tue, 05 May 2026 20:02:53 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 )
Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock
......................................................................
Patch Set 9: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/13458/
--
To view, visit http://gerrit.cloudera.org:8080/24275
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404
Gerrit-Change-Number: 24275
Gerrit-PatchSet: 9
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Tue, 12 May 2026 23:15:55 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-14949: Fix Catalogd Startup Deadlock
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 )
Change subject: [WIP] IMPALA-14949: Fix Catalogd Startup Deadlock
......................................................................
Patch Set 6:
Marking this change as WIP because the new custom cluster test is not passing during Jenkins builds.
--
To view, visit http://gerrit.cloudera.org:8080/24275
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404
Gerrit-Change-Number: 24275
Gerrit-PatchSet: 6
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Fri, 08 May 2026 22:00:10 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/24275 )
Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock
......................................................................
IMPALA-14949: Fix Catalogd Startup Deadlock
The getOrLoadTable functionality now waits for the initial database
metadata to be loaded for the given database before it takes the
versionLock_ read lock. This order of operations is reversed from its
previous ordering.
There was a deadlock condition during a slow Catalogd startup because
the thread that performs the initial global reset periodically
releases the versionLock_ write lock to allow for metadata operations
on already loaded databases to proceed. If the database was not yet
loaded, the read thread waited until it was loaded holding a read
lock on versionLock_. However, the initial global reset thread could
not proceed with loading any databases because it needed a write lock
on versionLock_.
Testing accomplished with a new custom cluster test. This new test
was run on a local Impala dev machine without the fix in place to
demonstrate it consistently failed because of the deadlock. It was
also successfully run with the fix in place both on a local Impala
dev machine and in an automated build.
Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M tests/custom_cluster/test_catalog_wait.py
2 files changed, 137 insertions(+), 11 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/75/24275/4
--
To view, visit http://gerrit.cloudera.org:8080/24275
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404
Gerrit-Change-Number: 24275
Gerrit-PatchSet: 4
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 )
Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock
......................................................................
Patch Set 8:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/24275/8/tests/custom_cluster/test_catalog_wait.py
File tests/custom_cluster/test_catalog_wait.py:
http://gerrit.cloudera.org:8080/#/c/24275/8/tests/custom_cluster/test_catalog_wait.py@186
PS8, Line 186: func=count_completed_queries, max_attempts=60, sleep_time_s=2, backoff=1), \
> I don't think ASAN jobs will run exhaustive tests. I specifically marked t
Ack
--
To view, visit http://gerrit.cloudera.org:8080/24275
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404
Gerrit-Change-Number: 24275
Gerrit-PatchSet: 8
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Tue, 12 May 2026 15:56:00 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 )
Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock
......................................................................
Patch Set 8: Code-Review+1
(1 comment)
http://gerrit.cloudera.org:8080/#/c/24275/8/tests/custom_cluster/test_catalog_wait.py
File tests/custom_cluster/test_catalog_wait.py:
http://gerrit.cloudera.org:8080/#/c/24275/8/tests/custom_cluster/test_catalog_wait.py@186
PS8, Line 186: func=count_completed_queries, max_attempts=60, sleep_time_s=2, backoff=1), \
Should we check these in an asan run too if they need lots of time?
--
To view, visit http://gerrit.cloudera.org:8080/24275
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404
Gerrit-Change-Number: 24275
Gerrit-PatchSet: 8
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Mon, 11 May 2026 21:10:29 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/24275 )
Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock
......................................................................
IMPALA-14949: Fix Catalogd Startup Deadlock
The getOrLoadTable functionality now waits for the initial database
metadata to be loaded for the given database before it takes the
versionLock_ read lock. This order of operations is reversed from its
previous ordering.
There was a deadlock condition during a slow Catalogd startup because
the thread that performs the initial global reset periodically
releases the versionLock_ write lock to allow for metadata operations
on already loaded databases to proceed. If the database was not yet
loaded, the read thread waited until it was loaded holding a read
lock on versionLock_. However, the initial global reset thread could
not proceed with loading any databases because it needed a write lock
on versionLock_.
Testing accomplished with a new custom cluster test. This new test
was run on a local Impala dev machine without the fix in place to
demonstrate it consistently failed because of the deadlock. It was
also successfully run with the fix in place both on a local Impala
dev machine and in an automated build.
Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M tests/custom_cluster/test_catalog_wait.py
2 files changed, 137 insertions(+), 11 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/75/24275/8
--
To view, visit http://gerrit.cloudera.org:8080/24275
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404
Gerrit-Change-Number: 24275
Gerrit-PatchSet: 8
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 )
Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock
......................................................................
Patch Set 9:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/13458/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/24275
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404
Gerrit-Change-Number: 24275
Gerrit-PatchSet: 9
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Tue, 12 May 2026 18:32:28 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has uploaded a new patch set (#9). ( http://gerrit.cloudera.org:8080/24275 )
Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock
......................................................................
IMPALA-14949: Fix Catalogd Startup Deadlock
The getOrLoadTable functionality now waits for the initial database
metadata to be loaded for the given database before it takes the
versionLock_ read lock. This order of operations is reversed from its
previous ordering.
There was a deadlock condition during a slow Catalogd startup because
the thread that performs the initial global reset periodically
releases the versionLock_ write lock to allow for metadata operations
on already loaded databases to proceed. If the database was not yet
loaded, the read thread waited until it was loaded holding a read
lock on versionLock_. However, the initial global reset thread could
not proceed with loading any databases because it needed a write lock
on versionLock_.
Testing accomplished with a new custom cluster test. This new test
was run on a local Impala dev machine without the fix in place to
demonstrate it consistently failed because of the deadlock. It was
also successfully run with the fix in place both on a local Impala
dev machine and in an automated build.
Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M tests/custom_cluster/test_catalog_wait.py
2 files changed, 137 insertions(+), 11 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/75/24275/9
--
To view, visit http://gerrit.cloudera.org:8080/24275
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404
Gerrit-Change-Number: 24275
Gerrit-PatchSet: 9
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 )
Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/24275/1/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:
http://gerrit.cloudera.org:8080/#/c/24275/1/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2611
PS1, Line 2611: triggeredInitialReset_
> The behavior matches syncDdl, which semantically makes sense for startup. H
In a failover situation today, the deadlock won't happen even if CatalogServiceCatalog.rebuildDbCache releases the versionLock_ write lock because triggeredInitialReset_ will be `true` and thus other threads will not sleep in CatalogServiceCatalog.waitInitialResetCompletion.
I also considered making the reset that happens after a Catalogd failover not yield versionLock_ write lock, but that is a behavior change from what happens today. Maybe it is a change we want to make though?
--
To view, visit http://gerrit.cloudera.org:8080/24275
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404
Gerrit-Change-Number: 24275
Gerrit-PatchSet: 1
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Tue, 05 May 2026 23:45:55 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 )
Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock
......................................................................
Patch Set 1: Code-Review+1
(1 comment)
http://gerrit.cloudera.org:8080/#/c/24275/1/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:
http://gerrit.cloudera.org:8080/#/c/24275/1/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2611
PS1, Line 2611: triggeredInitialReset_
> In a failover situation today, the deadlock won't happen even if CatalogSer
That makes sense. I'm ok with this change, but I'd like Quanlong or Sai to take a look since they helped with the IMPALA-13850.
--
To view, visit http://gerrit.cloudera.org:8080/24275
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404
Gerrit-Change-Number: 24275
Gerrit-PatchSet: 1
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Tue, 05 May 2026 23:52:41 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 )
Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock
......................................................................
Patch Set 4:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/22364/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24275
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404
Gerrit-Change-Number: 24275
Gerrit-PatchSet: 4
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Fri, 08 May 2026 18:07:31 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 )
Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock
......................................................................
Patch Set 3:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/24275/3/tests/custom_cluster/test_catalog_wait.py
File tests/custom_cluster/test_catalog_wait.py:
http://gerrit.cloudera.org:8080/#/c/24275/3/tests/custom_cluster/test_catalog_wait.py@142
PS3, Line 142: lock = threading.Lock()
> The lock isn't needed; Python GIL ensures basic assignment is atomic. But i
Yeah, I think we don't need the lock. We have other examples like
https://github.com/apache/impala/blob/f7b4601482ab698ff524a68e89c9485299879062/tests/custom_cluster/test_local_catalog.py#L441-L462
https://github.com/apache/impala/blob/f7b4601482ab698ff524a68e89c9485299879062/tests/custom_cluster/test_concurrent_ddls.py#L181-L198
http://gerrit.cloudera.org:8080/#/c/24275/3/tests/custom_cluster/test_catalog_wait.py@179
PS3, Line 179: sleep(5)
> Could this be a check for inflight queries from a metric or the /queries pa
I think we can let the thread add a marker and wait for the marker to appear.
http://gerrit.cloudera.org:8080/#/c/24275/3/tests/custom_cluster/test_catalog_wait.py@218
PS3, Line 218: sleep(1)
By not killing the impalads, I found the queries all finish at the end. The longest duration is 34s. If I increase the sleep interval to 5s, the test passes without the fix.
http://gerrit.cloudera.org:8080/#/c/24275/3/tests/custom_cluster/test_catalog_wait.py@226
PS3, Line 226: impalad.kill()
Why should we explicitly kill the impalads?
--
To view, visit http://gerrit.cloudera.org:8080/24275
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404
Gerrit-Change-Number: 24275
Gerrit-PatchSet: 3
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Fri, 08 May 2026 13:18:35 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Posted by "Yida Wu (Code Review)" <ge...@cloudera.org>.
Yida Wu has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 )
Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock
......................................................................
Patch Set 2:
Have you tried to inject a new DebugAction to force the sleep for the custom test? There are some existing ones like DebugUtils.RESET_METADATA_LOOP_LOCKED in CatalogServiceCatalog.java used in custom_cluster/test_catalogd_ha.py, maybe a new DebugAction would help?
--
To view, visit http://gerrit.cloudera.org:8080/24275
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404
Gerrit-Change-Number: 24275
Gerrit-PatchSet: 2
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Wed, 06 May 2026 23:44:36 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-14949: Fix Catalogd Startup Deadlock
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 )
Change subject: [WIP] IMPALA-14949: Fix Catalogd Startup Deadlock
......................................................................
Patch Set 5:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/22367/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24275
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404
Gerrit-Change-Number: 24275
Gerrit-PatchSet: 5
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Fri, 08 May 2026 22:09:22 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 )
Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock
......................................................................
Patch Set 1:
(2 comments)
Thanks for reproducing the issue and the quick fix! The patch is safe and correct but I hope we can keep the optimization we did for IMPALA-13850 (part 4).
http://gerrit.cloudera.org:8080/#/c/24275/1/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:
http://gerrit.cloudera.org:8080/#/c/24275/1/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2611
PS1, Line 2611: triggeredInitialReset_
> same here.
This reverts the optimization we did in IMPALA-13850 (part 4) so catalogd requests still need to wait for the initial reset to finish, no matter whether their dbs have been loaded.
I think we can fix the issue like IMPALA-14400 by using ReadLockAndLookupDb. Left a comment in getOrLoadTable().
http://gerrit.cloudera.org:8080/#/c/24275/1/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2808
PS1, Line 2808: acquireVersionReadLock(catalogTimeline);
The problem is we acquire the lock first before the waiting. I think we should actually wait before the lock by using ReadLockAndLookupDb:
- acquireVersionReadLock(catalogTimeline);
- try {
- tbl = getTable(dbName, tblName);
+ try (ReadLockAndLookupDb result = new ReadLockAndLookupDb(dbName)) {
+ catalogTimeline.markEvent(GOT_CATALOG_VERSION_READ_LOCK);
+ Db db = result.getDb();
+ if (db == null) {
+ throw new DatabaseNotFoundException("Database '" + dbName + "' not found");
+ }
+ tbl = db.getTable(tblName);
So once the initial reset finishes loading the db of this request, this thread can move forward when the initial reset temporarily releases the versionLock_ write lock.
--
To view, visit http://gerrit.cloudera.org:8080/24275
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404
Gerrit-Change-Number: 24275
Gerrit-PatchSet: 1
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Wed, 06 May 2026 03:28:46 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/24275 )
Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock
......................................................................
IMPALA-14949: Fix Catalogd Startup Deadlock
The getOrLoadTable functionality now waits for the initial database
metadata to be loaded for the given database before it takes the
versionLock_ read lock. This order of operations is reversed from its
previous ordering.
There was a deadlock condition during a slow Catalogd startup because
the thread that performs the initial global reset periodically
releases the versionLock_ write lock to allow for metadata operations
on already loaded databases to proceed. If the database was not yet
loaded, the read thread waited until it was loaded holding a read
lock on versionLock_. However, the initial global reset thread could
not proceed with loading any databases because it needed a write lock
on versionLock_.
Testing accomplished with a new custom cluster test. This new test
was run on a local Impala dev machine without the fix in place to
demonstrate it consistently failed because of the deadlock. It was
also successfully run with the fix in place both on a local Impala
dev machine and in an automated build.
Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404
Reviewed-on: http://gerrit.cloudera.org:8080/24275
Reviewed-by: Yida Wu <wy...@gmail.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M tests/custom_cluster/test_catalog_wait.py
2 files changed, 137 insertions(+), 11 deletions(-)
Approvals:
Yida Wu: Looks good to me, approved
Impala Public Jenkins: Verified
--
To view, visit http://gerrit.cloudera.org:8080/24275
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404
Gerrit-Change-Number: 24275
Gerrit-PatchSet: 10
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Posted by "Yida Wu (Code Review)" <ge...@cloudera.org>.
Yida Wu has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 )
Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock
......................................................................
Patch Set 9: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/24275
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404
Gerrit-Change-Number: 24275
Gerrit-PatchSet: 9
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Tue, 12 May 2026 22:39:08 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 )
Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock
......................................................................
Patch Set 9: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/24275
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404
Gerrit-Change-Number: 24275
Gerrit-PatchSet: 9
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Wed, 13 May 2026 04:14:25 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Posted by "Yida Wu (Code Review)" <ge...@cloudera.org>.
Yida Wu has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 )
Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock
......................................................................
Patch Set 3:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/24275/3/tests/custom_cluster/test_catalog_wait.py
File tests/custom_cluster/test_catalog_wait.py:
http://gerrit.cloudera.org:8080/#/c/24275/3/tests/custom_cluster/test_catalog_wait.py@141
PS3, Line 141: test_catalog_startup_deadlock
How about adding a comment like this?
""""Regression test for IMPALA-14949: Verifies no deadlock between
the initial global reset and concurrent metadata reload/query workload.""""
--
To view, visit http://gerrit.cloudera.org:8080/24275
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404
Gerrit-Change-Number: 24275
Gerrit-PatchSet: 3
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Thu, 07 May 2026 23:09:56 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/24275 )
Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock
......................................................................
IMPALA-14949: Fix Catalogd Startup Deadlock
The getOrLoadTable functionality now waits for the initial database
metadata to be loaded for the given database before it takes the
versionLock_ read lock. This order of operations is reversed from its
previous ordering.
There was a deadlock condition during a slow Catalogd startup because
the thread that performs the initial global reset periodically
releases the versionLock_ write lock to allow for metadata operations
on already loaded databases to proceed. If the database was not yet
loaded, the read thread waited until it was loaded holding a read
lock on versionLock_. However, the initial global reset thread could
not proceed with loading any databases because it needed a write lock
on versionLock_.
Manual testing replicated this issue and demonstrated the fix
effectiveness. The replication required adding an arbitrary sleep in
the CatalogServiceCatalog.rebuildDbCache function's while loop.
Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404
---
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
1 file changed, 8 insertions(+), 11 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/75/24275/2
--
To view, visit http://gerrit.cloudera.org:8080/24275
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404
Gerrit-Change-Number: 24275
Gerrit-PatchSet: 2
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Posted by "Yida Wu (Code Review)" <ge...@cloudera.org>.
Yida Wu has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 )
Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/24275/1/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:
http://gerrit.cloudera.org:8080/#/c/24275/1/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2611
PS1, Line 2611: triggeredInitialReset_
Thanks for the quick fix. One concern is that, it seems getCatalogDelta() will be completely blocked for the entire duration of the initial global reset. Do you see a potential problem there? Or is there any lighter way for the fix?
--
To view, visit http://gerrit.cloudera.org:8080/24275
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404
Gerrit-Change-Number: 24275
Gerrit-PatchSet: 1
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Tue, 05 May 2026 17:36:56 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 )
Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/24275/1/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
File fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java:
http://gerrit.cloudera.org:8080/#/c/24275/1/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java@2611
PS1, Line 2611: triggeredInitialReset_
> Thanks for the quick fix. One concern is that, it seems getCatalogDelta() w
Yeah, it will be blocked, but the initial refresh should be light since there won't be any data loaded. Also, a refresh caused by a failover won't block on this operation since both the active and passive catalogd run the initial refresh all.
--
To view, visit http://gerrit.cloudera.org:8080/24275
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404
Gerrit-Change-Number: 24275
Gerrit-PatchSet: 1
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Tue, 05 May 2026 19:49:47 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 )
Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock
......................................................................
Patch Set 8:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/24275/8/tests/custom_cluster/test_catalog_wait.py
File tests/custom_cluster/test_catalog_wait.py:
http://gerrit.cloudera.org:8080/#/c/24275/8/tests/custom_cluster/test_catalog_wait.py@186
PS8, Line 186: func=count_completed_queries, max_attempts=60, sleep_time_s=2, backoff=1), \
> Should we check these in an asan run too if they need lots of time?
I don't think ASAN jobs will run exhaustive tests. I specifically marked this test as exhaustive only since it takes so long to run (40+ seconds locally), I was concerned about it failing in ASAN runs, and it is checking for a deadlock in Java thus ASAN/TSAN won't apply.
--
To view, visit http://gerrit.cloudera.org:8080/24275
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404
Gerrit-Change-Number: 24275
Gerrit-PatchSet: 8
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Tue, 12 May 2026 14:41:53 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 )
Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock
......................................................................
Patch Set 8:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/22381/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24275
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404
Gerrit-Change-Number: 24275
Gerrit-PatchSet: 8
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Mon, 11 May 2026 21:06:23 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14949: Fix Catalogd Startup Deadlock
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24275 )
Change subject: IMPALA-14949: Fix Catalogd Startup Deadlock
......................................................................
Patch Set 9:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/13464/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/24275
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibe71980d6fed5a95b00bc7b710c781f909545404
Gerrit-Change-Number: 24275
Gerrit-PatchSet: 9
Gerrit-Owner: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sai Hemanth Gantasala <sa...@cloudera.com>
Gerrit-Reviewer: Yida Wu <wy...@gmail.com>
Gerrit-Comment-Date: Tue, 12 May 2026 23:57:30 +0000
Gerrit-HasComments: No