You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Riza Suminto (Code Review)" <ge...@cloudera.org> on 2025/07/14 16:08:38 UTC

[Impala-ASF-CR] IMPALA-14220: CatalogServer::IsActive must not hold catalog lock

Riza Suminto has uploaded this change for review. ( http://gerrit.cloudera.org:8080/23168


Change subject: IMPALA-14220: CatalogServer::IsActive must not hold catalog_lock_
......................................................................

IMPALA-14220: CatalogServer::IsActive must not hold catalog_lock_

When catalogd HA is enabled, catalogd will check whether it is the
active one, via CatalogServer::IsActive, before serving each request.
Calling CatalogServer::IsActive require obtaining catalog_lock_, which
can contend with long catalog operation such as
GatherCatalogUpdatesThread.

Checking current catalog active status does not need to obtain
catalog_lock_. Instead, it is sufficient to change is_active_ field from
boolean to AtomicBoolean. This patch applies that change.

Testing:
Run and pass test_catalogd_ha.py.

Change-Id: I15fb925f1eb4ea5d213075b66a676d2bc9b9e9f1
---
M be/src/catalog/catalog-server.cc
M be/src/catalog/catalog-server.h
M be/src/catalog/workload-management-init.cc
3 files changed, 11 insertions(+), 12 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/68/23168/1
-- 
To view, visit http://gerrit.cloudera.org:8080/23168
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I15fb925f1eb4ea5d213075b66a676d2bc9b9e9f1
Gerrit-Change-Number: 23168
Gerrit-PatchSet: 1
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>

[Impala-ASF-CR] IMPALA-14220: CatalogServer::IsActive must not hold catalog lock

Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/23168 )

Change subject: IMPALA-14220: CatalogServer::IsActive must not hold catalog_lock_
......................................................................


Patch Set 2:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/23168/2/be/src/catalog/catalog-server.h
File be/src/catalog/catalog-server.h:

http://gerrit.cloudera.org:8080/#/c/23168/2/be/src/catalog/catalog-server.h@228
PS2, Line 228: AtomicBool
> It seems we are not using any atomic feature like CAS operations. Maybe dec
Thanks Quanlong, that make sense.
Since both AtomicBoolean and volatile should work, I rather keep it as AtomicBoolean, just in case we need to change the writer assumption in the future. I assume the read performance will be negligible / still better than before the patch.
I will merge this as is.



-- 
To view, visit http://gerrit.cloudera.org:8080/23168
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15fb925f1eb4ea5d213075b66a676d2bc9b9e9f1
Gerrit-Change-Number: 23168
Gerrit-PatchSet: 2
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Tue, 15 Jul 2025 03:20:11 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-14220: CatalogServer::IsActive must not hold catalog lock

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/23168 )

Change subject: IMPALA-14220: CatalogServer::IsActive must not hold catalog_lock_
......................................................................


Patch Set 2:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/23168/2/be/src/catalog/catalog-server.h
File be/src/catalog/catalog-server.h:

http://gerrit.cloudera.org:8080/#/c/23168/2/be/src/catalog/catalog-server.h@228
PS2, Line 228: AtomicBool
It seems we are not using any atomic feature like CAS operations. Maybe declaring this as volatile is enough. The only writer is the UpdateActiveCatalogd thread which is always a single thread due to catalog_lock_.



-- 
To view, visit http://gerrit.cloudera.org:8080/23168
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15fb925f1eb4ea5d213075b66a676d2bc9b9e9f1
Gerrit-Change-Number: 23168
Gerrit-PatchSet: 2
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Tue, 15 Jul 2025 00:57:42 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-14220: CatalogServer::IsActive must not hold catalog lock

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23168 )

Change subject: IMPALA-14220: CatalogServer::IsActive must not hold catalog_lock_
......................................................................


Patch Set 2: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/23168
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15fb925f1eb4ea5d213075b66a676d2bc9b9e9f1
Gerrit-Change-Number: 23168
Gerrit-PatchSet: 2
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Tue, 15 Jul 2025 03:15:30 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14220: CatalogServer::IsActive must not hold catalog lock

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23168 )

Change subject: IMPALA-14220: CatalogServer::IsActive must not hold catalog_lock_
......................................................................


Patch Set 2:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/12177/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/23168
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15fb925f1eb4ea5d213075b66a676d2bc9b9e9f1
Gerrit-Change-Number: 23168
Gerrit-PatchSet: 2
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Mon, 14 Jul 2025 21:14:19 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14220: CatalogServer::IsActive must not hold catalog lock

Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/23168 )

Change subject: IMPALA-14220: CatalogServer::IsActive must not hold catalog_lock_
......................................................................


Patch Set 2: Code-Review+2

carry +1 from Abhishek.


-- 
To view, visit http://gerrit.cloudera.org:8080/23168
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15fb925f1eb4ea5d213075b66a676d2bc9b9e9f1
Gerrit-Change-Number: 23168
Gerrit-PatchSet: 2
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Mon, 14 Jul 2025 20:53:46 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14220: CatalogServer::IsActive must not hold catalog lock

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23168 )

Change subject: IMPALA-14220: CatalogServer::IsActive must not hold catalog_lock_
......................................................................


Patch Set 1:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/19869/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/23168
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15fb925f1eb4ea5d213075b66a676d2bc9b9e9f1
Gerrit-Change-Number: 23168
Gerrit-PatchSet: 1
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Mon, 14 Jul 2025 16:38:13 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-14220: CatalogServer::IsActive must not hold catalog lock

Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/23168 )

Change subject: IMPALA-14220: CatalogServer::IsActive must not hold catalog_lock_
......................................................................

IMPALA-14220: CatalogServer::IsActive must not hold catalog_lock_

When catalogd HA is enabled, catalogd will check whether it is the
active one, via CatalogServer::IsActive, before serving each request.
Calling CatalogServer::IsActive require obtaining catalog_lock_, which
can contend with long catalog operation such as
GatherCatalogUpdatesThread.

Checking current catalog active status does not need to obtain
catalog_lock_. Instead, it is sufficient to change is_active_ field from
boolean to AtomicBoolean. This patch applies that change.

With this change, CatalogServiceThriftIf::AcceptRequest can return
faster. Other CatalogServiceThriftIf methods that previously blocked on
AcceptRequest method can proceeed faster, but might still contend over
Catalog's versionLock_ in JVM later.

Testing:
Run and pass test_catalogd_ha.py.

Change-Id: I15fb925f1eb4ea5d213075b66a676d2bc9b9e9f1
Reviewed-on: http://gerrit.cloudera.org:8080/23168
Reviewed-by: Abhishek Rawat <ar...@cloudera.com>
Reviewed-by: Wenzhe Zhou <wz...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/catalog/catalog-server.cc
M be/src/catalog/catalog-server.h
M be/src/catalog/workload-management-init.cc
3 files changed, 11 insertions(+), 12 deletions(-)

Approvals:
  Abhishek Rawat: Looks good to me, but someone else must approve
  Wenzhe Zhou: Looks good to me, approved
  Impala Public Jenkins: Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/23168
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I15fb925f1eb4ea5d213075b66a676d2bc9b9e9f1
Gerrit-Change-Number: 23168
Gerrit-PatchSet: 3
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>

[Impala-ASF-CR] IMPALA-14220: CatalogServer::IsActive must not hold catalog lock

Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Hello Quanlong Huang, Wenzhe Zhou, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/23168

to look at the new patch set (#2).

Change subject: IMPALA-14220: CatalogServer::IsActive must not hold catalog_lock_
......................................................................

IMPALA-14220: CatalogServer::IsActive must not hold catalog_lock_

When catalogd HA is enabled, catalogd will check whether it is the
active one, via CatalogServer::IsActive, before serving each request.
Calling CatalogServer::IsActive require obtaining catalog_lock_, which
can contend with long catalog operation such as
GatherCatalogUpdatesThread.

Checking current catalog active status does not need to obtain
catalog_lock_. Instead, it is sufficient to change is_active_ field from
boolean to AtomicBoolean. This patch applies that change.

With this change, CatalogServiceThriftIf::AcceptRequest can return
faster. Other CatalogServiceThriftIf methods that previously blocked on
AcceptRequest method can proceeed faster, but might still contend over
Catalog's versionLock_ in JVM later.

Testing:
Run and pass test_catalogd_ha.py.

Change-Id: I15fb925f1eb4ea5d213075b66a676d2bc9b9e9f1
---
M be/src/catalog/catalog-server.cc
M be/src/catalog/catalog-server.h
M be/src/catalog/workload-management-init.cc
3 files changed, 11 insertions(+), 12 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/68/23168/2
-- 
To view, visit http://gerrit.cloudera.org:8080/23168
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I15fb925f1eb4ea5d213075b66a676d2bc9b9e9f1
Gerrit-Change-Number: 23168
Gerrit-PatchSet: 2
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>

[Impala-ASF-CR] IMPALA-14220: CatalogServer::IsActive must not hold catalog lock

Posted by "Abhishek Rawat (Code Review)" <ge...@cloudera.org>.
Abhishek Rawat has posted comments on this change. ( http://gerrit.cloudera.org:8080/23168 )

Change subject: IMPALA-14220: CatalogServer::IsActive must not hold catalog_lock_
......................................................................


Patch Set 2: Code-Review+1


-- 
To view, visit http://gerrit.cloudera.org:8080/23168
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I15fb925f1eb4ea5d213075b66a676d2bc9b9e9f1
Gerrit-Change-Number: 23168
Gerrit-PatchSet: 2
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Mon, 14 Jul 2025 20:47:51 +0000
Gerrit-HasComments: No