You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Andrew Sherman (Code Review)" <ge...@cloudera.org> on 2024/09/30 23:43:16 UTC
[Impala-ASF-CR] IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
Andrew Sherman has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21867
Change subject: IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
......................................................................
IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
The cluster_id flag was introduced in IMPALA-12426 to identify Impala
clusters in systems where a single query_log table could be shared. In
IMPALA-13208 the cluster_id flag was reused as a prefix to topic names
for backend membership, to allow sub-clusters of backends within a
Statestore service.
There have been some problems with the interaction of these two usages.
An important difference is that the query_log cluster_id must be set
only on coordinators, whereas the topic prefix cluster_id must be set
simultaneously on coordinators, executors, and admission daemons
(if present). If a system is started with cluster_id set only on
coordinators then there are split-brain problems where coordinators and
executors are tracked in different topics. In addition, the query_log
cluster_id is more likely to be user-settable as it is used for data in
query_log which will be read by humans, who may want to write queries
selecting data from their ‘production’ or ‘dev’ clusters.
Avoid these problems by using a separate flag for the topic prefix
cluster_id ‘cluster_membership_topic_id’.
Change-Id: Icd3f7e1c73c00a7aaeee79ecb461209e3939c422
---
M be/src/runtime/exec-env.cc
M be/src/scheduling/admission-controller.cc
M be/src/scheduling/admissiond-env.cc
M be/src/scheduling/cluster-membership-mgr.cc
M tests/custom_cluster/test_query_live.py
M tests/custom_cluster/test_shared_catalogd.py
6 files changed, 23 insertions(+), 15 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/67/21867/1
--
To view, visit http://gerrit.cloudera.org:8080/21867
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Icd3f7e1c73c00a7aaeee79ecb461209e3939c422
Gerrit-Change-Number: 21867
Gerrit-PatchSet: 1
Gerrit-Owner: Andrew Sherman <as...@cloudera.com>
[Impala-ASF-CR] IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has posted comments on this change. ( http://gerrit.cloudera.org:8080/21867 )
Change subject: IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
......................................................................
Patch Set 5:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/21867/5/tests/custom_cluster/test_query_live.py
File tests/custom_cluster/test_query_live.py:
http://gerrit.cloudera.org:8080/#/c/21867/5/tests/custom_cluster/test_query_live.py@281
PS5, Line 281: @CustomClusterTestSuite.with_args(
> But why does this test need cluster_membership_topic_id? It doesn't seem re
Also, the cluster_membership_topic_id flag is supposed to be optional.
--
To view, visit http://gerrit.cloudera.org:8080/21867
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd3f7e1c73c00a7aaeee79ecb461209e3939c422
Gerrit-Change-Number: 21867
Gerrit-PatchSet: 5
Gerrit-Owner: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 04 Oct 2024 20:31:27 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
Posted by "Andrew Sherman (Code Review)" <ge...@cloudera.org>.
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/21867 )
Change subject: IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
......................................................................
Patch Set 3:
I have one small test fix still to do
--
To view, visit http://gerrit.cloudera.org:8080/21867
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd3f7e1c73c00a7aaeee79ecb461209e3939c422
Gerrit-Change-Number: 21867
Gerrit-PatchSet: 3
Gerrit-Owner: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 02 Oct 2024 22:58:34 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
Posted by "Andrew Sherman (Code Review)" <ge...@cloudera.org>.
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/21867 )
Change subject: IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
......................................................................
Patch Set 2:
(2 comments)
Thanks Daniel for good questions
http://gerrit.cloudera.org:8080/#/c/21867/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/21867/2//COMMIT_MSG@9
PS2, Line 9: IMPALA-12426
> I'm not familiar with these changes but I can't find 'cluster_id' being int
There were multiple reviews under the same Jira id, I think it is https://gerrit.cloudera.org/#/c/20770/ in the file workload-management-flags.cc
http://gerrit.cloudera.org:8080/#/c/21867/2/be/src/scheduling/admission-controller.cc
File be/src/scheduling/admission-controller.cc:
http://gerrit.cloudera.org:8080/#/c/21867/2/be/src/scheduling/admission-controller.cc@82
PS2, Line 82: cluster_id
> Do we have 'cluster_id' left anywhere?
It still exists, defined in workload-management-flags.cc
The change here is because the usages are subtly different so I want to tease them apart.
--
To view, visit http://gerrit.cloudera.org:8080/21867
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd3f7e1c73c00a7aaeee79ecb461209e3939c422
Gerrit-Change-Number: 21867
Gerrit-PatchSet: 2
Gerrit-Owner: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 01 Oct 2024 16:16:52 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21867 )
Change subject: IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
......................................................................
Patch Set 8:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/11009/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/21867
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd3f7e1c73c00a7aaeee79ecb461209e3939c422
Gerrit-Change-Number: 21867
Gerrit-PatchSet: 8
Gerrit-Owner: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 09 Oct 2024 16:38:52 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21867 )
Change subject: IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
......................................................................
Patch Set 6:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/17126/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/21867
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd3f7e1c73c00a7aaeee79ecb461209e3939c422
Gerrit-Change-Number: 21867
Gerrit-PatchSet: 6
Gerrit-Owner: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 08 Oct 2024 22:56:55 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21867 )
Change subject: IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/17073/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/21867
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd3f7e1c73c00a7aaeee79ecb461209e3939c422
Gerrit-Change-Number: 21867
Gerrit-PatchSet: 1
Gerrit-Owner: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 01 Oct 2024 00:09:00 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21867 )
Change subject: IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/17074/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/21867
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd3f7e1c73c00a7aaeee79ecb461209e3939c422
Gerrit-Change-Number: 21867
Gerrit-PatchSet: 2
Gerrit-Owner: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 01 Oct 2024 00:18:32 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21867 )
Change subject: IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
......................................................................
Patch Set 5:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10997/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/21867
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd3f7e1c73c00a7aaeee79ecb461209e3939c422
Gerrit-Change-Number: 21867
Gerrit-PatchSet: 5
Gerrit-Owner: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 04 Oct 2024 16:58:22 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21867 )
Change subject: IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
......................................................................
Patch Set 7: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/11006/
--
To view, visit http://gerrit.cloudera.org:8080/21867
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd3f7e1c73c00a7aaeee79ecb461209e3939c422
Gerrit-Change-Number: 21867
Gerrit-PatchSet: 7
Gerrit-Owner: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 09 Oct 2024 04:03:46 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
Posted by "Andrew Sherman (Code Review)" <ge...@cloudera.org>.
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/21867 )
Change subject: IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
......................................................................
Patch Set 4:
Fix test
--
To view, visit http://gerrit.cloudera.org:8080/21867
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd3f7e1c73c00a7aaeee79ecb461209e3939c422
Gerrit-Change-Number: 21867
Gerrit-PatchSet: 4
Gerrit-Owner: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 04 Oct 2024 16:52:11 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/21867 )
Change subject: IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
......................................................................
Patch Set 5:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/21867/5/tests/custom_cluster/test_query_live.py
File tests/custom_cluster/test_query_live.py:
http://gerrit.cloudera.org:8080/#/c/21867/5/tests/custom_cluster/test_query_live.py@281
PS5, Line 281: @CustomClusterTestSuite.with_args(
> This test starts a cluster (using the @CustomClusterTestSuite.with_args), t
But why does this test need cluster_membership_topic_id? It doesn't seem related to that option.
--
To view, visit http://gerrit.cloudera.org:8080/21867
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd3f7e1c73c00a7aaeee79ecb461209e3939c422
Gerrit-Change-Number: 21867
Gerrit-PatchSet: 5
Gerrit-Owner: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 04 Oct 2024 19:50:34 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/21867 )
Change subject: IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
......................................................................
Patch Set 5: Code-Review+1
Reviewing one of the tests, hopefully this'll stop it from automatically merging.
--
To view, visit http://gerrit.cloudera.org:8080/21867
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd3f7e1c73c00a7aaeee79ecb461209e3939c422
Gerrit-Change-Number: 21867
Gerrit-PatchSet: 5
Gerrit-Owner: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 04 Oct 2024 21:06:23 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21867 )
Change subject: IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
......................................................................
Patch Set 3: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10993/
--
To view, visit http://gerrit.cloudera.org:8080/21867
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd3f7e1c73c00a7aaeee79ecb461209e3939c422
Gerrit-Change-Number: 21867
Gerrit-PatchSet: 3
Gerrit-Owner: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 03 Oct 2024 03:31:01 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21867 )
Change subject: IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
......................................................................
Patch Set 5: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10997/
--
To view, visit http://gerrit.cloudera.org:8080/21867
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd3f7e1c73c00a7aaeee79ecb461209e3939c422
Gerrit-Change-Number: 21867
Gerrit-PatchSet: 5
Gerrit-Owner: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 04 Oct 2024 22:09:09 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/21867 )
Change subject: IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
......................................................................
Patch Set 6: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/21867
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd3f7e1c73c00a7aaeee79ecb461209e3939c422
Gerrit-Change-Number: 21867
Gerrit-PatchSet: 6
Gerrit-Owner: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 08 Oct 2024 22:35:46 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21867 )
Change subject: IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
......................................................................
Patch Set 4:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/17099/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/21867
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd3f7e1c73c00a7aaeee79ecb461209e3939c422
Gerrit-Change-Number: 21867
Gerrit-PatchSet: 4
Gerrit-Owner: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 04 Oct 2024 17:17:45 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21867 )
Change subject: IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
......................................................................
Patch Set 3: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/21867
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd3f7e1c73c00a7aaeee79ecb461209e3939c422
Gerrit-Change-Number: 21867
Gerrit-PatchSet: 3
Gerrit-Owner: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 02 Oct 2024 22:21:10 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
Posted by "Andrew Sherman (Code Review)" <ge...@cloudera.org>.
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/21867 )
Change subject: IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
......................................................................
Patch Set 2:
(2 comments)
Thanks for more good questions
http://gerrit.cloudera.org:8080/#/c/21867/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/21867/2//COMMIT_MSG@16
PS2, Line 16: An important difference is that the query_log cluster_id must be set
> I'm not clear how this is enforced. I don't see anything that would error i
Yes, there is nothing preventing this being set, it will just be ignored. In my opinion it is not an error to set cluster_id elsewhere, and it may be useful to allow it it to be set universally.
http://gerrit.cloudera.org:8080/#/c/21867/2//COMMIT_MSG@18
PS2, Line 18: simultaneously on coordinators, executors, and admission daemons
> Also catalogd?
The idea behind IMPALA-13208 is (IIUIC) to have a single catalog service that manages multiple sub-clusters. So catalog service still uses a single topic when subscribing to statestore updates (in catalog-server.cc).
--
To view, visit http://gerrit.cloudera.org:8080/21867
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd3f7e1c73c00a7aaeee79ecb461209e3939c422
Gerrit-Change-Number: 21867
Gerrit-PatchSet: 2
Gerrit-Owner: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 01 Oct 2024 18:42:35 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/21867 )
Change subject: IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
......................................................................
Patch Set 2:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/21867/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/21867/2//COMMIT_MSG@16
PS2, Line 16: An important difference is that the query_log cluster_id must be set
> Yes, there is nothing preventing this being set, it will just be ignored. I
I guess I don't understand how "cluster_id must be set only on coordinators" then. This patch seems more like some semantic cleanup rather than a hard requirement.
http://gerrit.cloudera.org:8080/#/c/21867/2//COMMIT_MSG@18
PS2, Line 18: simultaneously on coordinators, executors, and admission daemons
> The idea behind IMPALA-13208 is (IIUIC) to have a single catalog service th
Oh right.
--
To view, visit http://gerrit.cloudera.org:8080/21867
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd3f7e1c73c00a7aaeee79ecb461209e3939c422
Gerrit-Change-Number: 21867
Gerrit-PatchSet: 2
Gerrit-Owner: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 01 Oct 2024 18:45:30 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
Posted by "Andrew Sherman (Code Review)" <ge...@cloudera.org>.
Andrew Sherman has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/21867 )
Change subject: IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
......................................................................
IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
The cluster_id flag was introduced in IMPALA-12426 to identify Impala
clusters in systems where a single query_log table could be shared. In
IMPALA-13208 the cluster_id flag was reused as a prefix to topic names
for backend membership, to allow sub-clusters of backends within a
Statestore service.
There have been some problems with the interaction of these two usages.
An important difference is that the query_log cluster_id must be set
only on coordinators, whereas the topic prefix cluster_id must be set
simultaneously on coordinators, executors, and admission daemons
(if present). If a system is started with cluster_id set only on
coordinators then there are split-brain problems where coordinators and
executors are tracked in different topics. In addition, the query_log
cluster_id is more likely to be user-settable as it is used for data in
query_log which will be read by humans, who may want to write queries
selecting data from their ‘production’ or ‘dev’ clusters.
Avoid these problems by using a separate flag for the topic prefix
cluster_id ‘cluster_membership_topic_id’.
Change-Id: Icd3f7e1c73c00a7aaeee79ecb461209e3939c422
---
M be/src/runtime/exec-env.cc
M be/src/scheduling/admission-controller.cc
M be/src/scheduling/admissiond-env.cc
M be/src/scheduling/cluster-membership-mgr.cc
M tests/custom_cluster/test_shared_catalogd.py
5 files changed, 21 insertions(+), 14 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/67/21867/6
--
To view, visit http://gerrit.cloudera.org:8080/21867
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Icd3f7e1c73c00a7aaeee79ecb461209e3939c422
Gerrit-Change-Number: 21867
Gerrit-PatchSet: 6
Gerrit-Owner: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
Posted by "Andrew Sherman (Code Review)" <ge...@cloudera.org>.
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/21867 )
Change subject: IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
......................................................................
Patch Set 5:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/21867/5/tests/custom_cluster/test_query_live.py
File tests/custom_cluster/test_query_live.py:
http://gerrit.cloudera.org:8080/#/c/21867/5/tests/custom_cluster/test_query_live.py@281
PS5, Line 281: @CustomClusterTestSuite.with_args(
> OK I will take another look at this, maybe I have just confused myself
Removed my unnecessary changes to test_query_live
--
To view, visit http://gerrit.cloudera.org:8080/21867
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd3f7e1c73c00a7aaeee79ecb461209e3939c422
Gerrit-Change-Number: 21867
Gerrit-PatchSet: 5
Gerrit-Owner: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 08 Oct 2024 22:30:54 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
Posted by "Andrew Sherman (Code Review)" <ge...@cloudera.org>.
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/21867 )
Change subject: IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
......................................................................
Patch Set 4: Code-Review+2
Carry forward +2
--
To view, visit http://gerrit.cloudera.org:8080/21867
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd3f7e1c73c00a7aaeee79ecb461209e3939c422
Gerrit-Change-Number: 21867
Gerrit-PatchSet: 4
Gerrit-Owner: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 04 Oct 2024 16:52:34 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21867 )
Change subject: IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
......................................................................
Patch Set 5: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/21867
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd3f7e1c73c00a7aaeee79ecb461209e3939c422
Gerrit-Change-Number: 21867
Gerrit-PatchSet: 5
Gerrit-Owner: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 04 Oct 2024 16:58:21 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21867 )
Change subject: IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
......................................................................
Patch Set 1:
(1 comment)
gerrit-auto-critic failed. You can reproduce it locally using command:
python2 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with virtualenv installed.
http://gerrit.cloudera.org:8080/#/c/21867/1/be/src/scheduling/admission-controller.cc
File be/src/scheduling/admission-controller.cc:
http://gerrit.cloudera.org:8080/#/c/21867/1/be/src/scheduling/admission-controller.cc@86
PS1, Line 86: "ids used for grouping Impala backends and Admission Daemons into sub-groups within an "
line too long (92 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/21867
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd3f7e1c73c00a7aaeee79ecb461209e3939c422
Gerrit-Change-Number: 21867
Gerrit-PatchSet: 1
Gerrit-Owner: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 30 Sep 2024 23:44:15 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
Posted by "Andrew Sherman (Code Review)" <ge...@cloudera.org>.
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/21867 )
Change subject: IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
......................................................................
Patch Set 5:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/21867/5/tests/custom_cluster/test_query_live.py
File tests/custom_cluster/test_query_live.py:
http://gerrit.cloudera.org:8080/#/c/21867/5/tests/custom_cluster/test_query_live.py@281
PS5, Line 281: @CustomClusterTestSuite.with_args(
> Also, the cluster_membership_topic_id flag is supposed to be optional.
OK I will take another look at this, maybe I have just confused myself
--
To view, visit http://gerrit.cloudera.org:8080/21867
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd3f7e1c73c00a7aaeee79ecb461209e3939c422
Gerrit-Change-Number: 21867
Gerrit-PatchSet: 5
Gerrit-Owner: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 04 Oct 2024 21:03:26 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/21867 )
Change subject: IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
......................................................................
Patch Set 2: Code-Review+1
(2 comments)
http://gerrit.cloudera.org:8080/#/c/21867/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/21867/2//COMMIT_MSG@16
PS2, Line 16: An important difference is that the query_log cluster_id must be set
I'm not clear how this is enforced. I don't see anything that would error if cluster_id is set on executors or admissiond.
http://gerrit.cloudera.org:8080/#/c/21867/2//COMMIT_MSG@18
PS2, Line 18: simultaneously on coordinators, executors, and admission daemons
Also catalogd?
--
To view, visit http://gerrit.cloudera.org:8080/21867
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd3f7e1c73c00a7aaeee79ecb461209e3939c422
Gerrit-Change-Number: 21867
Gerrit-PatchSet: 2
Gerrit-Owner: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 01 Oct 2024 17:52:43 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
Posted by "Andrew Sherman (Code Review)" <ge...@cloudera.org>.
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/21867 )
Change subject: IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
......................................................................
Patch Set 5:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/21867/5/tests/custom_cluster/test_query_live.py
File tests/custom_cluster/test_query_live.py:
http://gerrit.cloudera.org:8080/#/c/21867/5/tests/custom_cluster/test_query_live.py@281
PS5, Line 281: @CustomClusterTestSuite.with_args(
> Why did this one particular test require the new flag to be set?
This test starts a cluster (using the @CustomClusterTestSuite.with_args), then it adds executors (using the _start_impala_cluster), so they both have to have the new flag.
--
To view, visit http://gerrit.cloudera.org:8080/21867
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd3f7e1c73c00a7aaeee79ecb461209e3939c422
Gerrit-Change-Number: 21867
Gerrit-PatchSet: 5
Gerrit-Owner: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 04 Oct 2024 19:48:24 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has posted comments on this change. ( http://gerrit.cloudera.org:8080/21867 )
Change subject: IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
......................................................................
Patch Set 5:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/21867/5/tests/custom_cluster/test_query_live.py
File tests/custom_cluster/test_query_live.py:
http://gerrit.cloudera.org:8080/#/c/21867/5/tests/custom_cluster/test_query_live.py@281
PS5, Line 281: @CustomClusterTestSuite.with_args(
Why did this one particular test require the new flag to be set?
--
To view, visit http://gerrit.cloudera.org:8080/21867
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd3f7e1c73c00a7aaeee79ecb461209e3939c422
Gerrit-Change-Number: 21867
Gerrit-PatchSet: 5
Gerrit-Owner: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 04 Oct 2024 19:24:51 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21867 )
Change subject: IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
......................................................................
Patch Set 7:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/11006/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/21867
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd3f7e1c73c00a7aaeee79ecb461209e3939c422
Gerrit-Change-Number: 21867
Gerrit-PatchSet: 7
Gerrit-Owner: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 08 Oct 2024 22:36:15 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21867 )
Change subject: IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
......................................................................
Patch Set 7: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/21867
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd3f7e1c73c00a7aaeee79ecb461209e3939c422
Gerrit-Change-Number: 21867
Gerrit-PatchSet: 7
Gerrit-Owner: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 08 Oct 2024 22:36:14 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has removed a vote on this change.
Change subject: IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
......................................................................
Removed Code-Review+2 by Impala Public Jenkins <im...@cloudera.com>
--
To view, visit http://gerrit.cloudera.org:8080/21867
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: Icd3f7e1c73c00a7aaeee79ecb461209e3939c422
Gerrit-Change-Number: 21867
Gerrit-PatchSet: 5
Gerrit-Owner: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
Posted by "Andrew Sherman (Code Review)" <ge...@cloudera.org>.
Andrew Sherman has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/21867 )
Change subject: IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
......................................................................
IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
The cluster_id flag was introduced in IMPALA-12426 to identify Impala
clusters in systems where a single query_log table could be shared. In
IMPALA-13208 the cluster_id flag was reused as a prefix to topic names
for backend membership, to allow sub-clusters of backends within a
Statestore service.
There have been some problems with the interaction of these two usages.
An important difference is that the query_log cluster_id must be set
only on coordinators, whereas the topic prefix cluster_id must be set
simultaneously on coordinators, executors, and admission daemons
(if present). If a system is started with cluster_id set only on
coordinators then there are split-brain problems where coordinators and
executors are tracked in different topics. In addition, the query_log
cluster_id is more likely to be user-settable as it is used for data in
query_log which will be read by humans, who may want to write queries
selecting data from their ‘production’ or ‘dev’ clusters.
Avoid these problems by using a separate flag for the topic prefix
cluster_id ‘cluster_membership_topic_id’.
Change-Id: Icd3f7e1c73c00a7aaeee79ecb461209e3939c422
---
M be/src/runtime/exec-env.cc
M be/src/scheduling/admission-controller.cc
M be/src/scheduling/admissiond-env.cc
M be/src/scheduling/cluster-membership-mgr.cc
M tests/custom_cluster/test_query_live.py
M tests/custom_cluster/test_shared_catalogd.py
6 files changed, 36 insertions(+), 24 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/67/21867/4
--
To view, visit http://gerrit.cloudera.org:8080/21867
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Icd3f7e1c73c00a7aaeee79ecb461209e3939c422
Gerrit-Change-Number: 21867
Gerrit-PatchSet: 4
Gerrit-Owner: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21867 )
Change subject: IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
......................................................................
Patch Set 3:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10993/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/21867
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd3f7e1c73c00a7aaeee79ecb461209e3939c422
Gerrit-Change-Number: 21867
Gerrit-PatchSet: 3
Gerrit-Owner: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 02 Oct 2024 22:21:11 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/21867 )
Change subject: IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/21867/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/21867/2//COMMIT_MSG@18
PS2, Line 18: simultaneously on coordinators, executors, and admission daemons
> Oh right.
Ack
--
To view, visit http://gerrit.cloudera.org:8080/21867
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd3f7e1c73c00a7aaeee79ecb461209e3939c422
Gerrit-Change-Number: 21867
Gerrit-PatchSet: 2
Gerrit-Owner: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 01 Oct 2024 18:45:42 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21867 )
Change subject: IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
......................................................................
Patch Set 8: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/21867
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd3f7e1c73c00a7aaeee79ecb461209e3939c422
Gerrit-Change-Number: 21867
Gerrit-PatchSet: 8
Gerrit-Owner: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 09 Oct 2024 16:38:52 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/21867 )
Change subject: IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
......................................................................
IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
The cluster_id flag was introduced in IMPALA-12426 to identify Impala
clusters in systems where a single query_log table could be shared. In
IMPALA-13208 the cluster_id flag was reused as a prefix to topic names
for backend membership, to allow sub-clusters of backends within a
Statestore service.
There have been some problems with the interaction of these two usages.
An important difference is that the query_log cluster_id must be set
only on coordinators, whereas the topic prefix cluster_id must be set
simultaneously on coordinators, executors, and admission daemons
(if present). If a system is started with cluster_id set only on
coordinators then there are split-brain problems where coordinators and
executors are tracked in different topics. In addition, the query_log
cluster_id is more likely to be user-settable as it is used for data in
query_log which will be read by humans, who may want to write queries
selecting data from their ‘production’ or ‘dev’ clusters.
Avoid these problems by using a separate flag for the topic prefix
cluster_id ‘cluster_membership_topic_id’.
Change-Id: Icd3f7e1c73c00a7aaeee79ecb461209e3939c422
Reviewed-on: http://gerrit.cloudera.org:8080/21867
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/runtime/exec-env.cc
M be/src/scheduling/admission-controller.cc
M be/src/scheduling/admissiond-env.cc
M be/src/scheduling/cluster-membership-mgr.cc
M tests/custom_cluster/test_shared_catalogd.py
5 files changed, 21 insertions(+), 14 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/21867
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Icd3f7e1c73c00a7aaeee79ecb461209e3939c422
Gerrit-Change-Number: 21867
Gerrit-PatchSet: 9
Gerrit-Owner: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21867 )
Change subject: IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
......................................................................
Patch Set 8: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/21867
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd3f7e1c73c00a7aaeee79ecb461209e3939c422
Gerrit-Change-Number: 21867
Gerrit-PatchSet: 8
Gerrit-Owner: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 09 Oct 2024 21:46:46 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
Posted by "Daniel Becker (Code Review)" <ge...@cloudera.org>.
Daniel Becker has posted comments on this change. ( http://gerrit.cloudera.org:8080/21867 )
Change subject: IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
......................................................................
Patch Set 2:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/21867/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/21867/2//COMMIT_MSG@9
PS2, Line 9: IMPALA-12426
I'm not familiar with these changes but I can't find 'cluster_id' being introduced in IMPALA-12426 (https://gerrit.cloudera.org/#/c/20524/).
http://gerrit.cloudera.org:8080/#/c/21867/2/be/src/scheduling/admission-controller.cc
File be/src/scheduling/admission-controller.cc:
http://gerrit.cloudera.org:8080/#/c/21867/2/be/src/scheduling/admission-controller.cc@82
PS2, Line 82: cluster_id
Do we have 'cluster_id' left anywhere?
--
To view, visit http://gerrit.cloudera.org:8080/21867
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd3f7e1c73c00a7aaeee79ecb461209e3939c422
Gerrit-Change-Number: 21867
Gerrit-PatchSet: 2
Gerrit-Owner: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Tue, 01 Oct 2024 10:33:10 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
Posted by "Andrew Sherman (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/21867
to look at the new patch set (#2).
Change subject: IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
......................................................................
IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
The cluster_id flag was introduced in IMPALA-12426 to identify Impala
clusters in systems where a single query_log table could be shared. In
IMPALA-13208 the cluster_id flag was reused as a prefix to topic names
for backend membership, to allow sub-clusters of backends within a
Statestore service.
There have been some problems with the interaction of these two usages.
An important difference is that the query_log cluster_id must be set
only on coordinators, whereas the topic prefix cluster_id must be set
simultaneously on coordinators, executors, and admission daemons
(if present). If a system is started with cluster_id set only on
coordinators then there are split-brain problems where coordinators and
executors are tracked in different topics. In addition, the query_log
cluster_id is more likely to be user-settable as it is used for data in
query_log which will be read by humans, who may want to write queries
selecting data from their ‘production’ or ‘dev’ clusters.
Avoid these problems by using a separate flag for the topic prefix
cluster_id ‘cluster_membership_topic_id’.
Change-Id: Icd3f7e1c73c00a7aaeee79ecb461209e3939c422
---
M be/src/runtime/exec-env.cc
M be/src/scheduling/admission-controller.cc
M be/src/scheduling/admissiond-env.cc
M be/src/scheduling/cluster-membership-mgr.cc
M tests/custom_cluster/test_query_live.py
M tests/custom_cluster/test_shared_catalogd.py
6 files changed, 23 insertions(+), 15 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/67/21867/2
--
To view, visit http://gerrit.cloudera.org:8080/21867
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Icd3f7e1c73c00a7aaeee79ecb461209e3939c422
Gerrit-Change-Number: 21867
Gerrit-PatchSet: 2
Gerrit-Owner: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
Posted by "Jason Fehr (Code Review)" <ge...@cloudera.org>.
Jason Fehr has posted comments on this change. ( http://gerrit.cloudera.org:8080/21867 )
Change subject: IMPALA-13408: use a specific flag for the topic prefix cluster identifier.
......................................................................
Patch Set 2: Code-Review+2
(2 comments)
http://gerrit.cloudera.org:8080/#/c/21867/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/21867/2//COMMIT_MSG@16
PS2, Line 16: An important difference is that the query_log cluster_id must be set
> I guess I don't understand how "cluster_id must be set only on coordinators
There is nothing stopping any arbitrary flag from being set on any daemon. If that particular daemon does not use a flag, then it is ignored.
The issue we are trying to prevent is a split brain problem when cluster_id does not get specified everywhere. It's a difficult situation to detect and diagnose. Another potential issue is the users specifying the same cluster_id across multiple clusters which would result in cross-communication between two clusters.
TL;DR -- 'cluster_id' flag is meaningful to humans while 'cluster_membership_topic_id' flag is meaningful to cluster coordination.
http://gerrit.cloudera.org:8080/#/c/21867/2/be/src/scheduling/admission-controller.cc
File be/src/scheduling/admission-controller.cc:
http://gerrit.cloudera.org:8080/#/c/21867/2/be/src/scheduling/admission-controller.cc@82
PS2, Line 82: cluster_id
> It still exists, defined in workload-management-flags.cc
'cluster_id' flag is meaningful to humans and used in workload management while 'cluster_membership_topic_id' flag is meaningful to cluster coordination.
--
To view, visit http://gerrit.cloudera.org:8080/21867
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd3f7e1c73c00a7aaeee79ecb461209e3939c422
Gerrit-Change-Number: 21867
Gerrit-PatchSet: 2
Gerrit-Owner: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 02 Oct 2024 22:20:36 +0000
Gerrit-HasComments: Yes