You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org> on 2023/05/19 01:05:37 UTC
[Impala-ASF-CR] WIP IMPALA-12150: Allow executors to register to two statestores
Wenzhe Zhou has uploaded this change for review. ( http://gerrit.cloudera.org:8080/19905
Change subject: WIP IMPALA-12150: Allow executors to register to two statestores
......................................................................
WIP IMPALA-12150: Allow executors to register to two statestores
Impala assume single statestore instance in an Impala cluster and
each statestore subscriber can register to only one statestore
instance. Since cluster membership is maintained by statestore,
each executor and coordinator can join only one cluster.
This patch makes executors able to register to two statestore
instances so that we could allow the upgraded executors to join
two clusters during rolling upgrade.
In future, we will support statestore HA with active and standby
statestores in a cluster.
Testing:
- Passed the core tests.
- TODO unit-test
Change-Id: I5e2ff88a4a61279dea4be3432bb6460cbc33ef90
---
M be/src/catalog/catalog-server.cc
M be/src/runtime/exec-env.cc
M be/src/runtime/exec-env.h
M be/src/scheduling/admission-controller.cc
M be/src/scheduling/admission-controller.h
M be/src/scheduling/admissiond-env.cc
M be/src/scheduling/cluster-membership-mgr.cc
M be/src/scheduling/cluster-membership-mgr.h
M be/src/statestore/statestore-subscriber.cc
M be/src/statestore/statestore-subscriber.h
M be/src/statestore/statestore-test.cc
M be/src/statestore/statestore.cc
M be/src/statestore/statestore.h
M common/thrift/StatestoreService.thrift
M common/thrift/metrics.json
15 files changed, 595 insertions(+), 278 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/19905/1
--
To view, visit http://gerrit.cloudera.org:8080/19905
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I5e2ff88a4a61279dea4be3432bb6460cbc33ef90
Gerrit-Change-Number: 19905
Gerrit-PatchSet: 1
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] WIP IMPALA-12150: Allow executors to register to two statestores
Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has abandoned this change. ( http://gerrit.cloudera.org:8080/19905 )
Change subject: WIP IMPALA-12150: Allow executors to register to two statestores
......................................................................
Abandoned
design change
--
To view, visit http://gerrit.cloudera.org:8080/19905
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: abandon
Gerrit-Change-Id: I5e2ff88a4a61279dea4be3432bb6460cbc33ef90
Gerrit-Change-Number: 19905
Gerrit-PatchSet: 4
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
[Impala-ASF-CR] WIP IMPALA-12150: Allow executors to register to two statestores
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19905 )
Change subject: WIP IMPALA-12150: Allow executors to register to two statestores
......................................................................
Patch Set 2:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/13084/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/19905
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5e2ff88a4a61279dea4be3432bb6460cbc33ef90
Gerrit-Change-Number: 19905
Gerrit-PatchSet: 2
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Fri, 19 May 2023 01:35:10 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-12150: Allow executors to register to two statestores
Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/19905 )
Change subject: WIP IMPALA-12150: Allow executors to register to two statestores
......................................................................
WIP IMPALA-12150: Allow executors to register to two statestores
Impala assume single statestore instance in an Impala cluster and
each statestore subscriber can register to only one statestore
instance. Since cluster membership is maintained by statestore,
each executor and coordinator can join only one cluster.
This patch makes executors able to register to two statestore
instances so that we could allow the upgraded executors to join
two clusters during rolling upgrade.
In future, we will support statestore HA with active and standby
statestores in a cluster.
Testing:
- Passed the core tests.
- TODO unit-test
Change-Id: I5e2ff88a4a61279dea4be3432bb6460cbc33ef90
---
M be/src/catalog/catalog-server.cc
M be/src/runtime/exec-env.cc
M be/src/runtime/exec-env.h
M be/src/scheduling/admission-controller.cc
M be/src/scheduling/admission-controller.h
M be/src/scheduling/admissiond-env.cc
M be/src/scheduling/cluster-membership-mgr.cc
M be/src/scheduling/cluster-membership-mgr.h
M be/src/statestore/statestore-subscriber.cc
M be/src/statestore/statestore-subscriber.h
M be/src/statestore/statestore-test.cc
M be/src/statestore/statestore.cc
M be/src/statestore/statestore.h
M common/thrift/StatestoreService.thrift
M common/thrift/metrics.json
15 files changed, 623 insertions(+), 280 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/19905/3
--
To view, visit http://gerrit.cloudera.org:8080/19905
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I5e2ff88a4a61279dea4be3432bb6460cbc33ef90
Gerrit-Change-Number: 19905
Gerrit-PatchSet: 3
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
[Impala-ASF-CR] WIP IMPALA-12150: Allow executors to register to two statestores
Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/19905 )
Change subject: WIP IMPALA-12150: Allow executors to register to two statestores
......................................................................
WIP IMPALA-12150: Allow executors to register to two statestores
Impala assume single statestore instance in an Impala cluster and
each statestore subscriber can register to only one statestore
instance. Since cluster membership is maintained by statestore,
each executor and coordinator can join only one cluster.
This patch makes executors able to register to two statestore
instances so that we could allow the upgraded executors to join
two clusters during rolling upgrade.
In future, we will support statestore HA with active and standby
statestores in a cluster.
Testing:
- Passed the core tests.
- TODO unit-test
Change-Id: I5e2ff88a4a61279dea4be3432bb6460cbc33ef90
---
M be/src/catalog/catalog-server.cc
M be/src/runtime/exec-env.cc
M be/src/runtime/exec-env.h
M be/src/scheduling/admission-controller.cc
M be/src/scheduling/admission-controller.h
M be/src/scheduling/admissiond-env.cc
M be/src/scheduling/cluster-membership-mgr.cc
M be/src/scheduling/cluster-membership-mgr.h
M be/src/statestore/statestore-subscriber.cc
M be/src/statestore/statestore-subscriber.h
M be/src/statestore/statestore-test.cc
M be/src/statestore/statestore.cc
M be/src/statestore/statestore.h
M common/thrift/StatestoreService.thrift
M common/thrift/metrics.json
15 files changed, 595 insertions(+), 278 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/19905/2
--
To view, visit http://gerrit.cloudera.org:8080/19905
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I5e2ff88a4a61279dea4be3432bb6460cbc33ef90
Gerrit-Change-Number: 19905
Gerrit-PatchSet: 2
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
[Impala-ASF-CR] WIP IMPALA-12150: Allow executors to register to two statestores
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19905 )
Change subject: WIP IMPALA-12150: Allow executors to register to two statestores
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/13087/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/19905
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5e2ff88a4a61279dea4be3432bb6460cbc33ef90
Gerrit-Change-Number: 19905
Gerrit-PatchSet: 3
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Fri, 19 May 2023 19:23:27 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-12150: Allow executors to register to two statestores
Posted by "Wenzhe Zhou (Code Review)" <ge...@cloudera.org>.
Wenzhe Zhou has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/19905 )
Change subject: WIP IMPALA-12150: Allow executors to register to two statestores
......................................................................
WIP IMPALA-12150: Allow executors to register to two statestores
Impala assume single statestore instance in an Impala cluster and
each statestore subscriber can register to only one statestore
instance. Since cluster membership is maintained by statestore,
each executor and coordinator can join only one cluster.
This patch makes executors able to register to two statestore
instances so that we could allow the upgraded executors to join
two clusters during rolling upgrade.
In future, we will support statestore HA with active and standby
statestores in a cluster.
Testing:
- Passed the core tests.
- TODO unit-test
Change-Id: I5e2ff88a4a61279dea4be3432bb6460cbc33ef90
---
M be/src/catalog/catalog-server.cc
M be/src/runtime/exec-env.cc
M be/src/runtime/exec-env.h
M be/src/scheduling/admission-controller.cc
M be/src/scheduling/admission-controller.h
M be/src/scheduling/admissiond-env.cc
M be/src/scheduling/cluster-membership-mgr.cc
M be/src/scheduling/cluster-membership-mgr.h
M be/src/statestore/statestore-subscriber.cc
M be/src/statestore/statestore-subscriber.h
M be/src/statestore/statestore-test.cc
M be/src/statestore/statestore.cc
M be/src/statestore/statestore.h
M common/thrift/StatestoreService.thrift
M common/thrift/metrics.json
15 files changed, 624 insertions(+), 281 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/19905/4
--
To view, visit http://gerrit.cloudera.org:8080/19905
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I5e2ff88a4a61279dea4be3432bb6460cbc33ef90
Gerrit-Change-Number: 19905
Gerrit-PatchSet: 4
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
[Impala-ASF-CR] WIP IMPALA-12150: Allow executors to register to two statestores
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19905 )
Change subject: WIP IMPALA-12150: Allow executors to register to two statestores
......................................................................
Patch Set 1:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/19905/1/be/src/statestore/statestore-subscriber.h
File be/src/statestore/statestore-subscriber.h:
http://gerrit.cloudera.org:8080/#/c/19905/1/be/src/statestore/statestore-subscriber.h@264
PS1, Line 264: /// Returns OK if registration_id == registration_id_, or if registration_id_ is not yet
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/19905/1/be/src/statestore/statestore-subscriber.h@265
PS1, Line 265: /// set, an error otherwise. Used to confirm that RPCs from the statestore are intended
line too long (91 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/19905
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5e2ff88a4a61279dea4be3432bb6460cbc33ef90
Gerrit-Change-Number: 19905
Gerrit-PatchSet: 1
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 19 May 2023 01:06:34 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP IMPALA-12150: Allow executors to register to two statestores
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/19905 )
Change subject: WIP IMPALA-12150: Allow executors to register to two statestores
......................................................................
Patch Set 3:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/19905/3/common/thrift/StatestoreService.thrift
File common/thrift/StatestoreService.thrift:
http://gerrit.cloudera.org:8080/#/c/19905/3/common/thrift/StatestoreService.thrift@26
PS3, Line 26: // Change message formats to support 2 statestores
It seems the changes are backward compatible, i.e. just new optional fields added. Could you add commends on why we need a new version?
http://gerrit.cloudera.org:8080/#/c/19905/3/common/thrift/StatestoreService.thrift@236
PS3, Line 236: V1
Do we need this to be V2 as well?
--
To view, visit http://gerrit.cloudera.org:8080/19905
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5e2ff88a4a61279dea4be3432bb6460cbc33ef90
Gerrit-Change-Number: 19905
Gerrit-PatchSet: 3
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Mon, 22 May 2023 12:17:16 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP IMPALA-12150: Allow executors to register to two statestores
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19905 )
Change subject: WIP IMPALA-12150: Allow executors to register to two statestores
......................................................................
Patch Set 1:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/13083/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/19905
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5e2ff88a4a61279dea4be3432bb6460cbc33ef90
Gerrit-Change-Number: 19905
Gerrit-PatchSet: 1
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Fri, 19 May 2023 01:26:23 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-12150: Allow executors to register to two statestores
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/19905 )
Change subject: WIP IMPALA-12150: Allow executors to register to two statestores
......................................................................
Patch Set 4:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/13093/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/19905
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5e2ff88a4a61279dea4be3432bb6460cbc33ef90
Gerrit-Change-Number: 19905
Gerrit-PatchSet: 4
Gerrit-Owner: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Wenzhe Zhou <wz...@cloudera.com>
Gerrit-Comment-Date: Tue, 23 May 2023 01:06:28 +0000
Gerrit-HasComments: No