You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Thomas Tauber-Marshall (Code Review)" <ge...@cloudera.org> on 2020/12/21 19:30:57 UTC
[Impala-ASF-CR] IMPALA-9975 (part 2): Introduce new admission control daemon
Thomas Tauber-Marshall has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16891
Change subject: IMPALA-9975 (part 2): Introduce new admission control daemon
......................................................................
IMPALA-9975 (part 2): Introduce new admission control daemon
A recent patch (IMPALA-9930) introduces a new admission control rpc
service, which can be configured to perform admission control for
coordinators. In that patch, the admission service runs in an impalad.
This patch separates the service out to run in a new daemon, called
the admissiond.
Some notable changes:
- Adds a new class, AdmissiondEnv, which performs the same function
for the admissiond as ExecEnv does for impalads.
- The '/admission' http endpoint is exposed on the admissiond's webui
if the admission control service is in use, otherwise it is exposed
on coordinator impalad's webuis.
- Executors now never create an AdmissionController, and coordinators
do not create one if configured to use an admissiond. This requires
removing admission related info on the '/backends' endpoint and
completely removing the '/admission' endpoint for these daemons.
- start-impala-cluster.py takes a new flag --enable_admission_service
which configures the minicluster to have an admissiond with all
coordinators using it for admisison control.
TODO: integrate this with the Docker infrastructure
Testing:
- Existing tests for the admission control serivce are modified to run
with an admissiond.
Change-Id: Id677814b31e9193035e8cf0d08aba0ce388a0ad9
---
M CMakeLists.txt
M be/src/runtime/exec-env.cc
M be/src/runtime/exec-env.h
M be/src/scheduling/CMakeLists.txt
M be/src/scheduling/admission-control-client.cc
M be/src/scheduling/admission-control-service.cc
M be/src/scheduling/admission-control-service.h
A be/src/scheduling/admissiond-env.cc
A be/src/scheduling/admissiond-env.h
A be/src/scheduling/admissiond-main.cc
M be/src/service/CMakeLists.txt
M be/src/service/daemon-main.cc
M be/src/service/impala-http-handler.cc
M be/src/service/impala-http-handler.h
M be/src/service/impala-server.cc
M be/src/util/mem-info.cc
M be/src/util/mem-info.h
M bin/start-impala-cluster.py
M common/thrift/metrics.json
M tests/common/custom_cluster_test_suite.py
M tests/common/impala_cluster.py
M tests/common/impala_service.py
M tests/custom_cluster/test_admission_controller.py
M www/backends.tmpl
24 files changed, 574 insertions(+), 185 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/91/16891/1
--
To view, visit http://gerrit.cloudera.org:8080/16891
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Id677814b31e9193035e8cf0d08aba0ce388a0ad9
Gerrit-Change-Number: 16891
Gerrit-PatchSet: 1
Gerrit-Owner: Thomas Tauber-Marshall <tm...@cloudera.com>
[Impala-ASF-CR] IMPALA-9975 (part 2): Introduce new admission control daemon
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16891 )
Change subject: IMPALA-9975 (part 2): Introduce new admission control daemon
......................................................................
Patch Set 4:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6843/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/16891
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id677814b31e9193035e8cf0d08aba0ce388a0ad9
Gerrit-Change-Number: 16891
Gerrit-PatchSet: 4
Gerrit-Owner: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Wed, 13 Jan 2021 00:23:31 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9975 (part 2): Introduce new admission control daemon
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16891 )
Change subject: IMPALA-9975 (part 2): Introduce new admission control daemon
......................................................................
Patch Set 4: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/16891
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id677814b31e9193035e8cf0d08aba0ce388a0ad9
Gerrit-Change-Number: 16891
Gerrit-PatchSet: 4
Gerrit-Owner: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Wed, 13 Jan 2021 00:23:30 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9975 (part 2): Introduce new admission control daemon
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16891 )
Change subject: IMPALA-9975 (part 2): Introduce new admission control daemon
......................................................................
Patch Set 1:
(6 comments)
http://gerrit.cloudera.org:8080/#/c/16891/1/bin/start-impala-cluster.py
File bin/start-impala-cluster.py:
http://gerrit.cloudera.org:8080/#/c/16891/1/bin/start-impala-cluster.py@74
PS1, Line 74: t
flake8: E501 line too long (98 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/16891/1/bin/start-impala-cluster.py@293
PS1, Line 293: def build_admissiond_arg_list():
flake8: E302 expected 2 blank lines, found 1
http://gerrit.cloudera.org:8080/#/c/16891/1/tests/common/impala_cluster.py
File tests/common/impala_cluster.py:
http://gerrit.cloudera.org:8080/#/c/16891/1/tests/common/impala_cluster.py@106
PS1, Line 106: s
flake8: E501 line too long (102 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/16891/1/tests/common/impala_cluster.py@236
PS1, Line 236: i
flake8: E501 line too long (101 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/16891/1/tests/common/impala_cluster.py@566
PS1, Line 566: class AdmissiondProcess(BaseImpalaProcess):
flake8: E302 expected 2 blank lines, found 1
http://gerrit.cloudera.org:8080/#/c/16891/1/tests/common/impala_cluster.py@571
PS1, Line 571: #
flake8: E265 block comment should start with '# '
--
To view, visit http://gerrit.cloudera.org:8080/16891
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id677814b31e9193035e8cf0d08aba0ce388a0ad9
Gerrit-Change-Number: 16891
Gerrit-PatchSet: 1
Gerrit-Owner: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Mon, 21 Dec 2020 19:32:21 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9975 (part 2): Introduce new admission control daemon
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16891 )
Change subject: IMPALA-9975 (part 2): Introduce new admission control daemon
......................................................................
Patch Set 3:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6840/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/16891
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id677814b31e9193035e8cf0d08aba0ce388a0ad9
Gerrit-Change-Number: 16891
Gerrit-PatchSet: 3
Gerrit-Owner: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Fri, 08 Jan 2021 22:40:58 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9975 (part 2): Introduce new admission control daemon
Posted by "Thomas Tauber-Marshall (Code Review)" <ge...@cloudera.org>.
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/16891 )
Change subject: IMPALA-9975 (part 2): Introduce new admission control daemon
......................................................................
Patch Set 1:
(9 comments)
http://gerrit.cloudera.org:8080/#/c/16891/1/be/src/runtime/exec-env.cc
File be/src/runtime/exec-env.cc:
http://gerrit.cloudera.org:8080/#/c/16891/1/be/src/runtime/exec-env.cc@289
PS1, Line 289: if (FLAGS_is_coordinator && FLAGS_admission_control_service_addr.empty()) {
> IIRC the executors do actually need an AdmissionController to report the me
You're right.
There's a lot more thought that needs to go in to what actually needs to be sent to the statestore when the admission service is in use, and what changes can be made to the admission algorithm now that we're not relying as much on eventually consistent info about cluster load.
I think its best to leave this for follow up work, so I'll revert this to the original behavior. I filed IMPALA-10423 to track this
http://gerrit.cloudera.org:8080/#/c/16891/1/be/src/scheduling/admissiond-main.cc
File be/src/scheduling/admissiond-main.cc:
http://gerrit.cloudera.org:8080/#/c/16891/1/be/src/scheduling/admissiond-main.cc@44
PS1, Line 44: ABORT_IF_ERROR(daemon_env.Init(/* init_jvm */ false));
> Does the admission controller need to load the llama-site.xml and fair-sche
Done
http://gerrit.cloudera.org:8080/#/c/16891/1/bin/start-impala-cluster.py
File bin/start-impala-cluster.py:
http://gerrit.cloudera.org:8080/#/c/16891/1/bin/start-impala-cluster.py@74
PS1, Line 74: t
> flake8: E501 line too long (98 > 90 characters)
Done
http://gerrit.cloudera.org:8080/#/c/16891/1/bin/start-impala-cluster.py@293
PS1, Line 293: def build_admissiond_arg_list():
> flake8: E302 expected 2 blank lines, found 1
Done
http://gerrit.cloudera.org:8080/#/c/16891/1/bin/start-impala-cluster.py@413
PS1, Line 413: 127.0.0.1
> Maybe should be INTERNAL_LISTEN_HOST. Not sure it matters though.
Done.
This brought up the good point that as previously written admission_control_service_addr had to be a resolved IP address. I fixed that, and also changed it to just a hostname with the port taken from the admission_service_port flag, to be more consistent with how we handle this for the statestored and catalogd.
http://gerrit.cloudera.org:8080/#/c/16891/1/tests/common/impala_cluster.py
File tests/common/impala_cluster.py:
http://gerrit.cloudera.org:8080/#/c/16891/1/tests/common/impala_cluster.py@106
PS1, Line 106: s
> flake8: E501 line too long (102 > 90 characters)
Done
http://gerrit.cloudera.org:8080/#/c/16891/1/tests/common/impala_cluster.py@236
PS1, Line 236: i
> flake8: E501 line too long (101 > 90 characters)
Done
http://gerrit.cloudera.org:8080/#/c/16891/1/tests/common/impala_cluster.py@566
PS1, Line 566: class AdmissiondProcess(BaseImpalaProcess):
> flake8: E302 expected 2 blank lines, found 1
Done
http://gerrit.cloudera.org:8080/#/c/16891/1/tests/common/impala_cluster.py@571
PS1, Line 571: #
> flake8: E265 block comment should start with '# '
Done
--
To view, visit http://gerrit.cloudera.org:8080/16891
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id677814b31e9193035e8cf0d08aba0ce388a0ad9
Gerrit-Change-Number: 16891
Gerrit-PatchSet: 1
Gerrit-Owner: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Fri, 08 Jan 2021 22:18:04 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9975 (part 2): Introduce new admission control daemon
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16891 )
Change subject: IMPALA-9975 (part 2): Introduce new admission control daemon
......................................................................
Patch Set 3: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/16891
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id677814b31e9193035e8cf0d08aba0ce388a0ad9
Gerrit-Change-Number: 16891
Gerrit-PatchSet: 3
Gerrit-Owner: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Sat, 09 Jan 2021 04:27:17 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9975 (part 2): Introduce new admission control daemon
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16891 )
Change subject: IMPALA-9975 (part 2): Introduce new admission control daemon
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/7977/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16891
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id677814b31e9193035e8cf0d08aba0ce388a0ad9
Gerrit-Change-Number: 16891
Gerrit-PatchSet: 2
Gerrit-Owner: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Fri, 08 Jan 2021 22:38:03 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9975 (part 2): Introduce new admission control daemon
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16891 )
Change subject: IMPALA-9975 (part 2): Introduce new admission control daemon
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/7888/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16891
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id677814b31e9193035e8cf0d08aba0ce388a0ad9
Gerrit-Change-Number: 16891
Gerrit-PatchSet: 1
Gerrit-Owner: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Mon, 21 Dec 2020 19:52:59 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9975 (part 2): Introduce new admission control daemon
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16891 )
Change subject: IMPALA-9975 (part 2): Introduce new admission control daemon
......................................................................
Patch Set 1:
(3 comments)
Nice refactoring. I had a couple of high-level questions.
http://gerrit.cloudera.org:8080/#/c/16891/1/be/src/runtime/exec-env.cc
File be/src/runtime/exec-env.cc:
http://gerrit.cloudera.org:8080/#/c/16891/1/be/src/runtime/exec-env.cc@289
PS1, Line 289: if (FLAGS_is_coordinator && FLAGS_admission_control_service_addr.empty()) {
IIRC the executors do actually need an AdmissionController to report the memory usage from the pool MemTrackers. Looking at the code there's a lot of indirection, but basically UpdateMemTrackerStats() gets the memory usage from the pool mem trackers and that gets added to the topic update.
I think this is not as relevant as it once was, but it does play a role when we're running without memory limits or where there's significant untracked memory. So we want to preserve this behaviour at least in the standard config.
We could consider disabling this when the admission control daemon is in use - it's less relevant when we don't need to deal with distributed admission control.
http://gerrit.cloudera.org:8080/#/c/16891/1/be/src/scheduling/admissiond-main.cc
File be/src/scheduling/admissiond-main.cc:
http://gerrit.cloudera.org:8080/#/c/16891/1/be/src/scheduling/admissiond-main.cc@44
PS1, Line 44: ABORT_IF_ERROR(daemon_env.Init(/* init_jvm */ false));
Does the admission controller need to load the llama-site.xml and fair-scheduler.xml configs? Wondering if it might actually need a JVM for that purpose.
http://gerrit.cloudera.org:8080/#/c/16891/1/bin/start-impala-cluster.py
File bin/start-impala-cluster.py:
http://gerrit.cloudera.org:8080/#/c/16891/1/bin/start-impala-cluster.py@413
PS1, Line 413: 127.0.0.1
Maybe should be INTERNAL_LISTEN_HOST. Not sure it matters though.
--
To view, visit http://gerrit.cloudera.org:8080/16891
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id677814b31e9193035e8cf0d08aba0ce388a0ad9
Gerrit-Change-Number: 16891
Gerrit-PatchSet: 1
Gerrit-Owner: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 22 Dec 2020 18:32:07 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9975 (part 2): Introduce new admission control daemon
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16891 )
Change subject: IMPALA-9975 (part 2): Introduce new admission control daemon
......................................................................
Patch Set 3: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/16891
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id677814b31e9193035e8cf0d08aba0ce388a0ad9
Gerrit-Change-Number: 16891
Gerrit-PatchSet: 3
Gerrit-Owner: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 12 Jan 2021 18:01:26 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9975 (part 2): Introduce new admission control daemon
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16891 )
Change subject: IMPALA-9975 (part 2): Introduce new admission control daemon
......................................................................
Patch Set 4: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/16891
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id677814b31e9193035e8cf0d08aba0ce388a0ad9
Gerrit-Change-Number: 16891
Gerrit-PatchSet: 4
Gerrit-Owner: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Wed, 13 Jan 2021 06:03:36 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9975 (part 2): Introduce new admission control daemon
Posted by "Thomas Tauber-Marshall (Code Review)" <ge...@cloudera.org>.
Hello Tim Armstrong, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/16891
to look at the new patch set (#2).
Change subject: IMPALA-9975 (part 2): Introduce new admission control daemon
......................................................................
IMPALA-9975 (part 2): Introduce new admission control daemon
A recent patch (IMPALA-9930) introduces a new admission control rpc
service, which can be configured to perform admission control for
coordinators. In that patch, the admission service runs in an impalad.
This patch separates the service out to run in a new daemon, called
the admissiond. It also integrates this new daemon with the build
infrastructure around Docker.
Some notable changes:
- Adds a new class, AdmissiondEnv, which performs the same function
for the admissiond as ExecEnv does for impalads.
- The '/admission' http endpoint is exposed on the admissiond's webui
if the admission control service is in use, otherwise it is exposed
on coordinator impalad's webuis.
- start-impala-cluster.py takes a new flag --enable_admission_service
which configures the minicluster to have an admissiond with all
coordinators using it for admission control.
- Coordinators are now configured to use the admission service by
specifying the startup flag --admission_service_host. This is
intended to mirror the configuration of the statestored/catalogd
location.
Testing:
- Existing tests for the admission control serivce are modified to run
with an admissiond.
- Manually ran start-impala-cluster.py with --enable_admission_service
and --docker_network to verify Docker integration.
Change-Id: Id677814b31e9193035e8cf0d08aba0ce388a0ad9
---
M CMakeLists.txt
M be/src/runtime/exec-env.cc
M be/src/runtime/exec-env.h
M be/src/scheduling/CMakeLists.txt
M be/src/scheduling/admission-control-client.cc
M be/src/scheduling/admission-control-service.cc
M be/src/scheduling/admission-control-service.h
M be/src/scheduling/admission-controller.h
A be/src/scheduling/admissiond-env.cc
A be/src/scheduling/admissiond-env.h
A be/src/scheduling/admissiond-main.cc
M be/src/scheduling/remote-admission-control-client.cc
M be/src/scheduling/remote-admission-control-client.h
M be/src/service/CMakeLists.txt
M be/src/service/daemon-main.cc
M be/src/service/impala-http-handler.cc
M be/src/service/impala-http-handler.h
M be/src/service/impala-server.cc
M be/src/util/mem-info.cc
M be/src/util/mem-info.h
M bin/start-impala-cluster.py
M common/thrift/metrics.json
M docker/CMakeLists.txt
A docker/admissiond/Dockerfile
M docker/impala_base/Dockerfile
M tests/common/custom_cluster_test_suite.py
M tests/common/impala_cluster.py
M tests/common/impala_service.py
M tests/custom_cluster/test_admission_controller.py
29 files changed, 658 insertions(+), 206 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/91/16891/2
--
To view, visit http://gerrit.cloudera.org:8080/16891
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id677814b31e9193035e8cf0d08aba0ce388a0ad9
Gerrit-Change-Number: 16891
Gerrit-PatchSet: 2
Gerrit-Owner: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-9975 (part 2): Introduce new admission control daemon
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16891 )
Change subject: IMPALA-9975 (part 2): Introduce new admission control daemon
......................................................................
IMPALA-9975 (part 2): Introduce new admission control daemon
A recent patch (IMPALA-9930) introduces a new admission control rpc
service, which can be configured to perform admission control for
coordinators. In that patch, the admission service runs in an impalad.
This patch separates the service out to run in a new daemon, called
the admissiond. It also integrates this new daemon with the build
infrastructure around Docker.
Some notable changes:
- Adds a new class, AdmissiondEnv, which performs the same function
for the admissiond as ExecEnv does for impalads.
- The '/admission' http endpoint is exposed on the admissiond's webui
if the admission control service is in use, otherwise it is exposed
on coordinator impalad's webuis.
- start-impala-cluster.py takes a new flag --enable_admission_service
which configures the minicluster to have an admissiond with all
coordinators using it for admission control.
- Coordinators are now configured to use the admission service by
specifying the startup flag --admission_service_host. This is
intended to mirror the configuration of the statestored/catalogd
location.
Testing:
- Existing tests for the admission control serivce are modified to run
with an admissiond.
- Manually ran start-impala-cluster.py with --enable_admission_service
and --docker_network to verify Docker integration.
Change-Id: Id677814b31e9193035e8cf0d08aba0ce388a0ad9
Reviewed-on: http://gerrit.cloudera.org:8080/16891
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M CMakeLists.txt
M be/src/runtime/exec-env.cc
M be/src/runtime/exec-env.h
M be/src/scheduling/CMakeLists.txt
M be/src/scheduling/admission-control-client.cc
M be/src/scheduling/admission-control-service.cc
M be/src/scheduling/admission-control-service.h
M be/src/scheduling/admission-controller.h
A be/src/scheduling/admissiond-env.cc
A be/src/scheduling/admissiond-env.h
A be/src/scheduling/admissiond-main.cc
M be/src/scheduling/remote-admission-control-client.cc
M be/src/scheduling/remote-admission-control-client.h
M be/src/service/CMakeLists.txt
M be/src/service/daemon-main.cc
M be/src/service/impala-http-handler.cc
M be/src/service/impala-http-handler.h
M be/src/service/impala-server.cc
M be/src/util/mem-info.cc
M be/src/util/mem-info.h
M bin/start-impala-cluster.py
M common/thrift/metrics.json
M docker/CMakeLists.txt
A docker/admissiond/Dockerfile
M docker/impala_base/Dockerfile
M tests/common/custom_cluster_test_suite.py
M tests/common/impala_cluster.py
M tests/common/impala_service.py
M tests/custom_cluster/test_admission_controller.py
29 files changed, 658 insertions(+), 206 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/16891
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Id677814b31e9193035e8cf0d08aba0ce388a0ad9
Gerrit-Change-Number: 16891
Gerrit-PatchSet: 5
Gerrit-Owner: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>