You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Sahil Takiar (Code Review)" <ge...@cloudera.org> on 2020/08/10 17:14:11 UTC
[Impala-ASF-CR] IMPALA-10016: Split jars for Impala exec and coord Docker images
Sahil Takiar has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16320
Change subject: IMPALA-10016: Split jars for Impala exec and coord Docker images
......................................................................
IMPALA-10016: Split jars for Impala exec and coord Docker images
Maven Changes:
Splits out all executor specific jar files into a separate pom file
under mvn-deps/executor-deps. The new pom file lists out all executor
specific jar files. fe/pom.xml has a dependency on
mvn-deps/executor-deps/pom.xml so that all executor specific jars are
still built as part of the fe/ build. mvn-deps/executor-deps/pom.xml
writes out a build-classpath.txt file that contains all dependencies in
the pom.xml file (similar to what is already done in fe/pom.xml).
Docker Build Changes:
setup_build_context.py was changed to leverage the aformentioned Maven
changes. The script still copies all dependencies into the lib/ folder,
but also creates an exec-lib/ and statestore-lib/ folder. The exec-lib/
folder contains all dependencies necessary to run Impala Executors, but
excludes any dependencies that are Coordinator specific. The
statestore-lib/ folder excludes all jar files entirely since it does not
run an embedded JVM.
The docker/CMakeLists.txt was modified to support the new library layout
created by setup_build_context.py. Prior to this patch only the build
for the Impala base image has access to the dependencies created by
setup_build_context.py. This patch changes the build logic so all images
have access to the dependencies. This does increase build time because
the built context has to be copied and sent to the Docker daemon for
each image build.
Docker Image Changes:
The copy command for the lib/ folder was removed from the impala_base
Dockerfile and a corresponding copy command was added to each daemon
Docker image. This only each daemon image to only copy in the
dependencies it actually requires to run.
Other:
* Deleted the hive-3 profile since Impala 4.0 only supports hive-3 builds
* Moved shaded-deps into the mvn-deps folder
Overall, this decreases the size of the impalad_executor image by 70 MB,
and the statestored image by 700 MB.
Further improvements might be possible by decreasing the number of
transitive dependencies in mvn-deps/executor-deps/pom.xml. Moreover,
any new Coordinator specific jar files will not be included in the
Executor image.
Testing:
* Ran core tests
Change-Id: I899859a38d8ccab890de889a49ef132a89289dfd
---
M CMakeLists.txt
M docker/CMakeLists.txt
M docker/catalogd/Dockerfile
M docker/impala_base/Dockerfile
M docker/impalad_coord_exec/Dockerfile
M docker/impalad_coordinator/Dockerfile
M docker/impalad_executor/Dockerfile
M docker/setup_build_context.py
M docker/statestored/Dockerfile
M fe/CMakeLists.txt
M fe/pom.xml
A mvn-deps/executor-deps/CMakeLists.txt
A mvn-deps/executor-deps/pom.xml
R mvn-deps/shaded-deps/.gitignore
R mvn-deps/shaded-deps/CMakeLists.txt
R mvn-deps/shaded-deps/pom.xml
16 files changed, 414 insertions(+), 270 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/20/16320/1
--
To view, visit http://gerrit.cloudera.org:8080/16320
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I899859a38d8ccab890de889a49ef132a89289dfd
Gerrit-Change-Number: 16320
Gerrit-PatchSet: 1
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
[Impala-ASF-CR] IMPALA-10016: Split jars for Impala exec and coord Docker images
Posted by "Sahil Takiar (Code Review)" <ge...@cloudera.org>.
Hello Tim Armstrong, Joe McDonnell, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/16320
to look at the new patch set (#6).
Change subject: IMPALA-10016: Split jars for Impala exec and coord Docker images
......................................................................
IMPALA-10016: Split jars for Impala exec and coord Docker images
Maven Changes:
Splits out all executor specific jar files into a separate pom file
under mvn-deps/executor-deps. The new pom file lists out all executor
specific jar files. fe/pom.xml has a dependency on
mvn-deps/executor-deps/pom.xml so that all executor specific jars are
still built as part of the fe/ build. mvn-deps/executor-deps/pom.xml
writes out a build-classpath.txt file that contains all dependencies in
the pom.xml file (similar to what is already done in fe/pom.xml).
Docker Build Changes:
setup_build_context.py was changed to leverage the aformentioned Maven
changes. The script still symlinks all dependencies into the lib/ folder,
but also creates an exec-lib/ and statestore-lib/ folder. The exec-lib/
folder contains all dependencies necessary to run Impala Executors, but
excludes any dependencies that are Coordinator specific. The
statestore-lib/ folder excludes all jar files entirely since it does not
run an embedded JVM.
The docker/CMakeLists.txt was modified to support the new library layout
created by setup_build_context.py. Prior to this patch only the build
for the Impala base image has access to the dependencies created by
setup_build_context.py. This patch changes the build logic so all images
have access to the dependencies. This does increase build time because
the built context has to be copied and sent to the Docker daemon for
each image build.
Docker Image Changes:
The copy command for the lib/ folder was removed from the impala_base
Dockerfile and a corresponding copy command was added to each daemon
Docker image. This allows each daemon image to only copy in the
dependencies it actually requires to run.
Other:
* Deleted the hive-3 profile since Impala 4.0 only supports hive-3 builds
* Moved shaded-deps into the mvn-deps folder
Overall, this decreases the size of the impalad_executor image by 120 MB,
and the statestored image by 700 MB.
impalad_coordinator and impalad_coordinator images are now 771 MB, and
impalad_executor images are 651MB.
Further improvements might be possible by decreasing the number of
transitive dependencies in mvn-deps/executor-deps/pom.xml. Moreover,
any new Coordinator specific jar files will not be included in the
Executor image.
Testing:
* Ran core tests
Change-Id: I899859a38d8ccab890de889a49ef132a89289dfd
---
M CMakeLists.txt
M docker/CMakeLists.txt
M docker/catalogd/Dockerfile
M docker/impala_base/Dockerfile
M docker/impalad_coord_exec/Dockerfile
M docker/impalad_coordinator/Dockerfile
M docker/impalad_executor/Dockerfile
M docker/setup_build_context.py
M docker/statestored/Dockerfile
M fe/CMakeLists.txt
M fe/pom.xml
A mvn-deps/executor-deps/CMakeLists.txt
A mvn-deps/executor-deps/pom.xml
R mvn-deps/shaded-deps/hive-exec/CMakeLists.txt
R mvn-deps/shaded-deps/hive-exec/pom.xml
R mvn-deps/shaded-deps/s3a-aws-sdk/CMakeLists.txt
R mvn-deps/shaded-deps/s3a-aws-sdk/pom.xml
17 files changed, 398 insertions(+), 251 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/20/16320/6
--
To view, visit http://gerrit.cloudera.org:8080/16320
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I899859a38d8ccab890de889a49ef132a89289dfd
Gerrit-Change-Number: 16320
Gerrit-PatchSet: 6
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-10016: Split jars for Impala exec and coord Docker images
Posted by "Sahil Takiar (Code Review)" <ge...@cloudera.org>.
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/16320 )
Change subject: IMPALA-10016: Split jars for Impala exec and coord Docker images
......................................................................
Patch Set 6:
> > (1 comment)
> >
> > The code changes basically look good. Let me know how testing
> goes,
> > I can approve it then.
>
> Is there a change that we could make to the dockerized tests that
> would help us test the coordinator-only and executor-only images?
> At the moment, the dockerized tests use the impalad_coord_exec
> docker image, and that has been fine for coverage because
> impalad_coord_exec, impalad_executor, and impalad_coordinator have
> been so similar. I wonder how much work it would be to migrate to
> using one impalad_coordinator and three impalad_executor nodes (or
> one impalad_coord_exec and two impalad_executor nodes). Would this
> be a useful direction?
Filed IMPALA-10191 for this.
--
To view, visit http://gerrit.cloudera.org:8080/16320
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I899859a38d8ccab890de889a49ef132a89289dfd
Gerrit-Change-Number: 16320
Gerrit-PatchSet: 6
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Thu, 24 Sep 2020 23:54:11 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10016: Split jars for Impala exec and coord Docker images
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16320 )
Change subject: IMPALA-10016: Split jars for Impala exec and coord Docker images
......................................................................
Patch Set 5:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/7281/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16320
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I899859a38d8ccab890de889a49ef132a89289dfd
Gerrit-Change-Number: 16320
Gerrit-PatchSet: 5
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Fri, 25 Sep 2020 00:06:19 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10016: Split jars for Impala exec and coord Docker images
Posted by "Sahil Takiar (Code Review)" <ge...@cloudera.org>.
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/16320 )
Change subject: IMPALA-10016: Split jars for Impala exec and coord Docker images
......................................................................
Patch Set 9:
* Rebased
* Fixed some bugs from a previous rebase
* For some reason statestored needs kudu libs, so added those back in
--
To view, visit http://gerrit.cloudera.org:8080/16320
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I899859a38d8ccab890de889a49ef132a89289dfd
Gerrit-Change-Number: 16320
Gerrit-PatchSet: 9
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Thu, 08 Oct 2020 16:30:27 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10016: Split jars for Impala exec and coord Docker images
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16320 )
Change subject: IMPALA-10016: Split jars for Impala exec and coord Docker images
......................................................................
Patch Set 2:
(1 comment)
The code changes basically look good. Let me know how testing goes, I can approve it then.
http://gerrit.cloudera.org:8080/#/c/16320/2/docker/impalad_coord_exec/Dockerfile
File docker/impalad_coord_exec/Dockerfile:
PS2:
> I removed the Dockerfile for the impalad_coord_exec and replaced it with a
Is docker smart enough to deduplicate these by caching the previous build?
I think we might need to override the flags in start-impala-cluster.py change since I think the script assumes that the is_coordinator and is_executor flags are both set in the container.
TBH I think setting the flags in the dockerfiles may have been a mistake, since most frameworks, like k8s, ignores the command line anyway.
--
To view, visit http://gerrit.cloudera.org:8080/16320
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I899859a38d8ccab890de889a49ef132a89289dfd
Gerrit-Change-Number: 16320
Gerrit-PatchSet: 2
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Sat, 29 Aug 2020 00:28:13 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10016: Split jars for Impala exec and coord Docker images
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16320 )
Change subject: IMPALA-10016: Split jars for Impala exec and coord Docker images
......................................................................
Patch Set 7:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6502/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/16320
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I899859a38d8ccab890de889a49ef132a89289dfd
Gerrit-Change-Number: 16320
Gerrit-PatchSet: 7
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Fri, 02 Oct 2020 14:09:45 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10016: Split jars for Impala exec and coord Docker images
Posted by "Sahil Takiar (Code Review)" <ge...@cloudera.org>.
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/16320 )
Change subject: IMPALA-10016: Split jars for Impala exec and coord Docker images
......................................................................
Patch Set 4:
> Did you think at all about combining the coordinator and catalog base images, since they would need basically the same jars. And they might get scheduled on the same node so could benefit significantly from sharing the base image.
Are they always scheduled on the same node? I thought that the catalogd gets scheduled a lighter-weight node and coordinators get scheduled on heavier-weight nodes (e.g. executor nodes).
I was thinking IMPALA-10068 as the long-term way to decrease the size of the catalogd images.
--
To view, visit http://gerrit.cloudera.org:8080/16320
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I899859a38d8ccab890de889a49ef132a89289dfd
Gerrit-Change-Number: 16320
Gerrit-PatchSet: 4
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Fri, 28 Aug 2020 19:32:10 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10016: Split jars for Impala exec and coord Docker images
Posted by "Sahil Takiar (Code Review)" <ge...@cloudera.org>.
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/16320 )
Change subject: IMPALA-10016: Split jars for Impala exec and coord Docker images
......................................................................
Patch Set 9: Code-Review+2
Carrying +2 from Tim.
--
To view, visit http://gerrit.cloudera.org:8080/16320
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I899859a38d8ccab890de889a49ef132a89289dfd
Gerrit-Change-Number: 16320
Gerrit-PatchSet: 9
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Thu, 08 Oct 2020 23:11:51 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10016: Split jars for Impala exec and coord Docker images
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16320 )
Change subject: IMPALA-10016: Split jars for Impala exec and coord Docker images
......................................................................
Patch Set 2:
(2 comments)
Did you think at all about combining the coordinator and catalog base images, since they would need basically the same jars. And they might get scheduled on the same node so could benefit significantly from sharing the base image.
http://gerrit.cloudera.org:8080/#/c/16320/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/16320/2//COMMIT_MSG@58
PS2, Line 58: * Ran core tests
How confident are we that that precommit tests will catch missing jars from the executor?
http://gerrit.cloudera.org:8080/#/c/16320/2/docker/impalad_coord_exec/Dockerfile
File docker/impalad_coord_exec/Dockerfile:
PS2:
We could probably just make this a copy of the coordinator container since it has the same requirements.
TBH we could get rid of the coord_exec container since I don't think it adds much over the coordinator.
--
To view, visit http://gerrit.cloudera.org:8080/16320
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I899859a38d8ccab890de889a49ef132a89289dfd
Gerrit-Change-Number: 16320
Gerrit-PatchSet: 2
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 11 Aug 2020 19:05:09 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10016: Split jars for Impala exec and coord Docker images
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16320 )
Change subject: IMPALA-10016: Split jars for Impala exec and coord Docker images
......................................................................
Patch Set 4:
We're kinda agnostic about how the catalog and coordinator would get scheduled - it could make sense to pack them into the same node - or let k8s decide to do that. It probably doesn't matter that much in the scheme of things since there's only going to be 1 catalogd vs N executors per deployment.
--
To view, visit http://gerrit.cloudera.org:8080/16320
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I899859a38d8ccab890de889a49ef132a89289dfd
Gerrit-Change-Number: 16320
Gerrit-PatchSet: 4
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Sat, 29 Aug 2020 00:24:18 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10016: Split jars for Impala exec and coord Docker images
Posted by "Sahil Takiar (Code Review)" <ge...@cloudera.org>.
Hello Tim Armstrong, Joe McDonnell, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/16320
to look at the new patch set (#8).
Change subject: IMPALA-10016: Split jars for Impala exec and coord Docker images
......................................................................
IMPALA-10016: Split jars for Impala exec and coord Docker images
Maven Changes:
Splits out all executor specific jar files into a separate pom file
under mvn-deps/executor-deps. The new pom file lists out all executor
specific jar files. fe/pom.xml has a dependency on
mvn-deps/executor-deps/pom.xml so that all executor specific jars are
still built as part of the fe/ build. mvn-deps/executor-deps/pom.xml
writes out a build-classpath.txt file that contains all dependencies in
the pom.xml file (similar to what is already done in fe/pom.xml).
Docker Build Changes:
setup_build_context.py was changed to leverage the aformentioned Maven
changes. The script still symlinks all dependencies into the lib/ folder,
but also creates an exec-lib/ and statestore-lib/ folder. The exec-lib/
folder contains all dependencies necessary to run Impala Executors, but
excludes any dependencies that are Coordinator specific. The
statestore-lib/ folder excludes all jar files entirely since it does not
run an embedded JVM.
The docker/CMakeLists.txt was modified to support the new library layout
created by setup_build_context.py. Prior to this patch only the build
for the Impala base image has access to the dependencies created by
setup_build_context.py. This patch changes the build logic so all images
have access to the dependencies. This does increase build time because
the built context has to be copied and sent to the Docker daemon for
each image build.
Docker Image Changes:
The copy command for the lib/ folder was removed from the impala_base
Dockerfile and a corresponding copy command was added to each daemon
Docker image. This allows each daemon image to only copy in the
dependencies it actually requires to run.
Other:
* Deleted the hive-3 profile since Impala 4.0 only supports hive-3 builds
* Moved shaded-deps into the mvn-deps folder
Overall, this decreases the size of the impalad_executor image by 120 MB,
and the statestored image by 700 MB.
impalad_coordinator and impalad_coordinator images are now 771 MB, and
impalad_executor images are 651MB.
Further improvements might be possible by decreasing the number of
transitive dependencies in mvn-deps/executor-deps/pom.xml. Moreover,
any new Coordinator specific jar files will not be included in the
Executor image.
Testing:
* Ran core tests
Change-Id: I899859a38d8ccab890de889a49ef132a89289dfd
---
M CMakeLists.txt
M docker/CMakeLists.txt
M docker/catalogd/Dockerfile
M docker/impala_base/Dockerfile
M docker/impalad_coord_exec/Dockerfile
M docker/impalad_coordinator/Dockerfile
M docker/impalad_executor/Dockerfile
M docker/setup_build_context.py
M docker/statestored/Dockerfile
M fe/CMakeLists.txt
M fe/pom.xml
A mvn-deps/executor-deps/CMakeLists.txt
A mvn-deps/executor-deps/pom.xml
R mvn-deps/shaded-deps/hive-exec/CMakeLists.txt
R mvn-deps/shaded-deps/hive-exec/pom.xml
R mvn-deps/shaded-deps/s3a-aws-sdk/CMakeLists.txt
R mvn-deps/shaded-deps/s3a-aws-sdk/pom.xml
17 files changed, 441 insertions(+), 251 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/20/16320/8
--
To view, visit http://gerrit.cloudera.org:8080/16320
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I899859a38d8ccab890de889a49ef132a89289dfd
Gerrit-Change-Number: 16320
Gerrit-PatchSet: 8
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-10016: Split jars for Impala exec and coord Docker images
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16320 )
Change subject: IMPALA-10016: Split jars for Impala exec and coord Docker images
......................................................................
Patch Set 7: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6502/
--
To view, visit http://gerrit.cloudera.org:8080/16320
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I899859a38d8ccab890de889a49ef132a89289dfd
Gerrit-Change-Number: 16320
Gerrit-PatchSet: 7
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Fri, 02 Oct 2020 19:34:09 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10016: Split jars for Impala exec and coord Docker images
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16320 )
Change subject: IMPALA-10016: Split jars for Impala exec and coord Docker images
......................................................................
Patch Set 9:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/7385/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16320
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I899859a38d8ccab890de889a49ef132a89289dfd
Gerrit-Change-Number: 16320
Gerrit-PatchSet: 9
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Thu, 08 Oct 2020 16:48:09 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10016: Split jars for Impala exec and coord Docker images
Posted by "Sahil Takiar (Code Review)" <ge...@cloudera.org>.
Sahil Takiar has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16320 )
Change subject: IMPALA-10016: Split jars for Impala exec and coord Docker images
......................................................................
IMPALA-10016: Split jars for Impala exec and coord Docker images
Maven Changes:
Splits out all executor specific jar files into a separate pom file
under mvn-deps/executor-deps. The new pom file lists out all executor
specific jar files. fe/pom.xml has a dependency on
mvn-deps/executor-deps/pom.xml so that all executor specific jars are
still built as part of the fe/ build. mvn-deps/executor-deps/pom.xml
writes out a build-classpath.txt file that contains all dependencies in
the pom.xml file (similar to what is already done in fe/pom.xml).
Docker Build Changes:
setup_build_context.py was changed to leverage the aformentioned Maven
changes. The script still symlinks all dependencies into the lib/ folder,
but also creates an exec-lib/ and statestore-lib/ folder. The exec-lib/
folder contains all dependencies necessary to run Impala Executors, but
excludes any dependencies that are Coordinator specific. The
statestore-lib/ folder excludes all jar files entirely since it does not
run an embedded JVM.
The docker/CMakeLists.txt was modified to support the new library layout
created by setup_build_context.py. Prior to this patch only the build
for the Impala base image has access to the dependencies created by
setup_build_context.py. This patch changes the build logic so all images
have access to the dependencies. This does increase build time because
the built context has to be copied and sent to the Docker daemon for
each image build.
Docker Image Changes:
The copy command for the lib/ folder was removed from the impala_base
Dockerfile and a corresponding copy command was added to each daemon
Docker image. This allows each daemon image to only copy in the
dependencies it actually requires to run.
Other:
* Deleted the hive-3 profile since Impala 4.0 only supports hive-3 builds
* Moved shaded-deps into the mvn-deps folder
Overall, this decreases the size of the impalad_executor image by 120 MB,
and the statestored image by 700 MB.
impalad_coordinator and impalad_coordinator images are now 771 MB, and
impalad_executor images are 651MB.
Further improvements might be possible by decreasing the number of
transitive dependencies in mvn-deps/executor-deps/pom.xml. Moreover,
any new Coordinator specific jar files will not be included in the
Executor image.
Testing:
* Ran core tests
Change-Id: I899859a38d8ccab890de889a49ef132a89289dfd
Reviewed-on: http://gerrit.cloudera.org:8080/16320
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Sahil Takiar <st...@cloudera.com>
---
M CMakeLists.txt
M docker/CMakeLists.txt
M docker/catalogd/Dockerfile
M docker/impala_base/Dockerfile
M docker/impalad_coord_exec/Dockerfile
M docker/impalad_coordinator/Dockerfile
M docker/impalad_executor/Dockerfile
M docker/setup_build_context.py
M docker/statestored/Dockerfile
M fe/CMakeLists.txt
M fe/pom.xml
A mvn-deps/executor-deps/CMakeLists.txt
A mvn-deps/executor-deps/pom.xml
R mvn-deps/shaded-deps/hive-exec/CMakeLists.txt
R mvn-deps/shaded-deps/hive-exec/pom.xml
R mvn-deps/shaded-deps/s3a-aws-sdk/CMakeLists.txt
R mvn-deps/shaded-deps/s3a-aws-sdk/pom.xml
17 files changed, 441 insertions(+), 296 deletions(-)
Approvals:
Impala Public Jenkins: Verified
Sahil Takiar: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/16320
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I899859a38d8ccab890de889a49ef132a89289dfd
Gerrit-Change-Number: 16320
Gerrit-PatchSet: 10
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-10016: Split jars for Impala exec and coord Docker images
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16320 )
Change subject: IMPALA-10016: Split jars for Impala exec and coord Docker images
......................................................................
Patch Set 6: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/16320
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I899859a38d8ccab890de889a49ef132a89289dfd
Gerrit-Change-Number: 16320
Gerrit-PatchSet: 6
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Fri, 02 Oct 2020 05:43:52 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10016: Split jars for Impala exec and coord Docker images
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16320 )
Change subject: IMPALA-10016: Split jars for Impala exec and coord Docker images
......................................................................
Patch Set 3:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/7036/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/16320
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I899859a38d8ccab890de889a49ef132a89289dfd
Gerrit-Change-Number: 16320
Gerrit-PatchSet: 3
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Fri, 28 Aug 2020 17:58:52 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10016: Split jars for Impala exec and coord Docker images
Posted by "Sahil Takiar (Code Review)" <ge...@cloudera.org>.
Hello Tim Armstrong, Joe McDonnell, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/16320
to look at the new patch set (#5).
Change subject: IMPALA-10016: Split jars for Impala exec and coord Docker images
......................................................................
IMPALA-10016: Split jars for Impala exec and coord Docker images
Maven Changes:
Splits out all executor specific jar files into a separate pom file
under mvn-deps/executor-deps. The new pom file lists out all executor
specific jar files. fe/pom.xml has a dependency on
mvn-deps/executor-deps/pom.xml so that all executor specific jars are
still built as part of the fe/ build. mvn-deps/executor-deps/pom.xml
writes out a build-classpath.txt file that contains all dependencies in
the pom.xml file (similar to what is already done in fe/pom.xml).
Docker Build Changes:
setup_build_context.py was changed to leverage the aformentioned Maven
changes. The script still symlinks all dependencies into the lib/ folder,
but also creates an exec-lib/ and statestore-lib/ folder. The exec-lib/
folder contains all dependencies necessary to run Impala Executors, but
excludes any dependencies that are Coordinator specific. The
statestore-lib/ folder excludes all jar files entirely since it does not
run an embedded JVM.
The docker/CMakeLists.txt was modified to support the new library layout
created by setup_build_context.py. Prior to this patch only the build
for the Impala base image has access to the dependencies created by
setup_build_context.py. This patch changes the build logic so all images
have access to the dependencies. This does increase build time because
the built context has to be copied and sent to the Docker daemon for
each image build.
Docker Image Changes:
The copy command for the lib/ folder was removed from the impala_base
Dockerfile and a corresponding copy command was added to each daemon
Docker image. This allows each daemon image to only copy in the
dependencies it actually requires to run.
Other:
* Deleted the hive-3 profile since Impala 4.0 only supports hive-3 builds
* Moved shaded-deps into the mvn-deps folder
Overall, this decreases the size of the impalad_executor image by 120 MB,
and the statestored image by 700 MB.
impalad_coordinator and impalad_coordinator images are now 771 MB, and
impalad_executor images are 651MB.
Further improvements might be possible by decreasing the number of
transitive dependencies in mvn-deps/executor-deps/pom.xml. Moreover,
any new Coordinator specific jar files will not be included in the
Executor image.
Testing:
* Ran core tests
Change-Id: I899859a38d8ccab890de889a49ef132a89289dfd
---
M CMakeLists.txt
M docker/CMakeLists.txt
M docker/catalogd/Dockerfile
M docker/impala_base/Dockerfile
M docker/impalad_coord_exec/Dockerfile
M docker/impalad_coordinator/Dockerfile
M docker/impalad_executor/Dockerfile
M docker/setup_build_context.py
M docker/statestored/Dockerfile
M fe/CMakeLists.txt
M fe/pom.xml
A mvn-deps/executor-deps/CMakeLists.txt
A mvn-deps/executor-deps/pom.xml
R mvn-deps/shaded-deps/hive-exec/CMakeLists.txt
R mvn-deps/shaded-deps/hive-exec/pom.xml
R mvn-deps/shaded-deps/s3a-aws-sdk/CMakeLists.txt
R mvn-deps/shaded-deps/s3a-aws-sdk/pom.xml
17 files changed, 398 insertions(+), 251 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/20/16320/5
--
To view, visit http://gerrit.cloudera.org:8080/16320
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I899859a38d8ccab890de889a49ef132a89289dfd
Gerrit-Change-Number: 16320
Gerrit-PatchSet: 5
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-10016: Split jars for Impala exec and coord Docker images
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16320 )
Change subject: IMPALA-10016: Split jars for Impala exec and coord Docker images
......................................................................
Patch Set 4:
I had thought about doing that split in the past - it seems like it would be useful. I don't see any obvious issues with doing it aside from tests making assumptions about scheduling.
--
To view, visit http://gerrit.cloudera.org:8080/16320
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I899859a38d8ccab890de889a49ef132a89289dfd
Gerrit-Change-Number: 16320
Gerrit-PatchSet: 4
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Mon, 31 Aug 2020 21:02:53 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10016: Split jars for Impala exec and coord Docker images
Posted by "Sahil Takiar (Code Review)" <ge...@cloudera.org>.
Hello Tim Armstrong, Joe McDonnell, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/16320
to look at the new patch set (#4).
Change subject: IMPALA-10016: Split jars for Impala exec and coord Docker images
......................................................................
IMPALA-10016: Split jars for Impala exec and coord Docker images
Maven Changes:
Splits out all executor specific jar files into a separate pom file
under mvn-deps/executor-deps. The new pom file lists out all executor
specific jar files. fe/pom.xml has a dependency on
mvn-deps/executor-deps/pom.xml so that all executor specific jars are
still built as part of the fe/ build. mvn-deps/executor-deps/pom.xml
writes out a build-classpath.txt file that contains all dependencies in
the pom.xml file (similar to what is already done in fe/pom.xml).
Docker Build Changes:
setup_build_context.py was changed to leverage the aformentioned Maven
changes. The script still copies all dependencies into the lib/ folder,
but also creates an exec-lib/ and statestore-lib/ folder. The exec-lib/
folder contains all dependencies necessary to run Impala Executors, but
excludes any dependencies that are Coordinator specific. The
statestore-lib/ folder excludes all jar files entirely since it does not
run an embedded JVM.
The docker/CMakeLists.txt was modified to support the new library layout
created by setup_build_context.py. Prior to this patch only the build
for the Impala base image has access to the dependencies created by
setup_build_context.py. This patch changes the build logic so all images
have access to the dependencies. This does increase build time because
the built context has to be copied and sent to the Docker daemon for
each image build.
Docker Image Changes:
The copy command for the lib/ folder was removed from the impala_base
Dockerfile and a corresponding copy command was added to each daemon
Docker image. This only each daemon image to only copy in the
dependencies it actually requires to run.
Other:
* Deleted the hive-3 profile since Impala 4.0 only supports hive-3 builds
* Moved shaded-deps into the mvn-deps folder
Overall, this decreases the size of the impalad_executor image by 70 MB,
and the statestored image by 700 MB.
Further improvements might be possible by decreasing the number of
transitive dependencies in mvn-deps/executor-deps/pom.xml. Moreover,
any new Coordinator specific jar files will not be included in the
Executor image.
Testing:
* Ran core tests
Change-Id: I899859a38d8ccab890de889a49ef132a89289dfd
---
M CMakeLists.txt
M bin/rat_exclude_files.txt
M docker/CMakeLists.txt
M docker/README.md
M docker/catalogd/Dockerfile
M docker/impala_base/Dockerfile
D docker/impalad_coord_exec/Dockerfile
A docker/impalad_coord_exec/Dockerfile
M docker/impalad_coordinator/Dockerfile
M docker/impalad_executor/Dockerfile
M docker/setup_build_context.py
M docker/statestored/Dockerfile
M fe/CMakeLists.txt
M fe/pom.xml
A mvn-deps/executor-deps/CMakeLists.txt
A mvn-deps/executor-deps/pom.xml
R mvn-deps/shaded-deps/.gitignore
R mvn-deps/shaded-deps/CMakeLists.txt
R mvn-deps/shaded-deps/pom.xml
19 files changed, 419 insertions(+), 306 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/20/16320/4
--
To view, visit http://gerrit.cloudera.org:8080/16320
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I899859a38d8ccab890de889a49ef132a89289dfd
Gerrit-Change-Number: 16320
Gerrit-PatchSet: 4
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-10016: Split jars for Impala exec and coord Docker images
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16320 )
Change subject: IMPALA-10016: Split jars for Impala exec and coord Docker images
......................................................................
Patch Set 9:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6530/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/16320
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I899859a38d8ccab890de889a49ef132a89289dfd
Gerrit-Change-Number: 16320
Gerrit-PatchSet: 9
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Thu, 08 Oct 2020 16:30:39 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10016: Split jars for Impala exec and coord Docker images
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16320 )
Change subject: IMPALA-10016: Split jars for Impala exec and coord Docker images
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/6857/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16320
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I899859a38d8ccab890de889a49ef132a89289dfd
Gerrit-Change-Number: 16320
Gerrit-PatchSet: 1
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 10 Aug 2020 17:35:29 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10016: Split jars for Impala exec and coord Docker images
Posted by "Sahil Takiar (Code Review)" <ge...@cloudera.org>.
Hello Tim Armstrong, Joe McDonnell, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/16320
to look at the new patch set (#3).
Change subject: IMPALA-10016: Split jars for Impala exec and coord Docker images
......................................................................
IMPALA-10016: Split jars for Impala exec and coord Docker images
Maven Changes:
Splits out all executor specific jar files into a separate pom file
under mvn-deps/executor-deps. The new pom file lists out all executor
specific jar files. fe/pom.xml has a dependency on
mvn-deps/executor-deps/pom.xml so that all executor specific jars are
still built as part of the fe/ build. mvn-deps/executor-deps/pom.xml
writes out a build-classpath.txt file that contains all dependencies in
the pom.xml file (similar to what is already done in fe/pom.xml).
Docker Build Changes:
setup_build_context.py was changed to leverage the aformentioned Maven
changes. The script still copies all dependencies into the lib/ folder,
but also creates an exec-lib/ and statestore-lib/ folder. The exec-lib/
folder contains all dependencies necessary to run Impala Executors, but
excludes any dependencies that are Coordinator specific. The
statestore-lib/ folder excludes all jar files entirely since it does not
run an embedded JVM.
The docker/CMakeLists.txt was modified to support the new library layout
created by setup_build_context.py. Prior to this patch only the build
for the Impala base image has access to the dependencies created by
setup_build_context.py. This patch changes the build logic so all images
have access to the dependencies. This does increase build time because
the built context has to be copied and sent to the Docker daemon for
each image build.
Docker Image Changes:
The copy command for the lib/ folder was removed from the impala_base
Dockerfile and a corresponding copy command was added to each daemon
Docker image. This only each daemon image to only copy in the
dependencies it actually requires to run.
Other:
* Deleted the hive-3 profile since Impala 4.0 only supports hive-3 builds
* Moved shaded-deps into the mvn-deps folder
Overall, this decreases the size of the impalad_executor image by 70 MB,
and the statestored image by 700 MB.
Further improvements might be possible by decreasing the number of
transitive dependencies in mvn-deps/executor-deps/pom.xml. Moreover,
any new Coordinator specific jar files will not be included in the
Executor image.
Testing:
* Ran core tests
Change-Id: I899859a38d8ccab890de889a49ef132a89289dfd
---
M CMakeLists.txt
M docker/CMakeLists.txt
M docker/README.md
M docker/catalogd/Dockerfile
M docker/impala_base/Dockerfile
D docker/impalad_coord_exec/Dockerfile
A docker/impalad_coord_exec/Dockerfile
M docker/impalad_coordinator/Dockerfile
M docker/impalad_executor/Dockerfile
M docker/setup_build_context.py
M docker/statestored/Dockerfile
M fe/CMakeLists.txt
M fe/pom.xml
A mvn-deps/executor-deps/CMakeLists.txt
A mvn-deps/executor-deps/pom.xml
R mvn-deps/shaded-deps/.gitignore
R mvn-deps/shaded-deps/CMakeLists.txt
R mvn-deps/shaded-deps/pom.xml
18 files changed, 418 insertions(+), 306 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/20/16320/3
--
To view, visit http://gerrit.cloudera.org:8080/16320
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I899859a38d8ccab890de889a49ef132a89289dfd
Gerrit-Change-Number: 16320
Gerrit-PatchSet: 3
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-10016: Split jars for Impala exec and coord Docker images
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16320 )
Change subject: IMPALA-10016: Split jars for Impala exec and coord Docker images
......................................................................
Patch Set 9: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/16320
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I899859a38d8ccab890de889a49ef132a89289dfd
Gerrit-Change-Number: 16320
Gerrit-PatchSet: 9
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Thu, 08 Oct 2020 21:56:02 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10016: Split jars for Impala exec and coord Docker images
Posted by "Sahil Takiar (Code Review)" <ge...@cloudera.org>.
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/16320 )
Change subject: IMPALA-10016: Split jars for Impala exec and coord Docker images
......................................................................
Patch Set 6:
Rebased and resolved a bunch of conflicts. I had to rework some of the changes I made to the 'docker build' commands because they were breaking during CI/CD tests.
I ran CI/CD tests that build the Docker images + run some cluster-level tests.
I backed out some of the changes I made to the impalad_coord_exec Dockerfile just because this patch is already pretty complex, and I don't want to tackle too much at once. I filed IMPALA-10190 and IMPALA-10191 as follow ups.
--
To view, visit http://gerrit.cloudera.org:8080/16320
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I899859a38d8ccab890de889a49ef132a89289dfd
Gerrit-Change-Number: 16320
Gerrit-PatchSet: 6
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Thu, 24 Sep 2020 23:53:27 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10016: Split jars for Impala exec and coord Docker images
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16320 )
Change subject: IMPALA-10016: Split jars for Impala exec and coord Docker images
......................................................................
Patch Set 4:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/7038/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16320
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I899859a38d8ccab890de889a49ef132a89289dfd
Gerrit-Change-Number: 16320
Gerrit-PatchSet: 4
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Fri, 28 Aug 2020 19:51:22 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10016: Split jars for Impala exec and coord Docker images
Posted by "Sahil Takiar (Code Review)" <ge...@cloudera.org>.
Hello Tim Armstrong, Joe McDonnell, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/16320
to look at the new patch set (#2).
Change subject: IMPALA-10016: Split jars for Impala exec and coord Docker images
......................................................................
IMPALA-10016: Split jars for Impala exec and coord Docker images
Maven Changes:
Splits out all executor specific jar files into a separate pom file
under mvn-deps/executor-deps. The new pom file lists out all executor
specific jar files. fe/pom.xml has a dependency on
mvn-deps/executor-deps/pom.xml so that all executor specific jars are
still built as part of the fe/ build. mvn-deps/executor-deps/pom.xml
writes out a build-classpath.txt file that contains all dependencies in
the pom.xml file (similar to what is already done in fe/pom.xml).
Docker Build Changes:
setup_build_context.py was changed to leverage the aformentioned Maven
changes. The script still copies all dependencies into the lib/ folder,
but also creates an exec-lib/ and statestore-lib/ folder. The exec-lib/
folder contains all dependencies necessary to run Impala Executors, but
excludes any dependencies that are Coordinator specific. The
statestore-lib/ folder excludes all jar files entirely since it does not
run an embedded JVM.
The docker/CMakeLists.txt was modified to support the new library layout
created by setup_build_context.py. Prior to this patch only the build
for the Impala base image has access to the dependencies created by
setup_build_context.py. This patch changes the build logic so all images
have access to the dependencies. This does increase build time because
the built context has to be copied and sent to the Docker daemon for
each image build.
Docker Image Changes:
The copy command for the lib/ folder was removed from the impala_base
Dockerfile and a corresponding copy command was added to each daemon
Docker image. This only each daemon image to only copy in the
dependencies it actually requires to run.
Other:
* Deleted the hive-3 profile since Impala 4.0 only supports hive-3 builds
* Moved shaded-deps into the mvn-deps folder
Overall, this decreases the size of the impalad_executor image by 70 MB,
and the statestored image by 700 MB.
Further improvements might be possible by decreasing the number of
transitive dependencies in mvn-deps/executor-deps/pom.xml. Moreover,
any new Coordinator specific jar files will not be included in the
Executor image.
Testing:
* Ran core tests
Change-Id: I899859a38d8ccab890de889a49ef132a89289dfd
---
M CMakeLists.txt
M docker/CMakeLists.txt
M docker/catalogd/Dockerfile
M docker/impala_base/Dockerfile
M docker/impalad_coord_exec/Dockerfile
M docker/impalad_coordinator/Dockerfile
M docker/impalad_executor/Dockerfile
M docker/setup_build_context.py
M docker/statestored/Dockerfile
M fe/CMakeLists.txt
M fe/pom.xml
A mvn-deps/executor-deps/CMakeLists.txt
A mvn-deps/executor-deps/pom.xml
R mvn-deps/shaded-deps/.gitignore
R mvn-deps/shaded-deps/CMakeLists.txt
R mvn-deps/shaded-deps/pom.xml
16 files changed, 414 insertions(+), 269 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/20/16320/2
--
To view, visit http://gerrit.cloudera.org:8080/16320
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I899859a38d8ccab890de889a49ef132a89289dfd
Gerrit-Change-Number: 16320
Gerrit-PatchSet: 2
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-10016: Split jars for Impala exec and coord Docker images
Posted by "Sahil Takiar (Code Review)" <ge...@cloudera.org>.
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/16320 )
Change subject: IMPALA-10016: Split jars for Impala exec and coord Docker images
......................................................................
Patch Set 3:
(2 comments)
Rebased the patch as well since it conflicted with IMPALA-10029
http://gerrit.cloudera.org:8080/#/c/16320/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/16320/2//COMMIT_MSG@58
PS2, Line 58: * Ran core tests
> How confident are we that that precommit tests will catch missing jars from
Fair point. I'll do some more extensive testing. I will probably batch all the changes under IMPALA-10028 and test them together.
http://gerrit.cloudera.org:8080/#/c/16320/2/docker/impalad_coord_exec/Dockerfile
File docker/impalad_coord_exec/Dockerfile:
PS2:
> We could probably just make this a copy of the coordinator container since
I removed the Dockerfile for the impalad_coord_exec and replaced it with a symlink to the impalad_coordinator.
--
To view, visit http://gerrit.cloudera.org:8080/16320
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I899859a38d8ccab890de889a49ef132a89289dfd
Gerrit-Change-Number: 16320
Gerrit-PatchSet: 3
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Fri, 28 Aug 2020 17:44:01 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10016: Split jars for Impala exec and coord Docker images
Posted by "Sahil Takiar (Code Review)" <ge...@cloudera.org>.
Hello Tim Armstrong, Joe McDonnell, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/16320
to look at the new patch set (#9).
Change subject: IMPALA-10016: Split jars for Impala exec and coord Docker images
......................................................................
IMPALA-10016: Split jars for Impala exec and coord Docker images
Maven Changes:
Splits out all executor specific jar files into a separate pom file
under mvn-deps/executor-deps. The new pom file lists out all executor
specific jar files. fe/pom.xml has a dependency on
mvn-deps/executor-deps/pom.xml so that all executor specific jars are
still built as part of the fe/ build. mvn-deps/executor-deps/pom.xml
writes out a build-classpath.txt file that contains all dependencies in
the pom.xml file (similar to what is already done in fe/pom.xml).
Docker Build Changes:
setup_build_context.py was changed to leverage the aformentioned Maven
changes. The script still symlinks all dependencies into the lib/ folder,
but also creates an exec-lib/ and statestore-lib/ folder. The exec-lib/
folder contains all dependencies necessary to run Impala Executors, but
excludes any dependencies that are Coordinator specific. The
statestore-lib/ folder excludes all jar files entirely since it does not
run an embedded JVM.
The docker/CMakeLists.txt was modified to support the new library layout
created by setup_build_context.py. Prior to this patch only the build
for the Impala base image has access to the dependencies created by
setup_build_context.py. This patch changes the build logic so all images
have access to the dependencies. This does increase build time because
the built context has to be copied and sent to the Docker daemon for
each image build.
Docker Image Changes:
The copy command for the lib/ folder was removed from the impala_base
Dockerfile and a corresponding copy command was added to each daemon
Docker image. This allows each daemon image to only copy in the
dependencies it actually requires to run.
Other:
* Deleted the hive-3 profile since Impala 4.0 only supports hive-3 builds
* Moved shaded-deps into the mvn-deps folder
Overall, this decreases the size of the impalad_executor image by 120 MB,
and the statestored image by 700 MB.
impalad_coordinator and impalad_coordinator images are now 771 MB, and
impalad_executor images are 651MB.
Further improvements might be possible by decreasing the number of
transitive dependencies in mvn-deps/executor-deps/pom.xml. Moreover,
any new Coordinator specific jar files will not be included in the
Executor image.
Testing:
* Ran core tests
Change-Id: I899859a38d8ccab890de889a49ef132a89289dfd
---
M CMakeLists.txt
M docker/CMakeLists.txt
M docker/catalogd/Dockerfile
M docker/impala_base/Dockerfile
M docker/impalad_coord_exec/Dockerfile
M docker/impalad_coordinator/Dockerfile
M docker/impalad_executor/Dockerfile
M docker/setup_build_context.py
M docker/statestored/Dockerfile
M fe/CMakeLists.txt
M fe/pom.xml
A mvn-deps/executor-deps/CMakeLists.txt
A mvn-deps/executor-deps/pom.xml
R mvn-deps/shaded-deps/hive-exec/CMakeLists.txt
R mvn-deps/shaded-deps/hive-exec/pom.xml
R mvn-deps/shaded-deps/s3a-aws-sdk/CMakeLists.txt
R mvn-deps/shaded-deps/s3a-aws-sdk/pom.xml
17 files changed, 441 insertions(+), 296 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/20/16320/9
--
To view, visit http://gerrit.cloudera.org:8080/16320
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I899859a38d8ccab890de889a49ef132a89289dfd
Gerrit-Change-Number: 16320
Gerrit-PatchSet: 9
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-10016: Split jars for Impala exec and coord Docker images
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16320 )
Change subject: IMPALA-10016: Split jars for Impala exec and coord Docker images
......................................................................
Patch Set 8:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/7384/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16320
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I899859a38d8ccab890de889a49ef132a89289dfd
Gerrit-Change-Number: 16320
Gerrit-PatchSet: 8
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Thu, 08 Oct 2020 16:25:19 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10016: Split jars for Impala exec and coord Docker images
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/16320 )
Change subject: IMPALA-10016: Split jars for Impala exec and coord Docker images
......................................................................
Patch Set 4:
> (1 comment)
>
> The code changes basically look good. Let me know how testing goes,
> I can approve it then.
Is there a change that we could make to the dockerized tests that would help us test the coordinator-only and executor-only images? At the moment, the dockerized tests use the impalad_coord_exec docker image, and that has been fine for coverage because impalad_coord_exec, impalad_executor, and impalad_coordinator have been so similar. I wonder how much work it would be to migrate to using one impalad_coordinator and three impalad_executor nodes (or one impalad_coord_exec and two impalad_executor nodes). Would this be a useful direction?
--
To view, visit http://gerrit.cloudera.org:8080/16320
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I899859a38d8ccab890de889a49ef132a89289dfd
Gerrit-Change-Number: 16320
Gerrit-PatchSet: 4
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Mon, 31 Aug 2020 20:10:13 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10016: Split jars for Impala exec and coord Docker images
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16320 )
Change subject: IMPALA-10016: Split jars for Impala exec and coord Docker images
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/6882/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16320
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I899859a38d8ccab890de889a49ef132a89289dfd
Gerrit-Change-Number: 16320
Gerrit-PatchSet: 2
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 11 Aug 2020 17:58:51 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10016: Split jars for Impala exec and coord Docker images
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16320 )
Change subject: IMPALA-10016: Split jars for Impala exec and coord Docker images
......................................................................
Patch Set 6:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/7282/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16320
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I899859a38d8ccab890de889a49ef132a89289dfd
Gerrit-Change-Number: 16320
Gerrit-PatchSet: 6
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Fri, 25 Sep 2020 00:11:07 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10016: Split jars for Impala exec and coord Docker images
Posted by "Sahil Takiar (Code Review)" <ge...@cloudera.org>.
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/16320 )
Change subject: IMPALA-10016: Split jars for Impala exec and coord Docker images
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/16320/2/docker/impalad_coord_exec/Dockerfile
File docker/impalad_coord_exec/Dockerfile:
PS2:
> Is docker smart enough to deduplicate these by caching the previous build?
I decided to just keep this file as is (e.g. I reverted the change where I deleted this file and replaced it with a symlink). I filed IMPALA-10190 as a follow up.
--
To view, visit http://gerrit.cloudera.org:8080/16320
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I899859a38d8ccab890de889a49ef132a89289dfd
Gerrit-Change-Number: 16320
Gerrit-PatchSet: 2
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Thu, 24 Sep 2020 23:55:02 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10016: Split jars for Impala exec and coord Docker images
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16320 )
Change subject: IMPALA-10016: Split jars for Impala exec and coord Docker images
......................................................................
Patch Set 7: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/16320
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I899859a38d8ccab890de889a49ef132a89289dfd
Gerrit-Change-Number: 16320
Gerrit-PatchSet: 7
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Fri, 02 Oct 2020 14:09:44 +0000
Gerrit-HasComments: No