You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Grant Henke (Code Review)" <ge...@cloudera.org> on 2020/03/10 02:25:41 UTC
[kudu-CR] [docker] Add an Impala quickstart image
Grant Henke has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15393
Change subject: [docker] Add an Impala quickstart image
......................................................................
[docker] Add an Impala quickstart image
This patch adds an `impala` target that can be built and
published for use in quickstart examples. The image is a
single node Impala cluster with a Hive metastore backed
by Derby. This image is purpose build for quickstart usage
only and shouldn't be used for production or multi-node
deployments. The final image size is ~2GB uncompressed.
The build and runtime images reuse the dev and runtime base
images from Kudu given the additional libraries needed are
nominal in size and the reuse cuts back on code duplication.
This image will be used in an Impala quickstart guide that
will be added in a follow on patch.
Change-Id: I2d88bcb52f793c0fc83c4f84a9b56c5b961e58d5
---
M .dockerignore
M docker/Dockerfile
M docker/bootstrap-dev-env.sh
A docker/bootstrap-java-env.sh
A docker/bootstrap-maven-env.sh
M docker/bootstrap-runtime-env.sh
A docker/impala-entrypoint.sh
A docker/impala/etc/hadoop/conf/core-site.xml
A docker/impala/etc/hive/conf/hive-site.xml
9 files changed, 437 insertions(+), 26 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/93/15393/1
--
To view, visit http://gerrit.cloudera.org:8080/15393
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I2d88bcb52f793c0fc83c4f84a9b56c5b961e58d5
Gerrit-Change-Number: 15393
Gerrit-PatchSet: 1
Gerrit-Owner: Grant Henke <gr...@apache.org>
[kudu-CR] [docker] Add an Impala quickstart image
Posted by "Grant Henke (Code Review)" <ge...@cloudera.org>.
Hello Alexey Serbin, Kudu Jenkins, Andrew Wong,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15393
to look at the new patch set (#4).
Change subject: [docker] Add an Impala quickstart image
......................................................................
[docker] Add an Impala quickstart image
This patch adds an `impala` target that can be built and
published for use in quickstart examples. The image is a
single node Impala cluster with a Hive metastore backed
by Derby. This image is purpose build for quickstart usage
only and shouldn't be used for production or multi-node
deployments. The final image size is ~2GB uncompressed.
The build and runtime images reuse the dev and runtime base
images from Kudu given the additional libraries needed are
nominal in size and the reuse cuts back on code duplication.
This image will be used in an Impala quickstart guide that
will be added in a follow on patch.
Change-Id: I2d88bcb52f793c0fc83c4f84a9b56c5b961e58d5
---
M .dockerignore
M docker/Dockerfile
M docker/bootstrap-dev-env.sh
A docker/bootstrap-java-env.sh
A docker/bootstrap-maven-env.sh
M docker/bootstrap-runtime-env.sh
A docker/impala-entrypoint.sh
A docker/impala/etc/hadoop/conf/core-site.xml
A docker/impala/etc/hive/conf/hive-site.xml
9 files changed, 480 insertions(+), 26 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/93/15393/4
--
To view, visit http://gerrit.cloudera.org:8080/15393
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2d88bcb52f793c0fc83c4f84a9b56c5b961e58d5
Gerrit-Change-Number: 15393
Gerrit-PatchSet: 4
Gerrit-Owner: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [docker] Add an Impala quickstart image
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/15393 )
Change subject: [docker] Add an Impala quickstart image
......................................................................
Patch Set 4:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/15393/4/docker/impala-entrypoint.sh
File docker/impala-entrypoint.sh:
http://gerrit.cloudera.org:8080/#/c/15393/4/docker/impala-entrypoint.sh@24
PS4, Line 24: set -e
> I am not sure what could fail in a bad or unclear way. Especially consideri
From 'man bash':
pipefail
If set, the return value of a pipeline is the value of the
last (rightmost) command to exit with a non-zero status, or
zero if all commands in the pipeline exit successfully.
This option is disabled by default.
Try this with and without 'set -o pipefail'
-----
#!/bin/bash
set -e
#set -o pipefail
var=$(which ls | grep ls | sed "s:mega::")
echo "past the failure"
-----
--
To view, visit http://gerrit.cloudera.org:8080/15393
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2d88bcb52f793c0fc83c4f84a9b56c5b961e58d5
Gerrit-Change-Number: 15393
Gerrit-PatchSet: 4
Gerrit-Owner: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Thu, 12 Mar 2020 17:57:39 +0000
Gerrit-HasComments: Yes
[kudu-CR] [docker] Add an Impala quickstart image
Posted by "Grant Henke (Code Review)" <ge...@cloudera.org>.
Grant Henke has posted comments on this change. ( http://gerrit.cloudera.org:8080/15393 )
Change subject: [docker] Add an Impala quickstart image
......................................................................
Patch Set 4:
(15 comments)
http://gerrit.cloudera.org:8080/#/c/15393/4/docker/Dockerfile
File docker/Dockerfile:
http://gerrit.cloudera.org:8080/#/c/15393/4/docker/Dockerfile@356
PS4, Line 356: # Build Impala
: RUN source bin/impala-config.sh \
: && ./buildall.sh -release -noclean -notests \
: && docker/setup_build_context.py
> Is this able to succeed outside of the internal cloudera network? I rememb
This is pulling an Apache source release, so it should be able to complete outside of Cloudera. If it can't I will raise that with the Apache Impala community. In the mean time I can build and publish the image from within if there is an issue.
I also ran a full build on Centos 7 when testing this. I think Impala development is mean to be done on Ubuntu 16 but builds and packaging can be done on many OSs. There are a couple issues with `setup_build_context.py` and `daemon_entrypoint.sh` that I will address in a follow up patch.
http://gerrit.cloudera.org:8080/#/c/15393/4/docker/Dockerfile@474
PS4, Line 474:
> nit: will this extra spacing end up into a too-long space in the result ima
It does include the extra spaces. I will fixes all these.
http://gerrit.cloudera.org:8080/#/c/15393/4/docker/Dockerfile@478
PS4, Line 478: $MAINTAINER
> What if MAINTAINER's value contains spaces?
That is okay. Docker handles this correctly.
http://gerrit.cloudera.org:8080/#/c/15393/4/docker/Dockerfile@482
PS4, Line 482: $VCS_URL
> Does it work as expected if VCS_URL contains '#' or something like that? M
Yes this is just a text tag. Docker handles this correctly.
http://gerrit.cloudera.org:8080/#/c/15393/4/docker/bootstrap-java-env.sh
File docker/bootstrap-java-env.sh:
http://gerrit.cloudera.org:8080/#/c/15393/4/docker/bootstrap-java-env.sh@27
PS4, Line 27:
> Enable 'pipefail' option since to spot issues with VERSION_NAME early?
See my comment on the entrypoint script.
http://gerrit.cloudera.org:8080/#/c/15393/4/docker/bootstrap-java-env.sh@39
PS4, Line 39: /tmp/* /var/tmp/*
> I'm not sure this is safe: in /tmp and /var/tmp might be various files whic
This is a part of bootstrapping and building a docker image. It's not meant to run on anyones machine. This is fairly common practice in Docker scripts.
http://gerrit.cloudera.org:8080/#/c/15393/4/docker/bootstrap-maven-env.sh
File docker/bootstrap-maven-env.sh:
http://gerrit.cloudera.org:8080/#/c/15393/4/docker/bootstrap-maven-env.sh@29
PS4, Line 29: apache-maven-3.6.3-bin.tar.gz
> Create variable for this and re-use it everywhere in the script?
Done
http://gerrit.cloudera.org:8080/#/c/15393/4/docker/bootstrap-maven-env.sh@31
PS4, Line 31: z
> nit: 'z' isn't necessary here, can be omitted
Done
http://gerrit.cloudera.org:8080/#/c/15393/4/docker/bootstrap-maven-env.sh@32
PS4, Line 32: /usr/local/bin
> What if /usr/local/bin didn't exist prior to this invocation?
I haven't seen a docker base OS image where it doesn't yet. I would favor simplicity over defensiveness here given docker is meant for reproducible build.
http://gerrit.cloudera.org:8080/#/c/15393/4/docker/bootstrap-maven-env.sh@33
PS4, Line 33: rm
> rm -f ?
Done
http://gerrit.cloudera.org:8080/#/c/15393/4/docker/impala-entrypoint.sh
File docker/impala-entrypoint.sh:
http://gerrit.cloudera.org:8080/#/c/15393/4/docker/impala-entrypoint.sh@24
PS4, Line 24: set -e
> Maybe, set the 'pipefail' option as well to catch the issues with JAVA_HOME
I am not sure what could fail in a bad or unclear way. Especially considering this is a docker image where we control the environment. To maintain minimalism I think I would like to leave it out.
http://gerrit.cloudera.org:8080/#/c/15393/4/docker/impala-entrypoint.sh@46
PS4, Line 46: which
> Built the Impala environment with centos:7. Upon going through the quicksta
I fixed this, but there are a couple more issues in `daemon_entrypoint.sh` and `setup_build_context.py` that I will address in a follow on patch. For now, this should be okay given we will only publish the Ubuntu image.
http://gerrit.cloudera.org:8080/#/c/15393/4/docker/impala/etc/hadoop/conf/core-site.xml
File docker/impala/etc/hadoop/conf/core-site.xml:
http://gerrit.cloudera.org:8080/#/c/15393/4/docker/impala/etc/hadoop/conf/core-site.xml@26
PS4, Line 26: <name>dfs.client.read.shortcircuit</name>
: <value>true</value>
> Could you add a comment explaining the reason for this customization?
I removed it since it may not be needed.
http://gerrit.cloudera.org:8080/#/c/15393/4/docker/impala/etc/hadoop/conf/core-site.xml@29
PS4, Line 29: <property>
: <name>hadoop.proxyuser.root.hosts</name>
: <value>*</value>
: </property>
: <property>
: <name>hadoop.proxyuser.root.groups</name>
: <value>*</value>
: </property>
: <property>
: <name>hadoop.proxyuser.impala.hosts</name>
: <value>*</value>
: </property>
: <property>
: <name>hadoop.proxyuser.impala.groups</name>
: <value>*</value>
: </property>
> Do you mind adding a small comment to explain why the defaults are not good
Done
http://gerrit.cloudera.org:8080/#/c/15393/4/docker/impala/etc/hive/conf/hive-site.xml
File docker/impala/etc/hive/conf/hive-site.xml:
http://gerrit.cloudera.org:8080/#/c/15393/4/docker/impala/etc/hive/conf/hive-site.xml@38
PS4, Line 38: <name>hive.metastore.schema.verification</name>
: <value>false</value>
> nit: do you mind adding a comment to explain why it's crucial to have this
It was left over from when I was using auto schema generation. I will remove it.
--
To view, visit http://gerrit.cloudera.org:8080/15393
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2d88bcb52f793c0fc83c4f84a9b56c5b961e58d5
Gerrit-Change-Number: 15393
Gerrit-PatchSet: 4
Gerrit-Owner: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Thu, 12 Mar 2020 14:01:12 +0000
Gerrit-HasComments: Yes
[kudu-CR] [docker] Add an Impala quickstart image
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15393 )
Change subject: [docker] Add an Impala quickstart image
......................................................................
Patch Set 4: -Code-Review
(1 comment)
http://gerrit.cloudera.org:8080/#/c/15393/4/docker/impala-entrypoint.sh
File docker/impala-entrypoint.sh:
http://gerrit.cloudera.org:8080/#/c/15393/4/docker/impala-entrypoint.sh@46
PS4, Line 46: which
Built the Impala environment with centos:7. Upon going through the quickstart, I ran into:
/impala-entrypoint.sh: line 46: which: command not found
readlink: missing operand
Try 'readlink --help' for more information.
Supported commands:
impala - Start a single node, Kudu only, Impala cluster
impala-shell - Start the Impala shell CLI
help - print useful information and exit
Other commands can be specified to run shell commands.
Environment variables:
KUDU_MASTERS:
Defines the kudu-master configured master addresses.
Defaults to kudu-master-1:7051,kudu-master-2:7151,kudu-master-3:7251
--
To view, visit http://gerrit.cloudera.org:8080/15393
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2d88bcb52f793c0fc83c4f84a9b56c5b961e58d5
Gerrit-Change-Number: 15393
Gerrit-PatchSet: 4
Gerrit-Owner: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Wed, 11 Mar 2020 23:21:19 +0000
Gerrit-HasComments: Yes
[kudu-CR] [docker] Add an Impala quickstart image
Posted by "Grant Henke (Code Review)" <ge...@cloudera.org>.
Grant Henke has posted comments on this change. ( http://gerrit.cloudera.org:8080/15393 )
Change subject: [docker] Add an Impala quickstart image
......................................................................
Patch Set 6:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/15393/4/docker/impala-entrypoint.sh
File docker/impala-entrypoint.sh:
http://gerrit.cloudera.org:8080/#/c/15393/4/docker/impala-entrypoint.sh@24
PS4, Line 24: set -xe
> From 'man bash':
I added pipefail to all the scripts.
http://gerrit.cloudera.org:8080/#/c/15393/4/docker/impala-entrypoint.sh@46
PS4, Line 46: which
> Cool. Maybe note in the commit message that because Impala's scripts are ta
Done
--
To view, visit http://gerrit.cloudera.org:8080/15393
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2d88bcb52f793c0fc83c4f84a9b56c5b961e58d5
Gerrit-Change-Number: 15393
Gerrit-PatchSet: 6
Gerrit-Owner: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 13 Mar 2020 17:35:43 +0000
Gerrit-HasComments: Yes
[kudu-CR] [docker] Add an Impala quickstart image
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15393 )
Change subject: [docker] Add an Impala quickstart image
......................................................................
Patch Set 4: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/15393
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2d88bcb52f793c0fc83c4f84a9b56c5b961e58d5
Gerrit-Change-Number: 15393
Gerrit-PatchSet: 4
Gerrit-Owner: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Wed, 11 Mar 2020 20:37:27 +0000
Gerrit-HasComments: No
[kudu-CR] [docker] Add an Impala quickstart image
Posted by "Grant Henke (Code Review)" <ge...@cloudera.org>.
Hello Alexey Serbin, Kudu Jenkins, Andrew Wong,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15393
to look at the new patch set (#6).
Change subject: [docker] Add an Impala quickstart image
......................................................................
[docker] Add an Impala quickstart image
This patch adds an `impala` target that can be built and
published for use in quickstart examples. The image is a
single node Impala cluster with a Hive metastore backed
by Derby. This image is purpose build for quickstart usage
only and shouldn't be used for production or multi-node
deployments. The final image size is ~2GB uncompressed.
The build and runtime images reuse the dev and runtime base
images from Kudu given the additional libraries needed are
nominal in size and the reuse cuts back on code duplication.
This image will be used in an Impala quickstart guide that
will be added in a follow on patch.
Currently the image only works on Ubuntu distributions due
to Impala side script assumptions. Follow on patches will
fix these issues for other distributions.
Change-Id: I2d88bcb52f793c0fc83c4f84a9b56c5b961e58d5
---
M .dockerignore
M docker/Dockerfile
M docker/bootstrap-dev-env.sh
A docker/bootstrap-java-env.sh
A docker/bootstrap-maven-env.sh
M docker/bootstrap-python-env.sh
M docker/bootstrap-runtime-env.sh
M docker/docker-build.sh
A docker/impala-entrypoint.sh
A docker/impala/etc/hadoop/conf/core-site.xml
A docker/impala/etc/hive/conf/hive-site.xml
M docker/kudu-entrypoint.sh
12 files changed, 491 insertions(+), 33 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/93/15393/6
--
To view, visit http://gerrit.cloudera.org:8080/15393
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2d88bcb52f793c0fc83c4f84a9b56c5b961e58d5
Gerrit-Change-Number: 15393
Gerrit-PatchSet: 6
Gerrit-Owner: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [docker] Add an Impala quickstart image
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15393 )
Change subject: [docker] Add an Impala quickstart image
......................................................................
Patch Set 5:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/15393/4/docker/impala-entrypoint.sh
File docker/impala-entrypoint.sh:
http://gerrit.cloudera.org:8080/#/c/15393/4/docker/impala-entrypoint.sh@46
PS4, Line 46: which
> I fixed this, but there are a couple more issues in `daemon_entrypoint.sh`
Cool. Maybe note in the commit message that because Impala's scripts are tailored for Ubuntu, this may not work with other distros?
--
To view, visit http://gerrit.cloudera.org:8080/15393
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2d88bcb52f793c0fc83c4f84a9b56c5b961e58d5
Gerrit-Change-Number: 15393
Gerrit-PatchSet: 5
Gerrit-Owner: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Thu, 12 Mar 2020 21:10:03 +0000
Gerrit-HasComments: Yes
[kudu-CR] [docker] Add an Impala quickstart image
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/15393 )
Change subject: [docker] Add an Impala quickstart image
......................................................................
Patch Set 4:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/15393/4/docker/Dockerfile
File docker/Dockerfile:
http://gerrit.cloudera.org:8080/#/c/15393/4/docker/Dockerfile@356
PS4, Line 356: # Build Impala
: RUN source bin/impala-config.sh \
: && ./buildall.sh -release -noclean -notests \
: && docker/setup_build_context.py
Is this able to succeed outside of the internal cloudera network? I remember there were some issues with that some time ago. Are they gone at this point?
Also, I've heard it's not possible to build Impala out-of-the-box anywhere except for Ubuntu 16.08. Is that still an issue? If yes, maybe add a verification step for the distro and the version to bail out earlier if it's not the desired Ubuntu 16.08
http://gerrit.cloudera.org:8080/#/c/15393/4/docker/Dockerfile@474
PS4, Line 474:
nit: will this extra spacing end up into a too-long space in the result image's metadata?
http://gerrit.cloudera.org:8080/#/c/15393/4/docker/Dockerfile@478
PS4, Line 478: $MAINTAINER
What if MAINTAINER's value contains spaces?
http://gerrit.cloudera.org:8080/#/c/15393/4/docker/Dockerfile@482
PS4, Line 482: $VCS_URL
Does it work as expected if VCS_URL contains '#' or something like that? Maybe, wrap this into double quotes?
--
To view, visit http://gerrit.cloudera.org:8080/15393
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2d88bcb52f793c0fc83c4f84a9b56c5b961e58d5
Gerrit-Change-Number: 15393
Gerrit-PatchSet: 4
Gerrit-Owner: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Wed, 11 Mar 2020 22:13:29 +0000
Gerrit-HasComments: Yes
[kudu-CR] [docker] Add an Impala quickstart image
Posted by "Grant Henke (Code Review)" <ge...@cloudera.org>.
Grant Henke has posted comments on this change. ( http://gerrit.cloudera.org:8080/15393 )
Change subject: [docker] Add an Impala quickstart image
......................................................................
Patch Set 3: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/15393
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2d88bcb52f793c0fc83c4f84a9b56c5b961e58d5
Gerrit-Change-Number: 15393
Gerrit-PatchSet: 3
Gerrit-Owner: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Wed, 11 Mar 2020 02:19:29 +0000
Gerrit-HasComments: No
[kudu-CR] [docker] Add an Impala quickstart image
Posted by "Grant Henke (Code Review)" <ge...@cloudera.org>.
Grant Henke has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15393 )
Change subject: [docker] Add an Impala quickstart image
......................................................................
[docker] Add an Impala quickstart image
This patch adds an `impala` target that can be built and
published for use in quickstart examples. The image is a
single node Impala cluster with a Hive metastore backed
by Derby. This image is purpose build for quickstart usage
only and shouldn't be used for production or multi-node
deployments. The final image size is ~2GB uncompressed.
The build and runtime images reuse the dev and runtime base
images from Kudu given the additional libraries needed are
nominal in size and the reuse cuts back on code duplication.
This image will be used in an Impala quickstart guide that
will be added in a follow on patch.
Currently the image only works on Ubuntu distributions due
to Impala side script assumptions. Follow on patches will
fix these issues for other distributions.
Change-Id: I2d88bcb52f793c0fc83c4f84a9b56c5b961e58d5
Reviewed-on: http://gerrit.cloudera.org:8080/15393
Tested-by: Grant Henke <gr...@apache.org>
Reviewed-by: Andrew Wong <aw...@cloudera.com>
Reviewed-by: Alexey Serbin <as...@cloudera.com>
Tested-by: Kudu Jenkins
---
M .dockerignore
M docker/Dockerfile
M docker/bootstrap-dev-env.sh
A docker/bootstrap-java-env.sh
A docker/bootstrap-maven-env.sh
M docker/bootstrap-python-env.sh
M docker/bootstrap-runtime-env.sh
M docker/docker-build.sh
A docker/impala-entrypoint.sh
A docker/impala/etc/hadoop/conf/core-site.xml
A docker/impala/etc/hive/conf/hive-site.xml
M docker/kudu-entrypoint.sh
12 files changed, 493 insertions(+), 33 deletions(-)
Approvals:
Grant Henke: Verified
Andrew Wong: Looks good to me, approved
Alexey Serbin: Looks good to me, approved
Kudu Jenkins: Verified
--
To view, visit http://gerrit.cloudera.org:8080/15393
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I2d88bcb52f793c0fc83c4f84a9b56c5b961e58d5
Gerrit-Change-Number: 15393
Gerrit-PatchSet: 8
Gerrit-Owner: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [docker] Add an Impala quickstart image
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15393 )
Change subject: [docker] Add an Impala quickstart image
......................................................................
Patch Set 6: Code-Review+1
(1 comment)
http://gerrit.cloudera.org:8080/#/c/15393/6/docker/impala-entrypoint.sh
File docker/impala-entrypoint.sh:
http://gerrit.cloudera.org:8080/#/c/15393/6/docker/impala-entrypoint.sh@24
PS6, Line 24: set -xe
No pipefail here? Is that intentional?
--
To view, visit http://gerrit.cloudera.org:8080/15393
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2d88bcb52f793c0fc83c4f84a9b56c5b961e58d5
Gerrit-Change-Number: 15393
Gerrit-PatchSet: 6
Gerrit-Owner: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 13 Mar 2020 18:35:02 +0000
Gerrit-HasComments: Yes
[kudu-CR] [docker] Add an Impala quickstart image
Posted by "Grant Henke (Code Review)" <ge...@cloudera.org>.
Hello Alexey Serbin, Kudu Jenkins, Andrew Wong,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15393
to look at the new patch set (#3).
Change subject: [docker] Add an Impala quickstart image
......................................................................
[docker] Add an Impala quickstart image
This patch adds an `impala` target that can be built and
published for use in quickstart examples. The image is a
single node Impala cluster with a Hive metastore backed
by Derby. This image is purpose build for quickstart usage
only and shouldn't be used for production or multi-node
deployments. The final image size is ~2GB uncompressed.
The build and runtime images reuse the dev and runtime base
images from Kudu given the additional libraries needed are
nominal in size and the reuse cuts back on code duplication.
This image will be used in an Impala quickstart guide that
will be added in a follow on patch.
Change-Id: I2d88bcb52f793c0fc83c4f84a9b56c5b961e58d5
---
M .dockerignore
M docker/Dockerfile
M docker/bootstrap-dev-env.sh
A docker/bootstrap-java-env.sh
A docker/bootstrap-maven-env.sh
M docker/bootstrap-runtime-env.sh
A docker/impala-entrypoint.sh
A docker/impala/etc/hadoop/conf/core-site.xml
A docker/impala/etc/hive/conf/hive-site.xml
9 files changed, 442 insertions(+), 26 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/93/15393/3
--
To view, visit http://gerrit.cloudera.org:8080/15393
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2d88bcb52f793c0fc83c4f84a9b56c5b961e58d5
Gerrit-Change-Number: 15393
Gerrit-PatchSet: 3
Gerrit-Owner: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [docker] Add an Impala quickstart image
Posted by "Grant Henke (Code Review)" <ge...@cloudera.org>.
Grant Henke has removed a vote on this change.
Change subject: [docker] Add an Impala quickstart image
......................................................................
Removed Verified-1 by Kudu Jenkins (120)
--
To view, visit http://gerrit.cloudera.org:8080/15393
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: I2d88bcb52f793c0fc83c4f84a9b56c5b961e58d5
Gerrit-Change-Number: 15393
Gerrit-PatchSet: 2
Gerrit-Owner: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [docker] Add an Impala quickstart image
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/15393 )
Change subject: [docker] Add an Impala quickstart image
......................................................................
Patch Set 7: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/15393
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2d88bcb52f793c0fc83c4f84a9b56c5b961e58d5
Gerrit-Change-Number: 15393
Gerrit-PatchSet: 7
Gerrit-Owner: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 13 Mar 2020 19:17:19 +0000
Gerrit-HasComments: No
[kudu-CR] [docker] Add an Impala quickstart image
Posted by "Grant Henke (Code Review)" <ge...@cloudera.org>.
Grant Henke has posted comments on this change. ( http://gerrit.cloudera.org:8080/15393 )
Change subject: [docker] Add an Impala quickstart image
......................................................................
Patch Set 7: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/15393
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2d88bcb52f793c0fc83c4f84a9b56c5b961e58d5
Gerrit-Change-Number: 15393
Gerrit-PatchSet: 7
Gerrit-Owner: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 13 Mar 2020 19:03:07 +0000
Gerrit-HasComments: No
[kudu-CR] [docker] Add an Impala quickstart image
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/15393 )
Change subject: [docker] Add an Impala quickstart image
......................................................................
Patch Set 4:
> (1 comment)
Ah, that's the right example:
-----
#!/bin/bash
set -e
#set -o pipefail
var=$(which ls | grep x | sed -e ":x::")
echo "past the failure"
-----
Anyway, feel free to keep it as is if you think it's not worth it.
--
To view, visit http://gerrit.cloudera.org:8080/15393
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2d88bcb52f793c0fc83c4f84a9b56c5b961e58d5
Gerrit-Change-Number: 15393
Gerrit-PatchSet: 4
Gerrit-Owner: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Thu, 12 Mar 2020 18:03:28 +0000
Gerrit-HasComments: No
[kudu-CR] [docker] Add an Impala quickstart image
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15393 )
Change subject: [docker] Add an Impala quickstart image
......................................................................
Patch Set 7: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/15393
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2d88bcb52f793c0fc83c4f84a9b56c5b961e58d5
Gerrit-Change-Number: 15393
Gerrit-PatchSet: 7
Gerrit-Owner: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 13 Mar 2020 19:04:23 +0000
Gerrit-HasComments: No
[kudu-CR] [docker] Add an Impala quickstart image
Posted by "Grant Henke (Code Review)" <ge...@cloudera.org>.
Grant Henke has posted comments on this change. ( http://gerrit.cloudera.org:8080/15393 )
Change subject: [docker] Add an Impala quickstart image
......................................................................
Patch Set 6:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/15393/6/docker/impala-entrypoint.sh
File docker/impala-entrypoint.sh:
http://gerrit.cloudera.org:8080/#/c/15393/6/docker/impala-entrypoint.sh@24
PS6, Line 24: set -xe
> No pipefail here? Is that intentional?
Done
--
To view, visit http://gerrit.cloudera.org:8080/15393
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2d88bcb52f793c0fc83c4f84a9b56c5b961e58d5
Gerrit-Change-Number: 15393
Gerrit-PatchSet: 6
Gerrit-Owner: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 13 Mar 2020 19:02:56 +0000
Gerrit-HasComments: Yes
[kudu-CR] [docker] Add an Impala quickstart image
Posted by "Grant Henke (Code Review)" <ge...@cloudera.org>.
Grant Henke has posted comments on this change. ( http://gerrit.cloudera.org:8080/15393 )
Change subject: [docker] Add an Impala quickstart image
......................................................................
Patch Set 2: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/15393
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2d88bcb52f793c0fc83c4f84a9b56c5b961e58d5
Gerrit-Change-Number: 15393
Gerrit-PatchSet: 2
Gerrit-Owner: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Tue, 10 Mar 2020 14:46:15 +0000
Gerrit-HasComments: No
[kudu-CR] [docker] Add an Impala quickstart image
Posted by "Grant Henke (Code Review)" <ge...@cloudera.org>.
Hello Alexey Serbin, Kudu Jenkins, Andrew Wong,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15393
to look at the new patch set (#2).
Change subject: [docker] Add an Impala quickstart image
......................................................................
[docker] Add an Impala quickstart image
This patch adds an `impala` target that can be built and
published for use in quickstart examples. The image is a
single node Impala cluster with a Hive metastore backed
by Derby. This image is purpose build for quickstart usage
only and shouldn't be used for production or multi-node
deployments. The final image size is ~2GB uncompressed.
The build and runtime images reuse the dev and runtime base
images from Kudu given the additional libraries needed are
nominal in size and the reuse cuts back on code duplication.
This image will be used in an Impala quickstart guide that
will be added in a follow on patch.
Change-Id: I2d88bcb52f793c0fc83c4f84a9b56c5b961e58d5
---
M .dockerignore
M docker/Dockerfile
M docker/bootstrap-dev-env.sh
A docker/bootstrap-java-env.sh
A docker/bootstrap-maven-env.sh
M docker/bootstrap-runtime-env.sh
A docker/impala-entrypoint.sh
A docker/impala/etc/hadoop/conf/core-site.xml
A docker/impala/etc/hive/conf/hive-site.xml
9 files changed, 442 insertions(+), 26 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/93/15393/2
--
To view, visit http://gerrit.cloudera.org:8080/15393
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2d88bcb52f793c0fc83c4f84a9b56c5b961e58d5
Gerrit-Change-Number: 15393
Gerrit-PatchSet: 2
Gerrit-Owner: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [docker] Add an Impala quickstart image
Posted by "Grant Henke (Code Review)" <ge...@cloudera.org>.
Grant Henke has posted comments on this change. ( http://gerrit.cloudera.org:8080/15393 )
Change subject: [docker] Add an Impala quickstart image
......................................................................
Patch Set 2:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/15393/1/docker/Dockerfile
File docker/Dockerfile:
http://gerrit.cloudera.org:8080/#/c/15393/1/docker/Dockerfile@419
PS1, Line 419: WORKDIR $IMPALA_HOME
> Doesn't this link both statestored and catalogd to impalad?
It does. They are actually all the same binary, so this saves space.
http://gerrit.cloudera.org:8080/#/c/15393/1/docker/Dockerfile@448
PS1, Line 448: python-env.s
> Given the Impala version is pegged at 3.3, should this version be fixed too
It probably should be, though it doesn't matter at the moment. That will be fixed when the above TODO is complete in a follow on I suppose.
http://gerrit.cloudera.org:8080/#/c/15393/1/docker/Dockerfile@469
PS1, Line 469:
> nit: maybe note the Impala version?
I can add the Impala Version as a label below.
http://gerrit.cloudera.org:8080/#/c/15393/1/docker/bootstrap-dev-env.sh
File docker/bootstrap-dev-env.sh:
http://gerrit.cloudera.org:8080/#/c/15393/1/docker/bootstrap-dev-env.sh@62
PS1, Line 62:
: # Install exta impala packages for the impala images. They are nominal in size.
: # --no-install-recommends keeps the install smaller
: yum install -y --no-install-recommends \
: libffi-devel \
: lzo-devel \
: tzdata
> Should be yum?
Done
http://gerrit.cloudera.org:8080/#/c/15393/1/docker/bootstrap-runtime-env.sh
File docker/bootstrap-runtime-env.sh:
http://gerrit.cloudera.org:8080/#/c/15393/1/docker/bootstrap-runtime-env.sh@44
PS1, Line 44: yum install -y --no-install-recommends \
: cyrus-sasl-devel \
: lzo-devel \
: tzdata
> Here too
Done
--
To view, visit http://gerrit.cloudera.org:8080/15393
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2d88bcb52f793c0fc83c4f84a9b56c5b961e58d5
Gerrit-Change-Number: 15393
Gerrit-PatchSet: 2
Gerrit-Owner: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Tue, 10 Mar 2020 14:10:38 +0000
Gerrit-HasComments: Yes
[kudu-CR] [docker] Add an Impala quickstart image
Posted by "Grant Henke (Code Review)" <ge...@cloudera.org>.
Hello Alexey Serbin, Kudu Jenkins, Andrew Wong,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15393
to look at the new patch set (#7).
Change subject: [docker] Add an Impala quickstart image
......................................................................
[docker] Add an Impala quickstart image
This patch adds an `impala` target that can be built and
published for use in quickstart examples. The image is a
single node Impala cluster with a Hive metastore backed
by Derby. This image is purpose build for quickstart usage
only and shouldn't be used for production or multi-node
deployments. The final image size is ~2GB uncompressed.
The build and runtime images reuse the dev and runtime base
images from Kudu given the additional libraries needed are
nominal in size and the reuse cuts back on code duplication.
This image will be used in an Impala quickstart guide that
will be added in a follow on patch.
Currently the image only works on Ubuntu distributions due
to Impala side script assumptions. Follow on patches will
fix these issues for other distributions.
Change-Id: I2d88bcb52f793c0fc83c4f84a9b56c5b961e58d5
---
M .dockerignore
M docker/Dockerfile
M docker/bootstrap-dev-env.sh
A docker/bootstrap-java-env.sh
A docker/bootstrap-maven-env.sh
M docker/bootstrap-python-env.sh
M docker/bootstrap-runtime-env.sh
M docker/docker-build.sh
A docker/impala-entrypoint.sh
A docker/impala/etc/hadoop/conf/core-site.xml
A docker/impala/etc/hive/conf/hive-site.xml
M docker/kudu-entrypoint.sh
12 files changed, 493 insertions(+), 33 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/93/15393/7
--
To view, visit http://gerrit.cloudera.org:8080/15393
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2d88bcb52f793c0fc83c4f84a9b56c5b961e58d5
Gerrit-Change-Number: 15393
Gerrit-PatchSet: 7
Gerrit-Owner: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [docker] Add an Impala quickstart image
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/15393 )
Change subject: [docker] Add an Impala quickstart image
......................................................................
Patch Set 4:
(10 comments)
http://gerrit.cloudera.org:8080/#/c/15393/4/docker/bootstrap-java-env.sh
File docker/bootstrap-java-env.sh:
http://gerrit.cloudera.org:8080/#/c/15393/4/docker/bootstrap-java-env.sh@27
PS4, Line 27:
Enable 'pipefail' option since to spot issues with VERSION_NAME early?
http://gerrit.cloudera.org:8080/#/c/15393/4/docker/bootstrap-java-env.sh@39
PS4, Line 39: /tmp/* /var/tmp/*
I'm not sure this is safe: in /tmp and /var/tmp might be various files which are created and used by other components of the OS (e.g., there might be Kerberos cache files, SSH pipes, etc.)
http://gerrit.cloudera.org:8080/#/c/15393/4/docker/bootstrap-maven-env.sh
File docker/bootstrap-maven-env.sh:
http://gerrit.cloudera.org:8080/#/c/15393/4/docker/bootstrap-maven-env.sh@29
PS4, Line 29: apache-maven-3.6.3-bin.tar.gz
Create variable for this and re-use it everywhere in the script?
http://gerrit.cloudera.org:8080/#/c/15393/4/docker/bootstrap-maven-env.sh@31
PS4, Line 31: z
nit: 'z' isn't necessary here, can be omitted
http://gerrit.cloudera.org:8080/#/c/15393/4/docker/bootstrap-maven-env.sh@32
PS4, Line 32: /usr/local/bin
What if /usr/local/bin didn't exist prior to this invocation?
http://gerrit.cloudera.org:8080/#/c/15393/4/docker/bootstrap-maven-env.sh@33
PS4, Line 33: rm
rm -f ?
http://gerrit.cloudera.org:8080/#/c/15393/4/docker/impala-entrypoint.sh
File docker/impala-entrypoint.sh:
http://gerrit.cloudera.org:8080/#/c/15393/4/docker/impala-entrypoint.sh@24
PS4, Line 24: set -e
Maybe, set the 'pipefail' option as well to catch the issues with JAVA_HOME assignment earlier, if any?
http://gerrit.cloudera.org:8080/#/c/15393/4/docker/impala/etc/hadoop/conf/core-site.xml
File docker/impala/etc/hadoop/conf/core-site.xml:
http://gerrit.cloudera.org:8080/#/c/15393/4/docker/impala/etc/hadoop/conf/core-site.xml@26
PS4, Line 26: <name>dfs.client.read.shortcircuit</name>
: <value>true</value>
Could you add a comment explaining the reason for this customization?
http://gerrit.cloudera.org:8080/#/c/15393/4/docker/impala/etc/hadoop/conf/core-site.xml@29
PS4, Line 29: <property>
: <name>hadoop.proxyuser.root.hosts</name>
: <value>*</value>
: </property>
: <property>
: <name>hadoop.proxyuser.root.groups</name>
: <value>*</value>
: </property>
: <property>
: <name>hadoop.proxyuser.impala.hosts</name>
: <value>*</value>
: </property>
: <property>
: <name>hadoop.proxyuser.impala.groups</name>
: <value>*</value>
: </property>
Do you mind adding a small comment to explain why the defaults are not good enough and these properties are set to these custom values?
http://gerrit.cloudera.org:8080/#/c/15393/4/docker/impala/etc/hive/conf/hive-site.xml
File docker/impala/etc/hive/conf/hive-site.xml:
http://gerrit.cloudera.org:8080/#/c/15393/4/docker/impala/etc/hive/conf/hive-site.xml@38
PS4, Line 38: <name>hive.metastore.schema.verification</name>
: <value>false</value>
nit: do you mind adding a comment to explain why it's crucial to have this customization here? Why the default value isn't good enough?
--
To view, visit http://gerrit.cloudera.org:8080/15393
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2d88bcb52f793c0fc83c4f84a9b56c5b961e58d5
Gerrit-Change-Number: 15393
Gerrit-PatchSet: 4
Gerrit-Owner: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Wed, 11 Mar 2020 21:59:12 +0000
Gerrit-HasComments: Yes
[kudu-CR] [docker] Add an Impala quickstart image
Posted by "Grant Henke (Code Review)" <ge...@cloudera.org>.
Grant Henke has removed a vote on this change.
Change subject: [docker] Add an Impala quickstart image
......................................................................
Removed Verified-1 by Kudu Jenkins (120)
--
To view, visit http://gerrit.cloudera.org:8080/15393
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: I2d88bcb52f793c0fc83c4f84a9b56c5b961e58d5
Gerrit-Change-Number: 15393
Gerrit-PatchSet: 3
Gerrit-Owner: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [docker] Add an Impala quickstart image
Posted by "Grant Henke (Code Review)" <ge...@cloudera.org>.
Hello Alexey Serbin, Kudu Jenkins, Andrew Wong,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15393
to look at the new patch set (#5).
Change subject: [docker] Add an Impala quickstart image
......................................................................
[docker] Add an Impala quickstart image
This patch adds an `impala` target that can be built and
published for use in quickstart examples. The image is a
single node Impala cluster with a Hive metastore backed
by Derby. This image is purpose build for quickstart usage
only and shouldn't be used for production or multi-node
deployments. The final image size is ~2GB uncompressed.
The build and runtime images reuse the dev and runtime base
images from Kudu given the additional libraries needed are
nominal in size and the reuse cuts back on code duplication.
This image will be used in an Impala quickstart guide that
will be added in a follow on patch.
Change-Id: I2d88bcb52f793c0fc83c4f84a9b56c5b961e58d5
---
M .dockerignore
M docker/Dockerfile
M docker/bootstrap-dev-env.sh
A docker/bootstrap-java-env.sh
A docker/bootstrap-maven-env.sh
M docker/bootstrap-runtime-env.sh
A docker/impala-entrypoint.sh
A docker/impala/etc/hadoop/conf/core-site.xml
A docker/impala/etc/hive/conf/hive-site.xml
9 files changed, 480 insertions(+), 31 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/93/15393/5
--
To view, visit http://gerrit.cloudera.org:8080/15393
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2d88bcb52f793c0fc83c4f84a9b56c5b961e58d5
Gerrit-Change-Number: 15393
Gerrit-PatchSet: 5
Gerrit-Owner: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] [docker] Add an Impala quickstart image
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15393 )
Change subject: [docker] Add an Impala quickstart image
......................................................................
Patch Set 1:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/15393/1/docker/Dockerfile
File docker/Dockerfile:
http://gerrit.cloudera.org:8080/#/c/15393/1/docker/Dockerfile@419
PS1, Line 419: RUN ln -s impalad statestored && ln -s impalad catalogd
Doesn't this link both statestored and catalogd to impalad?
http://gerrit.cloudera.org:8080/#/c/15393/1/docker/Dockerfile@448
PS1, Line 448: impala-shell
Given the Impala version is pegged at 3.3, should this version be fixed too? Or does it not matter much here since it'll likely be backwards compatible anyway?
http://gerrit.cloudera.org:8080/#/c/15393/1/docker/Dockerfile@469
PS1, Line 469: An image with the Impala binaries pre-installed.
nit: maybe note the Impala version?
http://gerrit.cloudera.org:8080/#/c/15393/1/docker/bootstrap-dev-env.sh
File docker/bootstrap-dev-env.sh:
http://gerrit.cloudera.org:8080/#/c/15393/1/docker/bootstrap-dev-env.sh@62
PS1, Line 62:
: # Install exta impala packages for the impala images. They are nominal in size.
: # --no-install-recommends keeps the install smaller
: apt-get install -y --no-install-recommends \
: libffi-devel \
: lzo-devel \
: tzdata
Should be yum?
http://gerrit.cloudera.org:8080/#/c/15393/1/docker/bootstrap-runtime-env.sh
File docker/bootstrap-runtime-env.sh:
http://gerrit.cloudera.org:8080/#/c/15393/1/docker/bootstrap-runtime-env.sh@44
PS1, Line 44: apt-get install -y --no-install-recommends \
: cyrus-sasl-devel \
: lzo-devel \
: tzdata
Here too
--
To view, visit http://gerrit.cloudera.org:8080/15393
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2d88bcb52f793c0fc83c4f84a9b56c5b961e58d5
Gerrit-Change-Number: 15393
Gerrit-PatchSet: 1
Gerrit-Owner: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Tue, 10 Mar 2020 05:42:54 +0000
Gerrit-HasComments: Yes