You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@impala.apache.org by "Philip Zeyliger (Code Review)" <ge...@cloudera.org> on 2018/10/18 22:55:16 UTC

[Impala-ASF-CR] IMPALA-7698: Add centos support to bootstrap system.

Philip Zeyliger has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11731

Change subject: IMPALA-7698: Add centos support to bootstrap_system.
......................................................................

IMPALA-7698: Add centos support to bootstrap_system.

Largely, the changes involve conditionalizing some invocations to
account for differences between RH and Ubuntu. The trickiest bits were
postgresql permissions (need to accept md5 passwords from localhost) and
default ulimits (1024 user processes/threads is not enough).

To test this, I built using test-with-docker. In additional to the
ulimit issue, I ran into the fact that /tmp needed 1777 permissions for
the postgresql socket, and entrypoint.sh had a few places that needed
special cases. At the moment, the data load ran fine, as did most of the
tests. I observed a test that relied on a python2.7-ism fail, which is
part of the point of this.

A handful of tests fail with "Encounter parse error: failed to open
/usr/share/zoneinfo/GMT-08:00 - No such file or directory.", which
can be reproduced as follows:

[localhost:21000] default> use functional_orc_def; select * from alltypes;
...
WARNINGS: Encounter parse error: failed to open /usr/share/zoneinfo/GMT-08:00 - No such file or directory.

I'm not yet clear on why the ORC library is choosing to open that time zone
file (which doesn't exist), but this change seems workable even with the test
failures.

centos:7 is not addressed by this change. The move to systemd makes "service
sshd start" (and the same for postgresql) not work, and additional care needs
to be done to work around that.

This change is a joint effort with Laszlo Gaal.

Change-Id: Id54294d7607f51de87a9de373dcfc4a33f4bedf5
---
M bin/bootstrap_system.sh
M docker/entrypoint.sh
M docker/test-with-docker.py
3 files changed, 143 insertions(+), 58 deletions(-)

git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/11731/1
--
To view, visit http://gerrit.cloudera.org:8080/11731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Id54294d7607f51de87a9de373dcfc4a33f4bedf5
Gerrit-Change-Number: 11731
Gerrit-PatchSet: 1
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>

[Impala-ASF-CR] IMPALA-7698: Add centos support to bootstrap system.

Posted by "Philip Zeyliger (Code Review)" <ge...@cloudera.org>.

Hello Quanlong Huang, Laszlo Gaal, Jim Apple, Joe McDonnell, Csaba Ringhofer, Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/11731

to look at the new patch set (#2).

Change subject: IMPALA-7698: Add centos support to bootstrap_system.
......................................................................

IMPALA-7698: Add centos support to bootstrap_system.

A handful of tests fail with "Encounter parse error: failed to open
/usr/share/zoneinfo/GMT-08:00 - No such file or directory.", which
can be reproduced as follows:

[localhost:21000] default> use functional_orc_def; select * from alltypes;
...
WARNINGS: Encounter parse error: failed to open /usr/share/zoneinfo/GMT-08:00 - No such file or directory.

I'm not yet clear on why the ORC library is choosing to open that time
zone file (which doesn't exist), but this change seems workable even
with the test failures.

centos:7 is not addressed by this change. The move to systemd makes
"service sshd start" (and the same for postgresql) not work, and
additional care needs to be done to work around that.

This change is a joint effort with Laszlo Gaal.

Change-Id: Id54294d7607f51de87a9de373dcfc4a33f4bedf5
---
M bin/bootstrap_system.sh
M docker/entrypoint.sh
M docker/test-with-docker.py
3 files changed, 157 insertions(+), 59 deletions(-)

git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/11731/2
--
To view, visit http://gerrit.cloudera.org:8080/11731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id54294d7607f51de87a9de373dcfc4a33f4bedf5
Gerrit-Change-Number: 11731
Gerrit-PatchSet: 2
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>

[Impala-ASF-CR] IMPALA-7698: Add centos support to bootstrap system.

Posted by "Philip Zeyliger (Code Review)" <ge...@cloudera.org>.

Hello Quanlong Huang, Laszlo Gaal, Jim Apple, Joe McDonnell, Csaba Ringhofer, Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/11731

to look at the new patch set (#4).

Change subject: IMPALA-7698: Add centos support to bootstrap_system.
......................................................................

IMPALA-7698: Add centos support to bootstrap_system.

Largely, the changes involve conditionalizing some invocations to
account for differences between RH and Ubuntu. The trickiest bits were
timezone-related test errors (see below), postgresql permissions (need
to accept md5 passwords from localhost) and default ulimits (1024 user
processes/threads is not enough).

In the course of development, I encountered a handful of tests fail with
"Encounter parse error: failed to open /usr/share/zoneinfo/GMT-08:00 -
No such file or directory.", which was reproduced as follows:

[localhost:21000] default> use functional_orc_def; select * from alltypes;
...
WARNINGS: Encounter parse error: failed to open /usr/share/zoneinfo/GMT-08:00 - No such file or directory.

With Quanlong's help, I learned what was happening. test-with-docker was
translating my time zone (America/Los_Angeles) to US/Pacific-New,
because realpath(/etc/localtime) = US/Pacific-New. This timezone exists
in centos:6, so that wasn't a problem. However, this timezone does not
exist in the package "tzdata-java", which is the copy of the timezone
information used by Java. (There are bugs here that may have been fixed
in centos:7.) As a result, when ORC asks (by using
TimeZone.getDefault().getID()) the JDK
(src/solaris/native/java/util/TimeZone_md.c) for the default timezone,
it can't find the same name as /etc/localtime points to in its
repository and defaults to "GMT-08:00". This string then gets written
into the ORC files generated by Hive as part of data load, and then the
C++ library can't read them. This is fixed by changing "realpath"
to "readlink" in test-with-docker.py.

centos:7 is not addressed by this change. The move to systemd makes
"service sshd start" (and the same for postgresql) not work, and
additional care needs to be done to work around that.

This change is a joint effort with Laszlo Gaal.

Change-Id: Id54294d7607f51de87a9de373dcfc4a33f4bedf5
---
M bin/bootstrap_system.sh
M docker/entrypoint.sh
M docker/test-with-docker.py
3 files changed, 167 insertions(+), 61 deletions(-)

git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/11731/4
--
To view, visit http://gerrit.cloudera.org:8080/11731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id54294d7607f51de87a9de373dcfc4a33f4bedf5
Gerrit-Change-Number: 11731
Gerrit-PatchSet: 4
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>

[Impala-ASF-CR] IMPALA-7698: Add centos support to bootstrap system.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11731 )

Change subject: IMPALA-7698: Add centos support to bootstrap_system.
......................................................................


Patch Set 5:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3357/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/11731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id54294d7607f51de87a9de373dcfc4a33f4bedf5
Gerrit-Change-Number: 11731
Gerrit-PatchSet: 5
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 25 Oct 2018 20:30:48 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7698: Add centos support to bootstrap system.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11731 )

Change subject: IMPALA-7698: Add centos support to bootstrap_system.
......................................................................


Patch Set 6:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3360/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/11731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id54294d7607f51de87a9de373dcfc4a33f4bedf5
Gerrit-Change-Number: 11731
Gerrit-PatchSet: 6
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 25 Oct 2018 20:32:29 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7698: Add centos support to bootstrap system.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11731 )

Change subject: IMPALA-7698: Add centos support to bootstrap_system.
......................................................................


Patch Set 5:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/1157/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/11731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id54294d7607f51de87a9de373dcfc4a33f4bedf5
Gerrit-Change-Number: 11731
Gerrit-PatchSet: 5
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 25 Oct 2018 16:46:34 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7698: Add centos support to bootstrap system.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11731 )

Change subject: IMPALA-7698: Add centos support to bootstrap_system.
......................................................................


Patch Set 4:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/1151/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/11731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id54294d7607f51de87a9de373dcfc4a33f4bedf5
Gerrit-Change-Number: 11731
Gerrit-PatchSet: 4
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 25 Oct 2018 04:13:13 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7698: Add centos support to bootstrap system.

Posted by "Philip Zeyliger (Code Review)" <ge...@cloudera.org>.

Hello Quanlong Huang, Laszlo Gaal, Jim Apple, Joe McDonnell, Csaba Ringhofer, Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/11731

to look at the new patch set (#6).

Change subject: IMPALA-7698: Add centos support to bootstrap_system.
......................................................................

IMPALA-7698: Add centos support to bootstrap_system.

[localhost:21000] default> use functional_orc_def; select * from alltypes;
...
WARNINGS: Encounter parse error: failed to open /usr/share/zoneinfo/GMT-08:00 - No such file or directory.

centos:7 is not addressed by this change. The move to systemd makes
"service sshd start" (and the same for postgresql) not work, and
additional care needs to be done to work around that.

This change is a joint effort with Laszlo Gaal.

Change-Id: Id54294d7607f51de87a9de373dcfc4a33f4bedf5
---
M bin/bootstrap_system.sh
M docker/entrypoint.sh
M docker/test-with-docker.py
3 files changed, 166 insertions(+), 61 deletions(-)

git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/11731/6
--
To view, visit http://gerrit.cloudera.org:8080/11731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id54294d7607f51de87a9de373dcfc4a33f4bedf5
Gerrit-Change-Number: 11731
Gerrit-PatchSet: 6
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>

[Impala-ASF-CR] IMPALA-7698: Add centos support to bootstrap system.

Posted by "Jim Apple (Code Review)" <ge...@cloudera.org>.

Jim Apple has posted comments on this change. ( http://gerrit.cloudera.org:8080/11731 )

Change subject: IMPALA-7698: Add centos support to bootstrap_system.
......................................................................


Patch Set 3: Code-Review+2

(1 comment)

http://gerrit.cloudera.org:8080/#/c/11731/2/bin/bootstrap_system.sh
File bin/bootstrap_system.sh:

http://gerrit.cloudera.org:8080/#/c/11731/2/bin/bootstrap_system.sh@196
PS2, Line 196: redhat sudo service postgresql stop
"localhost"



-- 
To view, visit http://gerrit.cloudera.org:8080/11731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id54294d7607f51de87a9de373dcfc4a33f4bedf5
Gerrit-Change-Number: 11731
Gerrit-PatchSet: 3
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 24 Oct 2018 00:23:46 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7698: Add centos support to bootstrap system.

Posted by "Laszlo Gaal (Code Review)" <ge...@cloudera.org>.

Laszlo Gaal has posted comments on this change. ( http://gerrit.cloudera.org:8080/11731 )

Change subject: IMPALA-7698: Add centos support to bootstrap_system.
......................................................................


Patch Set 3:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/11731/3/bin/bootstrap_system.sh
File bin/bootstrap_system.sh:

http://gerrit.cloudera.org:8080/#/c/11731/3/bin/bootstrap_system.sh@139
PS3, Line 139: if lsb_release -r -s | grep '^7'; then
             :   redhat sudo yum install -y libev-devel pigz python-virtualenv
Since you install EPEL above, you may want to install these packages for CentOS 6 from there. EPEL offers the following versions:
- pigz-2.3.4-1.el6.x86_64
- libev-devel-4.03-3.el6.x86_64
- python-virtualenv-12.0.7-1.el6.noarch.rpm
- python-setuptools-0.6.10-4.el6_9.noarch.rpm



-- 
To view, visit http://gerrit.cloudera.org:8080/11731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id54294d7607f51de87a9de373dcfc4a33f4bedf5
Gerrit-Change-Number: 11731
Gerrit-PatchSet: 3
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 24 Oct 2018 17:36:27 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7698: Add centos support to bootstrap system.

Posted by "Philip Zeyliger (Code Review)" <ge...@cloudera.org>.

Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/11731 )

Change subject: IMPALA-7698: Add centos support to bootstrap_system.
......................................................................


Patch Set 4:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/11731/4/bin/bootstrap_system.sh
File bin/bootstrap_system.sh:

http://gerrit.cloudera.org:8080/#/c/11731/4/bin/bootstrap_system.sh@131
PS4, Line 131: pigz
> Nit: pigz isn't listed under Ubuntu. Let's add it.
I did the opposite; it doesn't exist in centos:6. Interestingly, yum doesn't fail if a package is missing. --setopt=skip_missing_names_on_install=False doesn't work on centos6.


http://gerrit.cloudera.org:8080/#/c/11731/4/bin/bootstrap_system.sh@132
PS4, Line 132: sudo
> Nit: sudo yum install sudo?
Removed.



-- 
To view, visit http://gerrit.cloudera.org:8080/11731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id54294d7607f51de87a9de373dcfc4a33f4bedf5
Gerrit-Change-Number: 11731
Gerrit-PatchSet: 4
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 25 Oct 2018 16:28:16 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7698: Add centos support to bootstrap system.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11731 )

Change subject: IMPALA-7698: Add centos support to bootstrap_system.
......................................................................


Patch Set 6:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3359/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/11731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id54294d7607f51de87a9de373dcfc4a33f4bedf5
Gerrit-Change-Number: 11731
Gerrit-PatchSet: 6
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 25 Oct 2018 20:32:02 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7698: Add centos support to bootstrap system.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11731 )

Change subject: IMPALA-7698: Add centos support to bootstrap_system.
......................................................................


Patch Set 3:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/11731/3/bin/bootstrap_system.sh
File bin/bootstrap_system.sh:

http://gerrit.cloudera.org:8080/#/c/11731/3/bin/bootstrap_system.sh@148
PS3, Line 148:   https://www-us.apache.org/dist/maven/maven-3/3.5.4/binaries/apache-maven-3.5.4-bin.tar.gz \
line too long (93 > 90)


http://gerrit.cloudera.org:8080/#/c/11731/3/bin/bootstrap_system.sh@150
PS3, Line 150: redhat sha512sum -c - <<< '2a803f578f341e164f6753e410413d16ab60fabe31dc491d1fe35c984a5cce696bc71f57757d4538fe7738be04065a216f3ebad4ef7e0ce1bb4c51bc36d6be86  apache-maven-3.5.4-bin.tar.gz'
line too long (187 > 90)


http://gerrit.cloudera.org:8080/#/c/11731/3/bin/bootstrap_system.sh@151
PS3, Line 151: redhat sha512sum -c - <<< 'c8321aa223f70d7e64d3d0274263000cfffb46fbea61488534e26f9f0245d99e9872d0888e35cd3274416392a13f80c748c07750caaeffa5f9cae1220020715f  apache-ant-1.9.13-bin.tar.gz'
line too long (186 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/11731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id54294d7607f51de87a9de373dcfc4a33f4bedf5
Gerrit-Change-Number: 11731
Gerrit-PatchSet: 3
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 24 Oct 2018 16:17:20 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7698: Add centos support to bootstrap system.

Posted by "Philip Zeyliger (Code Review)" <ge...@cloudera.org>.

Hello Quanlong Huang, Laszlo Gaal, Jim Apple, Joe McDonnell, Csaba Ringhofer, Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/11731

to look at the new patch set (#3).

Change subject: IMPALA-7698: Add centos support to bootstrap_system.
......................................................................

IMPALA-7698: Add centos support to bootstrap_system.

A handful of tests fail with "Encounter parse error: failed to open
/usr/share/zoneinfo/GMT-08:00 - No such file or directory.", which
can be reproduced as follows:

[localhost:21000] default> use functional_orc_def; select * from alltypes;
...
WARNINGS: Encounter parse error: failed to open /usr/share/zoneinfo/GMT-08:00 - No such file or directory.

I'm not yet clear on why the ORC library is choosing to open that time
zone file (which doesn't exist), but this change seems workable even
with the test failures.

centos:7 is not addressed by this change. The move to systemd makes
"service sshd start" (and the same for postgresql) not work, and
additional care needs to be done to work around that.

This change is a joint effort with Laszlo Gaal.

Change-Id: Id54294d7607f51de87a9de373dcfc4a33f4bedf5
---
M bin/bootstrap_system.sh
M docker/entrypoint.sh
M docker/test-with-docker.py
3 files changed, 160 insertions(+), 59 deletions(-)

git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/11731/3
--
To view, visit http://gerrit.cloudera.org:8080/11731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id54294d7607f51de87a9de373dcfc4a33f4bedf5
Gerrit-Change-Number: 11731
Gerrit-PatchSet: 3
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>

[Impala-ASF-CR] IMPALA-7698: Add centos support to bootstrap system.

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.

Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/11731 )

Change subject: IMPALA-7698: Add centos support to bootstrap_system.
......................................................................


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/11731/1//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/11731/1//COMMIT_MSG@29
PS1, Line 29: I'm not yet clear on why the ORC library is choosing to open that time zone
            : file (which doesn't exist), but this change seems workable even with the test
            : failures.
About the ORC timezone issues:
tldr: ORC's timezone handling is likely to be broken in Impala so we should patch it in the toolchain

The ORC library implements its own IANA timezone handling to convert stored timestamps from UTC to local time + do something similar for min/max stats. The writer's timezone can be also stored in .orc files and used instead of local timezone.

Impala's and ORC library's timezone can be different because of several reasons:
- ORC's timezone is not overridden by env var TZ and query option timezone
- ORC uses a simpler way to detect the local timezone which may not work on some Linux distros (see TimezoneDatabase::LocalZoneName in Impala vs LOCAL_TIMEZONE in Orc)
- .orc files can use any time zone as writer's timezone and we cannot be sure that it will exist on the reader machine

My suggestion is to patch the ORC library in the toolchain and remove timezone handling (e.g. by always using UTC, maybe depending on a flag), as the way it is currently working is likely to be broken and is surely not consistent with the rest of Impala.

I am not sure how timezones could be handled correctly in Orc + Impala. If someone plans to work on it, I would gladly help in the integration to Impala.



-- 
To view, visit http://gerrit.cloudera.org:8080/11731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id54294d7607f51de87a9de373dcfc4a33f4bedf5
Gerrit-Change-Number: 11731
Gerrit-PatchSet: 1
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Fri, 19 Oct 2018 16:03:08 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7698: Add centos support to bootstrap system.

Posted by "Philip Zeyliger (Code Review)" <ge...@cloudera.org>.

Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/11731 )

Change subject: IMPALA-7698: Add centos support to bootstrap_system.
......................................................................


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/11731/1//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/11731/1//COMMIT_MSG@29
PS1, Line 29: I'm not yet clear on why the ORC library is choosing to open that time zone
            : file (which doesn't exist), but this change seems workable even with the test
            : failures.
> About the ORC timezone issues:
For this particular change, I'm planning on not doing anything with Orc. I think it's useful to add centos6 support. If only data load goes through (which it does), we can catch a bunch of py2.6/py2.7 compatibility things during GVO, which is extremely handy.

I'm still puzzled about why physical machines that we use with CentOS 6 have different results. It might be as simple as non-determinism in file listing order or something. I think the next thing to do to figure it out might be either (a) debugger or (b) taking the files generated within the Docker context and copying them to the hardware context, or vice-versa, but I'm inclined to ignore that for now.



-- 
To view, visit http://gerrit.cloudera.org:8080/11731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id54294d7607f51de87a9de373dcfc4a33f4bedf5
Gerrit-Change-Number: 11731
Gerrit-PatchSet: 1
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 19 Oct 2018 16:43:58 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7698: Add centos support to bootstrap system.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11731 )

Change subject: IMPALA-7698: Add centos support to bootstrap_system.
......................................................................


Patch Set 5: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/3351/


-- 
To view, visit http://gerrit.cloudera.org:8080/11731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id54294d7607f51de87a9de373dcfc4a33f4bedf5
Gerrit-Change-Number: 11731
Gerrit-PatchSet: 5
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 25 Oct 2018 18:06:49 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7698: Add centos support to bootstrap system.

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.

Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/11731 )

Change subject: IMPALA-7698: Add centos support to bootstrap_system.
......................................................................


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/11731/1//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/11731/1//COMMIT_MSG@29
PS1, Line 29: I'm not yet clear on why the ORC library is choosing to open that time zone
            : file (which doesn't exist), but this change seems workable even with the test
            : failures.
> For this particular change, I'm planning on not doing anything with Orc. I 
I see, if these tests do not have to pass, then it is better not to jump into this too deeply. I just wanted to dump my thoughts about this when a I saw the review.

Thanks for creating the follow up Jira!



-- 
To view, visit http://gerrit.cloudera.org:8080/11731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id54294d7607f51de87a9de373dcfc4a33f4bedf5
Gerrit-Change-Number: 11731
Gerrit-PatchSet: 1
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 19 Oct 2018 16:55:58 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7698: Add centos support to bootstrap system.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11731 )

Change subject: IMPALA-7698: Add centos support to bootstrap_system.
......................................................................


Patch Set 7: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/11731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id54294d7607f51de87a9de373dcfc4a33f4bedf5
Gerrit-Change-Number: 11731
Gerrit-PatchSet: 7
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 26 Oct 2018 08:43:20 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7698: Add centos support to bootstrap system.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11731 )

Change subject: IMPALA-7698: Add centos support to bootstrap_system.
......................................................................


Patch Set 3:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/1140/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/11731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id54294d7607f51de87a9de373dcfc4a33f4bedf5
Gerrit-Change-Number: 11731
Gerrit-PatchSet: 3
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 24 Oct 2018 00:56:18 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7698: Add centos support to bootstrap system.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11731 )

Change subject: IMPALA-7698: Add centos support to bootstrap_system.
......................................................................


Patch Set 7:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3362/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/11731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id54294d7607f51de87a9de373dcfc4a33f4bedf5
Gerrit-Change-Number: 11731
Gerrit-PatchSet: 7
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 26 Oct 2018 04:47:34 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7698: Add centos support to bootstrap system.

Posted by "Philip Zeyliger (Code Review)" <ge...@cloudera.org>.

Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/11731 )

Change subject: IMPALA-7698: Add centos support to bootstrap_system.
......................................................................


Patch Set 4:

(6 comments)

Good news--the ORC / timestamp stuff is figured out. 

Attila--this changes a line you and I talked about in https://gerrit.cloudera.org/#/c/11730/ back to readlink() from realpath(). Definitely a curiosity I would not have predicted!

I'm comfortable carrying the +2, but I'll let this linger for a bit to see if anyone wants to make more comments. I'm waiting on the review for the parent change, anyway.

Thanks Quanlong for hinting me to the conclusion of the timezone stuff. The update there is in the commit message.

http://gerrit.cloudera.org:8080/#/c/11731/1//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/11731/1//COMMIT_MSG@29
PS1, Line 29: 
            : With Quanlong's help, I learned what was happening. test-with-docker was
            : translati
> The error is due to hive writes a wrong timezone name into the ORC files. I
Thank you!

You lead me on the right path. The updated commit message explains what happens, but basically the tz database for Java and Linux are different here, and Java is missing some entries.


http://gerrit.cloudera.org:8080/#/c/11731/2/bin/bootstrap_system.sh
File bin/bootstrap_system.sh:

http://gerrit.cloudera.org:8080/#/c/11731/2/bin/bootstrap_system.sh@145
PS2, Line 145:   https://www-us.apache.org/dist/ant/binaries/apache-ant-1.9.13-bin.tar.gz
> line too long (93 > 90)
ignored.


http://gerrit.cloudera.org:8080/#/c/11731/2/bin/bootstrap_system.sh@147
PS2, Line 147: redhat sha512sum -c - <<< 'c8321aa223f70d7e64d3d0274263000cfffb46fbea61488534e26f9f0245d99e9872d0888e35cd3274416392a13f80c748c07750caaeffa5f9cae1220020715f  apache-ant-1.9.13-bin.tar.gz'
> line too long (187 > 90)
ignored


http://gerrit.cloudera.org:8080/#/c/11731/2/bin/bootstrap_system.sh@148
PS2, Line 148: redhat sudo tar -C /usr/local -xzf apache-maven-3.5.4-bin.tar.gz
> line too long (186 > 90)
ignored


http://gerrit.cloudera.org:8080/#/c/11731/2/bin/bootstrap_system.sh@196
PS2, Line 196: # widely.
> "localhost"
Done


http://gerrit.cloudera.org:8080/#/c/11731/3/bin/bootstrap_system.sh
File bin/bootstrap_system.sh:

http://gerrit.cloudera.org:8080/#/c/11731/3/bin/bootstrap_system.sh@139
PS3, Line 139: # Clean up yum caches
             : redhat sudo yum clean all
> Since you install EPEL above, you may want to install these packages for Ce
I removed this entirely. They don't seem to be strictly necessary at the moment.



-- 
To view, visit http://gerrit.cloudera.org:8080/11731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id54294d7607f51de87a9de373dcfc4a33f4bedf5
Gerrit-Change-Number: 11731
Gerrit-PatchSet: 4
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 25 Oct 2018 03:39:53 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7698: Add centos support to bootstrap system.

Posted by "Philip Zeyliger (Code Review)" <ge...@cloudera.org>.

Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/11731 )

Change subject: IMPALA-7698: Add centos support to bootstrap_system.
......................................................................


Patch Set 7: Code-Review+2

Carrying +2.


-- 
To view, visit http://gerrit.cloudera.org:8080/11731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id54294d7607f51de87a9de373dcfc4a33f4bedf5
Gerrit-Change-Number: 11731
Gerrit-PatchSet: 7
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 26 Oct 2018 04:47:17 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7698: Add centos support to bootstrap system.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11731 )

Change subject: IMPALA-7698: Add centos support to bootstrap_system.
......................................................................


Patch Set 5:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3351/ DRY_RUN=false


-- 
To view, visit http://gerrit.cloudera.org:8080/11731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id54294d7607f51de87a9de373dcfc4a33f4bedf5
Gerrit-Change-Number: 11731
Gerrit-PatchSet: 5
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 25 Oct 2018 16:28:32 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7698: Add centos support to bootstrap system.

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.

Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/11731 )

Change subject: IMPALA-7698: Add centos support to bootstrap_system.
......................................................................


Patch Set 4:

(2 comments)

Thanks for doing this!

http://gerrit.cloudera.org:8080/#/c/11731/4/bin/bootstrap_system.sh
File bin/bootstrap_system.sh:

http://gerrit.cloudera.org:8080/#/c/11731/4/bin/bootstrap_system.sh@131
PS4, Line 131: pigz
Nit: pigz isn't listed under Ubuntu. Let's add it.


http://gerrit.cloudera.org:8080/#/c/11731/4/bin/bootstrap_system.sh@132
PS4, Line 132: sudo
Nit: sudo yum install sudo?



-- 
To view, visit http://gerrit.cloudera.org:8080/11731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id54294d7607f51de87a9de373dcfc4a33f4bedf5
Gerrit-Change-Number: 11731
Gerrit-PatchSet: 4
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 25 Oct 2018 16:18:34 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7698: Add centos support to bootstrap system.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11731 )

Change subject: IMPALA-7698: Add centos support to bootstrap_system.
......................................................................


Patch Set 6: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/11731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id54294d7607f51de87a9de373dcfc4a33f4bedf5
Gerrit-Change-Number: 11731
Gerrit-PatchSet: 6
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Fri, 26 Oct 2018 00:22:40 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7698: Add centos support to bootstrap system.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11731 )

Change subject: IMPALA-7698: Add centos support to bootstrap_system.
......................................................................


Patch Set 4:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/11731/4/bin/bootstrap_system.sh
File bin/bootstrap_system.sh:

http://gerrit.cloudera.org:8080/#/c/11731/4/bin/bootstrap_system.sh@144
PS4, Line 144:   https://www-us.apache.org/dist/maven/maven-3/3.5.4/binaries/apache-maven-3.5.4-bin.tar.gz \
line too long (93 > 90)


http://gerrit.cloudera.org:8080/#/c/11731/4/bin/bootstrap_system.sh@146
PS4, Line 146: redhat sha512sum -c - <<< '2a803f578f341e164f6753e410413d16ab60fabe31dc491d1fe35c984a5cce696bc71f57757d4538fe7738be04065a216f3ebad4ef7e0ce1bb4c51bc36d6be86  apache-maven-3.5.4-bin.tar.gz'
line too long (187 > 90)


http://gerrit.cloudera.org:8080/#/c/11731/4/bin/bootstrap_system.sh@147
PS4, Line 147: redhat sha512sum -c - <<< 'c8321aa223f70d7e64d3d0274263000cfffb46fbea61488534e26f9f0245d99e9872d0888e35cd3274416392a13f80c748c07750caaeffa5f9cae1220020715f  apache-ant-1.9.13-bin.tar.gz'
line too long (186 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/11731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id54294d7607f51de87a9de373dcfc4a33f4bedf5
Gerrit-Change-Number: 11731
Gerrit-PatchSet: 4
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Sat, 27 Oct 2018 16:58:23 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7698: Add centos support to bootstrap system.

Posted by "Jim Apple (Code Review)" <ge...@cloudera.org>.

Jim Apple has posted comments on this change. ( http://gerrit.cloudera.org:8080/11731 )

Change subject: IMPALA-7698: Add centos support to bootstrap_system.
......................................................................


Patch Set 1:

(6 comments)

Phil and Laszlo: thanks so much for doing this!

http://gerrit.cloudera.org:8080/#/c/11731/1//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/11731/1//COMMIT_MSG@33
PS1, Line 33: centos:7 is not addressed by this change. The move to systemd makes "service
nit: long lines


http://gerrit.cloudera.org:8080/#/c/11731/1/bin/bootstrap_system.sh
File bin/bootstrap_system.sh:

http://gerrit.cloudera.org:8080/#/c/11731/1/bin/bootstrap_system.sh@104
PS1, Line 104: REAL_APT_GET=$(ubuntu which apt-get)
Do you want to do something similar for yum? IIRC, i wrote this block because in a non-interactive setting, I expect that connectivity may be temporarily down, and that's not a very informative way for a build to fail.


http://gerrit.cloudera.org:8080/#/c/11731/1/bin/bootstrap_system.sh@143
PS1, Line 143: redhat sudo wget -nv http://www-us.apache.org/dist/maven/maven-3/3.5.4/binaries/apache-maven-3.5.4-bin.tar.gz http://www-us.apache.org/dist/ant/binaries/apache-ant-1.9.13-bin.tar.gz
> line too long (181 > 90)
httpS and check signatures and checksums?


http://gerrit.cloudera.org:8080/#/c/11731/1/bin/bootstrap_system.sh@194
PS1, Line 194: # redhat sudo bash -c "echo host all all 127.0.0.1/32 md5 >> /var/lib/pgsql/data/pg_hba.conf"
> line too long (93 > 90)
I think this line was accidentally left in.

Also, what are the security implications of this compared to what we do on Ubuntu? Is this way more secure?


http://gerrit.cloudera.org:8080/#/c/11731/1/bin/bootstrap_system.sh@274
PS1, Line 274:   SET_JAVA_HOME="export JAVA_HOME=$(echo /usr/lib/jvm/java-1.8.0-openjdk-* | head -n 1)"
Does this not work on Ubuntu, too?

Do you want to warn if there is more than one?


http://gerrit.cloudera.org:8080/#/c/11731/1/docker/entrypoint.sh
File docker/entrypoint.sh:

http://gerrit.cloudera.org:8080/#/c/11731/1/docker/entrypoint.sh@81
PS1, Line 81:       adduser --uid $1 impdev
Do you want to add a comment here about why the command is different for Ubuntu?



-- 
To view, visit http://gerrit.cloudera.org:8080/11731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id54294d7607f51de87a9de373dcfc4a33f4bedf5
Gerrit-Change-Number: 11731
Gerrit-PatchSet: 1
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Sat, 20 Oct 2018 14:02:42 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7698: Add centos support to bootstrap system.

Posted by "Philip Zeyliger (Code Review)" <ge...@cloudera.org>.

Hello Quanlong Huang, Laszlo Gaal, Jim Apple, Joe McDonnell, Csaba Ringhofer, Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/11731

to look at the new patch set (#5).

Change subject: IMPALA-7698: Add centos support to bootstrap_system.
......................................................................

IMPALA-7698: Add centos support to bootstrap_system.

[localhost:21000] default> use functional_orc_def; select * from alltypes;
...
WARNINGS: Encounter parse error: failed to open /usr/share/zoneinfo/GMT-08:00 - No such file or directory.

centos:7 is not addressed by this change. The move to systemd makes
"service sshd start" (and the same for postgresql) not work, and
additional care needs to be done to work around that.

This change is a joint effort with Laszlo Gaal.

Change-Id: Id54294d7607f51de87a9de373dcfc4a33f4bedf5
---
M bin/bootstrap_system.sh
M docker/entrypoint.sh
M docker/test-with-docker.py
3 files changed, 167 insertions(+), 61 deletions(-)

git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/11731/5
--
To view, visit http://gerrit.cloudera.org:8080/11731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id54294d7607f51de87a9de373dcfc4a33f4bedf5
Gerrit-Change-Number: 11731
Gerrit-PatchSet: 5
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>

[Impala-ASF-CR] IMPALA-7698: Add centos support to bootstrap system.

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.

Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/11731 )

Change subject: IMPALA-7698: Add centos support to bootstrap_system.
......................................................................


Patch Set 3:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/11731/1//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/11731/1//COMMIT_MSG@29
PS1, Line 29: I'm not yet clear on why the ORC library is choosing to open that time
            : zone file (which doesn't exist), but this change seems workable even
            : with the 
> Hi Quanlong!
The error is due to hive writes a wrong timezone name into the ORC files. I'm not sure why hive on CentOS will have this problem. Hive writers use TimeZone.getDefault().getID() as the timezone name: https://github.com/cloudera/hive/blob/0ac424f0a17b341efe299da167791112e4a953e9/orc/src/java/org/apache/orc/impl/WriterImpl.java#L782

Could you check whether this help? https://stackoverflow.com/questions/6392/java-time-zone-is-messed-up



-- 
To view, visit http://gerrit.cloudera.org:8080/11731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id54294d7607f51de87a9de373dcfc4a33f4bedf5
Gerrit-Change-Number: 11731
Gerrit-PatchSet: 3
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 24 Oct 2018 04:04:22 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7698: Add centos support to bootstrap system.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11731 )

Change subject: IMPALA-7698: Add centos support to bootstrap_system.
......................................................................


Patch Set 2:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/1139/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/11731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id54294d7607f51de87a9de373dcfc4a33f4bedf5
Gerrit-Change-Number: 11731
Gerrit-PatchSet: 2
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 24 Oct 2018 00:47:50 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7698: Add centos support to bootstrap system.

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.

Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/11731 )

Change subject: IMPALA-7698: Add centos support to bootstrap_system.
......................................................................


Patch Set 4:

Awesome! Good to know the ORC issue is resolved.


-- 
To view, visit http://gerrit.cloudera.org:8080/11731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id54294d7607f51de87a9de373dcfc4a33f4bedf5
Gerrit-Change-Number: 11731
Gerrit-PatchSet: 4
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 25 Oct 2018 04:53:10 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7698: Add centos support to bootstrap system.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11731 )

Change subject: IMPALA-7698: Add centos support to bootstrap_system.
......................................................................


Patch Set 6:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/1166/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/11731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id54294d7607f51de87a9de373dcfc4a33f4bedf5
Gerrit-Change-Number: 11731
Gerrit-PatchSet: 6
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Thu, 25 Oct 2018 21:08:42 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7698: Add centos support to bootstrap system.

Posted by "Philip Zeyliger (Code Review)" <ge...@cloudera.org>.

Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/11731 )

Change subject: IMPALA-7698: Add centos support to bootstrap_system.
......................................................................


Patch Set 1:

(10 comments)

Thanks for the reviews!

http://gerrit.cloudera.org:8080/#/c/11731/1//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/11731/1//COMMIT_MSG@29
PS1, Line 29: I'm not yet clear on why the ORC library is choosing to open that time zone
            : file (which doesn't exist), but this change seems workable even with the test
            : failures.
> At a short glance, it looks like the writer(hive) writes an abnormal timezo
Hi Quanlong!

The ORC files themselves are small (only 50KB) and I pushed them to https://github.com/philz/tmp/blob/master/orc.zip . I've attached them to the JIRA as well. The schema is as follows:

[localhost:21000] default> show create table functional_orc_def.alltypes;
Query: show create table functional_orc_def.alltypes
+-------------------------------------------------------------------+
| result                                                            |
+-------------------------------------------------------------------+
| CREATE EXTERNAL TABLE functional_orc_def.alltypes (               |
|   id INT COMMENT 'Add a comment',                                 |
|   bool_col BOOLEAN,                                               |
|   tinyint_col TINYINT,                                            |
|   smallint_col SMALLINT,                                          |
|   int_col INT,                                                    |
|   bigint_col BIGINT,                                              |
|   float_col FLOAT,                                                |
|   double_col DOUBLE,                                              |
|   date_string_col STRING,                                         |
|   string_col STRING,                                              |
|   timestamp_col TIMESTAMP                                         |
| )                                                                 |
| PARTITIONED BY (                                                  |
|   year INT,                                                       |
|   month INT                                                       |
| )                                                                 |
| STORED AS ORC                                                     |
| LOCATION 'hdfs://localhost:20500/test-warehouse/alltypes_orc_def' |
|                                                                   |
+-------------------------------------------------------------------+
Fetched 1 row(s) in 0.02s


http://gerrit.cloudera.org:8080/#/c/11731/1//COMMIT_MSG@33
PS1, Line 33: centos:7 is not addressed by this change. The move to systemd makes "service
> nit: long lines
Done


http://gerrit.cloudera.org:8080/#/c/11731/1/bin/bootstrap_system.sh
File bin/bootstrap_system.sh:

http://gerrit.cloudera.org:8080/#/c/11731/1/bin/bootstrap_system.sh@104
PS1, Line 104: REAL_APT_GET=$(ubuntu which apt-get)
> Do you want to do something similar for yum? IIRC, i wrote this block becau
Per https://linux.die.net/man/5/yum.conf, yum seems to have a default of 10. In practice, I've seen apt-get fail mostly when a machine is booting and some Ubuntu auto-upgrade thing has the apt lock. I've not yet experienced yum failing similarly.

Anyway, I just added this as a comment.


http://gerrit.cloudera.org:8080/#/c/11731/1/bin/bootstrap_system.sh@131
PS1, Line 131: 	psmisc lsof sudo openssh-server redhat-lsb java-1.8.0-openjdk-devel \
> tab used for whitespace
Done


http://gerrit.cloudera.org:8080/#/c/11731/1/bin/bootstrap_system.sh@143
PS1, Line 143: redhat sudo wget -nv http://www-us.apache.org/dist/maven/maven-3/3.5.4/binaries/apache-maven-3.5.4-bin.tar.gz http://www-us.apache.org/dist/ant/binaries/apache-ant-1.9.13-bin.tar.gz
> httpS and check signatures and checksums?
Done


http://gerrit.cloudera.org:8080/#/c/11731/1/bin/bootstrap_system.sh@143
PS1, Line 143: redhat sudo wget -nv http://www-us.apache.org/dist/maven/maven-3/3.5.4/binaries/apache-maven-3.5.4-bin.tar.gz http://www-us.apache.org/dist/ant/binaries/apache-ant-1.9.13-bin.tar.gz
> line too long (181 > 90)
I'm ignoring a certain amount of long line stuff here because shortening sha512sums and long urls isn't particularly useful.


http://gerrit.cloudera.org:8080/#/c/11731/1/bin/bootstrap_system.sh@194
PS1, Line 194: # redhat sudo bash -c "echo host all all 127.0.0.1/32 md5 >> /var/lib/pgsql/data/pg_hba.conf"
> I think this line was accidentally left in.
It's similar. I updated the comment about security implications, which I think is now truer than the current TODO.


http://gerrit.cloudera.org:8080/#/c/11731/1/bin/bootstrap_system.sh@257
PS1, Line 257: redhat sudo sed -i 's,\*\s*soft\s*nproc\s*1024,* soft nproc unlimited,' /etc/security/limits.d/90-nproc.conf
> line too long (108 > 90)
Done


http://gerrit.cloudera.org:8080/#/c/11731/1/bin/bootstrap_system.sh@274
PS1, Line 274:   SET_JAVA_HOME="export JAVA_HOME=$(echo /usr/lib/jvm/java-1.8.0-openjdk-* | head -n 1)"
> Does this not work on Ubuntu, too?
The paths are slightly different. I could glob it so that "1.8.0" and "-8-" both match, but it doesn't seem right.

I added an assertion that there's only one here.


http://gerrit.cloudera.org:8080/#/c/11731/1/docker/entrypoint.sh
File docker/entrypoint.sh:

http://gerrit.cloudera.org:8080/#/c/11731/1/docker/entrypoint.sh@81
PS1, Line 81:       adduser --uid $1 impdev
> Do you want to add a comment here about why the command is different for Ub
Done. I just said they're different. I can't actually say "why" they're different meaningfully :)



-- 
To view, visit http://gerrit.cloudera.org:8080/11731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id54294d7607f51de87a9de373dcfc4a33f4bedf5
Gerrit-Change-Number: 11731
Gerrit-PatchSet: 1
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 24 Oct 2018 00:13:37 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7698: Add centos support to bootstrap system.

Posted by "Philip Zeyliger (Code Review)" <ge...@cloudera.org>.

Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/11731 )

Change subject: IMPALA-7698: Add centos support to bootstrap_system.
......................................................................


Patch Set 1:

I triggered https://jenkins.impala.io/job/gerrit-verify-dryrun-external/335/ to make sure this doesn't break the existing Ubuntu runs.


-- 
To view, visit http://gerrit.cloudera.org:8080/11731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id54294d7607f51de87a9de373dcfc4a33f4bedf5
Gerrit-Change-Number: 11731
Gerrit-PatchSet: 1
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Thu, 18 Oct 2018 22:57:46 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7698: Add centos support to bootstrap system.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11731 )

Change subject: IMPALA-7698: Add centos support to bootstrap_system.
......................................................................


Patch Set 2:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/11731/2/bin/bootstrap_system.sh
File bin/bootstrap_system.sh:

http://gerrit.cloudera.org:8080/#/c/11731/2/bin/bootstrap_system.sh@145
PS2, Line 145:   https://www-us.apache.org/dist/maven/maven-3/3.5.4/binaries/apache-maven-3.5.4-bin.tar.gz \
line too long (93 > 90)


http://gerrit.cloudera.org:8080/#/c/11731/2/bin/bootstrap_system.sh@147
PS2, Line 147: redhat sha512sum -c - <<< '2a803f578f341e164f6753e410413d16ab60fabe31dc491d1fe35c984a5cce696bc71f57757d4538fe7738be04065a216f3ebad4ef7e0ce1bb4c51bc36d6be86  apache-maven-3.5.4-bin.tar.gz'
line too long (187 > 90)


http://gerrit.cloudera.org:8080/#/c/11731/2/bin/bootstrap_system.sh@148
PS2, Line 148: redhat sha512sum -c - <<< 'c8321aa223f70d7e64d3d0274263000cfffb46fbea61488534e26f9f0245d99e9872d0888e35cd3274416392a13f80c748c07750caaeffa5f9cae1220020715f  apache-ant-1.9.13-bin.tar.gz'
line too long (186 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/11731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id54294d7607f51de87a9de373dcfc4a33f4bedf5
Gerrit-Change-Number: 11731
Gerrit-PatchSet: 2
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Wed, 24 Oct 2018 14:33:01 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7698: Add centos support to bootstrap system.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11731 )

Change subject: IMPALA-7698: Add centos support to bootstrap_system.
......................................................................


Patch Set 1:

(6 comments)

http://gerrit.cloudera.org:8080/#/c/11731/1/bin/bootstrap_system.sh
File bin/bootstrap_system.sh:

http://gerrit.cloudera.org:8080/#/c/11731/1/bin/bootstrap_system.sh@131
PS1, Line 131: 	psmisc lsof sudo openssh-server redhat-lsb java-1.8.0-openjdk-devel \
tab used for whitespace


http://gerrit.cloudera.org:8080/#/c/11731/1/bin/bootstrap_system.sh@143
PS1, Line 143: redhat sudo wget -nv http://www-us.apache.org/dist/maven/maven-3/3.5.4/binaries/apache-maven-3.5.4-bin.tar.gz http://www-us.apache.org/dist/ant/binaries/apache-ant-1.9.13-bin.tar.gz
line too long (181 > 90)


http://gerrit.cloudera.org:8080/#/c/11731/1/bin/bootstrap_system.sh@191
PS1, Line 191: ubuntu sudo sed -ri 's/local +all +all +peer/local all all trust/g' /etc/postgresql/*/main/pg_hba.conf
line too long (102 > 90)


http://gerrit.cloudera.org:8080/#/c/11731/1/bin/bootstrap_system.sh@192
PS1, Line 192: redhat sudo sed -ri 's/local +all +all +ident/local all all trust/g' /var/lib/pgsql/data/pg_hba.conf
line too long (100 > 90)


http://gerrit.cloudera.org:8080/#/c/11731/1/bin/bootstrap_system.sh@194
PS1, Line 194: # redhat sudo bash -c "echo host all all 127.0.0.1/32 md5 >> /var/lib/pgsql/data/pg_hba.conf"
line too long (93 > 90)


http://gerrit.cloudera.org:8080/#/c/11731/1/bin/bootstrap_system.sh@257
PS1, Line 257: redhat sudo sed -i 's,\*\s*soft\s*nproc\s*1024,* soft nproc unlimited,' /etc/security/limits.d/90-nproc.conf
line too long (108 > 90)



-- 
To view, visit http://gerrit.cloudera.org:8080/11731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id54294d7607f51de87a9de373dcfc4a33f4bedf5
Gerrit-Change-Number: 11731
Gerrit-PatchSet: 1
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Fri, 19 Oct 2018 03:19:45 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7698: Add centos support to bootstrap system.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11731 )

Change subject: IMPALA-7698: Add centos support to bootstrap_system.
......................................................................

IMPALA-7698: Add centos support to bootstrap_system.

[localhost:21000] default> use functional_orc_def; select * from alltypes;
...
WARNINGS: Encounter parse error: failed to open /usr/share/zoneinfo/GMT-08:00 - No such file or directory.

centos:7 is not addressed by this change. The move to systemd makes
"service sshd start" (and the same for postgresql) not work, and
additional care needs to be done to work around that.

This change is a joint effort with Laszlo Gaal.

Change-Id: Id54294d7607f51de87a9de373dcfc4a33f4bedf5
Reviewed-on: http://gerrit.cloudera.org:8080/11731
Reviewed-by: Philip Zeyliger <ph...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M bin/bootstrap_system.sh
M docker/entrypoint.sh
M docker/test-with-docker.py
3 files changed, 166 insertions(+), 61 deletions(-)

Approvals:
Philip Zeyliger: Looks good to me, approved
Impala Public Jenkins: Verified

--
To view, visit http://gerrit.cloudera.org:8080/11731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Id54294d7607f51de87a9de373dcfc4a33f4bedf5
Gerrit-Change-Number: 11731
Gerrit-PatchSet: 8
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Jim Apple <jb...@apache.org>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>

[Impala-ASF-CR] IMPALA-7698: Add centos support to bootstrap system.

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.

Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11731 )

Change subject: IMPALA-7698: Add centos support to bootstrap_system.
......................................................................


Patch Set 1:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/1097/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/11731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id54294d7607f51de87a9de373dcfc4a33f4bedf5
Gerrit-Change-Number: 11731
Gerrit-PatchSet: 1
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Thu, 18 Oct 2018 23:41:45 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7698: Add centos support to bootstrap system.

Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.

Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/11731 )

Change subject: IMPALA-7698: Add centos support to bootstrap_system.
......................................................................

Patch Set 1:

(1 comment)

Thanks for this work, Philip! I only have a Ubuntu-12.04 environment. Could you give me the command to run this so I can reproduce the error?

http://gerrit.cloudera.org:8080/#/c/11731/1//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/11731/1//COMMIT_MSG@29
PS1, Line 29: I'm not yet clear on why the ORC library is choosing to open that time zone
: file (which doesn't exist), but this change seems workable even with the test
: failures.
> I see, if these tests do not have to pass, then it is better not to jump in
At a short glance, it looks like the writer(hive) writes an abnormal timezone string (GMT-08:00) in the ORC file. Thus the reader(orc lib) can't find the timezone file. Could you provide the underlying ORC file for me to double check this? This error may can be fixed in the data load phase.

The timestamp type in ORC actually means timestamp with timezone. It really causes some troubles (see ORC-320, ORC-322 or https://github.com/apache/orc/pull/233).

--
To view, visit http://gerrit.cloudera.org:8080/11731
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id54294d7607f51de87a9de373dcfc4a33f4bedf5
Gerrit-Change-Number: 11731
Gerrit-PatchSet: 1
Gerrit-Owner: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Laszlo Gaal <la...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Comment-Date: Sat, 20 Oct 2018 12:05:04 +0000
Gerrit-HasComments: Yes