You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Thomas Tauber-Marshall (Code Review)" <ge...@cloudera.org> on 2017/06/05 01:54:38 UTC
[Impala-ASF-CR] IMPALA-5428: update external hadoop ecosystem versions
Thomas Tauber-Marshall has uploaded a new change for review.
http://gerrit.cloudera.org:8080/7076
Change subject: IMPALA-5428: update external hadoop ecosystem versions
......................................................................
IMPALA-5428: update external hadoop ecosystem versions
This patch updates the versions of the Hadoop ecosystem components that
we used to run system/integration tests with Impala - Hadoop, Sentry,
HBase, Hive, Parquet, and Kudu.
The necessary tarballs are in S3, and I privately tested that an
exhaustive build+tests succeeds.
Change-Id: Ib96c7532d381e1527dfc653709c6ecba3bbd16c4
---
M bin/impala-config.sh
1 file changed, 6 insertions(+), 6 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/76/7076/1
--
To view, visit http://gerrit.cloudera.org:8080/7076
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ib96c7532d381e1527dfc653709c6ecba3bbd16c4
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Thomas Tauber-Marshall <tm...@cloudera.com>
[Impala-ASF-CR] IMPALA-5428: update external hadoop ecosystem versions
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change.
Change subject: IMPALA-5428: update external hadoop ecosystem versions
......................................................................
Patch Set 2: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/7076
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib96c7532d381e1527dfc653709c6ecba3bbd16c4
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5428: update external hadoop ecosystem versions
Posted by "Lars Volker (Code Review)" <ge...@cloudera.org>.
Lars Volker has posted comments on this change.
Change subject: IMPALA-5428: update external hadoop ecosystem versions
......................................................................
Patch Set 1:
> Is this change affected by the Hive schema version problem?
I tried this locally and after cherry picking this change, Hive wouldn't start anymore. I suspect it will take https://gerrit.cloudera.org/#/c/7072/ and a data reload to get it to work again.
--
To view, visit http://gerrit.cloudera.org:8080/7076
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib96c7532d381e1527dfc653709c6ecba3bbd16c4
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Michael Brown <mi...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5428: update external hadoop ecosystem versions
Posted by "Lars Volker (Code Review)" <ge...@cloudera.org>.
Lars Volker has posted comments on this change.
Change subject: IMPALA-5428: update external hadoop ecosystem versions
......................................................................
Patch Set 1:
Is this change affected by the Hive schema version problem?
--
To view, visit http://gerrit.cloudera.org:8080/7076
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib96c7532d381e1527dfc653709c6ecba3bbd16c4
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Michael Brown <mi...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5428: update external hadoop ecosystem versions
Posted by "Thomas Tauber-Marshall (Code Review)" <ge...@cloudera.org>.
Thomas Tauber-Marshall has posted comments on this change.
Change subject: IMPALA-5428: update external hadoop ecosystem versions
......................................................................
Patch Set 2: Code-Review+2
gvo failed because it needed to be rebased
--
To view, visit http://gerrit.cloudera.org:8080/7076
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib96c7532d381e1527dfc653709c6ecba3bbd16c4
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5428: update external hadoop ecosystem versions
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change.
Change subject: IMPALA-5428: update external hadoop ecosystem versions
......................................................................
Patch Set 1: Verified-1
Build failed: http://jenkins.impala.io:8080/job/gerrit-verify-dryrun/681/
--
To view, visit http://gerrit.cloudera.org:8080/7076
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib96c7532d381e1527dfc653709c6ecba3bbd16c4
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5428: update external hadoop ecosystem versions
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change.
Change subject: IMPALA-5428: update external hadoop ecosystem versions
......................................................................
Patch Set 1:
Build started: http://jenkins.impala.io:8080/job/gerrit-verify-dryrun/681/
--
To view, visit http://gerrit.cloudera.org:8080/7076
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib96c7532d381e1527dfc653709c6ecba3bbd16c4
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5428: update external hadoop ecosystem versions
Posted by "Michael Brown (Code Review)" <ge...@cloudera.org>.
Michael Brown has posted comments on this change.
Change subject: IMPALA-5428: update external hadoop ecosystem versions
......................................................................
Patch Set 1: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/7076
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib96c7532d381e1527dfc653709c6ecba3bbd16c4
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Michael Brown <mi...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5428: update external hadoop ecosystem versions
Posted by "Lars Volker (Code Review)" <ge...@cloudera.org>.
Lars Volker has posted comments on this change.
Change subject: IMPALA-5428: update external hadoop ecosystem versions
......................................................................
Patch Set 1: Code-Review+2
> Lars - sorry to not be clearer here. Yes, this patch will have to
> go in after the Hive schema bump, and then a metastore reload will
> be required.
After my yesterdays experiments it looks like the schematool can be used to upgrade the schema and get away without having to reload the metastore from scratch.
--
To view, visit http://gerrit.cloudera.org:8080/7076
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib96c7532d381e1527dfc653709c6ecba3bbd16c4
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5428: update external hadoop ecosystem versions
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged.
Change subject: IMPALA-5428: update external hadoop ecosystem versions
......................................................................
IMPALA-5428: update external hadoop ecosystem versions
This patch updates the versions of the Hadoop ecosystem components that
we used to run system/integration tests with Impala - Hadoop, Sentry,
HBase, Hive, Parquet, and Kudu.
The necessary tarballs are in S3, and I privately tested that an
exhaustive build+tests succeeds.
Change-Id: Ib96c7532d381e1527dfc653709c6ecba3bbd16c4
Reviewed-on: http://gerrit.cloudera.org:8080/7076
Reviewed-by: Thomas Tauber-Marshall <tm...@cloudera.com>
Tested-by: Impala Public Jenkins
---
M bin/impala-config.sh
1 file changed, 6 insertions(+), 6 deletions(-)
Approvals:
Impala Public Jenkins: Verified
Thomas Tauber-Marshall: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/7076
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Ib96c7532d381e1527dfc653709c6ecba3bbd16c4
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
[Impala-ASF-CR] IMPALA-5428: update external hadoop ecosystem versions
Posted by "Thomas Tauber-Marshall (Code Review)" <ge...@cloudera.org>.
Thomas Tauber-Marshall has posted comments on this change.
Change subject: IMPALA-5428: update external hadoop ecosystem versions
......................................................................
Patch Set 1:
Lars - sorry to not be clearer here. Yes, this patch will have to go in after the Hive schema bump, and then a metastore reload will be required.
--
To view, visit http://gerrit.cloudera.org:8080/7076
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib96c7532d381e1527dfc653709c6ecba3bbd16c4
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5428: update external hadoop ecosystem versions
Posted by "Lars Volker (Code Review)" <ge...@cloudera.org>.
Lars Volker has posted comments on this change.
Change subject: IMPALA-5428: update external hadoop ecosystem versions
......................................................................
Patch Set 1:
> > Is this change affected by the Hive schema version problem?
>
> I tried this locally and after cherry picking this change, Hive
> wouldn't start anymore. I suspect it will take https://gerrit.cloudera.org/#/c/7072/
> and a data reload to get it to work again.
I was able to use the Hive schematool to upgrade my schema like so:
/opt/Impala-Toolchain/cdh_components/hive-1.1.0-cdh5.13.0-SNAPSHOT/bin/schematool -upgradeSchema -dbType postgres
It printed an error but still seems to have worked:
✔ ~/i1(master) ✗$ /opt/Impala-Toolchain/cdh_components/hive-1.1.0-cdh5.13.0-SNAPSHOT/bin/schematool -upgradeSchema -dbType postgres
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/Impala-Toolchain/cdh_components/hbase-1.2.0-cdh5.13.0-SNAPSHOT/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/Impala-Toolchain/cdh_components/hadoop-2.6.0-cdh5.13.0-SNAPSHOT/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
17/06/04 22:20:00 WARN conf.HiveConf: HiveConf of name hive.access.conf.url does not exist
Comparing cdh5.12.0 with cdh5.13.0
Version 12 is lower than 13
Comparing cdh5.13.0 with cdh5.13.0
Metastore connection URL: jdbc:postgresql://localhost:5432/hive_impala
Metastore Connection Driver : org.postgresql.Driver
Metastore connection User: hiveuser
Starting upgrade metastore schema from version 1.1.0 to 1.1.0-cdh5.13.0
Upgrade script upgrade-1.1.0-to-1.1.0-cdh5.12.0.postgres.sql
Error: ERROR: column "SCHEMA_VERSION_V2" of relation "VERSION" does not exist (state=42703,code=0)
Warning in pre-upgrade script pre-0-upgrade-1.1.0-to-1.1.0-cdh5.12.0.postgres.sql: Schema script failed, errorcode 2
Completed upgrade-1.1.0-to-1.1.0-cdh5.12.0.postgres.sql
Upgrade script upgrade-1.1.0-cdh5.12.0-to-1.1.0-cdh5.13.0.postgres.sql
Completed upgrade-1.1.0-cdh5.12.0-to-1.1.0-cdh5.13.0.postgres.sql
17/06/04 22:20:00 INFO metastore.CDHMetaStoreSchemaInfo: Upstream versions are compatible, comparing downstream
Comparing cdh5.13.0 with cdh5.13.0
schemaTool completed
--
To view, visit http://gerrit.cloudera.org:8080/7076
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib96c7532d381e1527dfc653709c6ecba3bbd16c4
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Michael Brown <mi...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5428: update external hadoop ecosystem versions
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change.
Change subject: IMPALA-5428: update external hadoop ecosystem versions
......................................................................
Patch Set 2:
Build started: http://jenkins.impala.io:8080/job/gerrit-verify-dryrun/685/
--
To view, visit http://gerrit.cloudera.org:8080/7076
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib96c7532d381e1527dfc653709c6ecba3bbd16c4
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-HasComments: No