You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Zach Amsden (Code Review)" <ge...@cloudera.org> on 2017/11/30 00:28:53 UTC
[Impala-ASF-CR] IMPALA-6068: Scale back fixing functional-types
Zach Amsden has uploaded this change for review. ( http://gerrit.cloudera.org:8080/8690
Change subject: IMPALA-6068: Scale back fixing functional-types
......................................................................
IMPALA-6068: Scale back fixing functional-types
I re-created the original patch for IMPALA-6068, but only
performed what I believe to be the limited legal transformation
of data load: DEPENDENT_LOAD -> DEPENDENT_LOAD_HIVE.
Any place that directly uploads via hadoop or hdfs commands
was left alone as changing it can't be proven to be correct.
Change-Id: I6c242cca209a7138b10ad517076707709b5cd204
Testing: Data load in progress - stay tuned.
---
M testdata/bin/generate-schema-statements.py
M testdata/common/widetable.py
M testdata/datasets/functional/functional_schema_template.sql
3 files changed, 61 insertions(+), 38 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/8690/1
--
To view, visit http://gerrit.cloudera.org:8080/8690
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I6c242cca209a7138b10ad517076707709b5cd204
Gerrit-Change-Number: 8690
Gerrit-PatchSet: 1
Gerrit-Owner: Zach Amsden <za...@cloudera.com>
[Impala-ASF-CR] IMPALA-6068: Scale back fixing functional-types
Posted by "Zach Amsden (Code Review)" <ge...@cloudera.org>.
Zach Amsden has posted comments on this change. ( http://gerrit.cloudera.org:8080/8690 )
Change subject: IMPALA-6068: Scale back fixing functional-types
......................................................................
Patch Set 5: Code-Review+2
Rebased, carry the +2
--
To view, visit http://gerrit.cloudera.org:8080/8690
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6c242cca209a7138b10ad517076707709b5cd204
Gerrit-Change-Number: 8690
Gerrit-PatchSet: 5
Gerrit-Owner: Zach Amsden <za...@cloudera.com>
Gerrit-Reviewer: David Knupp <dk...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Zach Amsden <za...@cloudera.com>
Gerrit-Comment-Date: Mon, 04 Dec 2017 23:46:16 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6068: Scale back fixing functional-types
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/8690 )
Change subject: IMPALA-6068: Scale back fixing functional-types
......................................................................
Patch Set 4:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1552/
--
To view, visit http://gerrit.cloudera.org:8080/8690
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6c242cca209a7138b10ad517076707709b5cd204
Gerrit-Change-Number: 8690
Gerrit-PatchSet: 4
Gerrit-Owner: Zach Amsden <za...@cloudera.com>
Gerrit-Reviewer: David Knupp <dk...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Thu, 30 Nov 2017 22:17:07 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6068: Scale back fixing functional-types
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/8690 )
Change subject: IMPALA-6068: Scale back fixing functional-types
......................................................................
Patch Set 4: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/8690
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6c242cca209a7138b10ad517076707709b5cd204
Gerrit-Change-Number: 8690
Gerrit-PatchSet: 4
Gerrit-Owner: Zach Amsden <za...@cloudera.com>
Gerrit-Reviewer: David Knupp <dk...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Zach Amsden <za...@cloudera.com>
Gerrit-Comment-Date: Mon, 04 Dec 2017 22:58:03 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6068: Scale back fixing functional-types
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/8690 )
Change subject: IMPALA-6068: Scale back fixing functional-types
......................................................................
Patch Set 2:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1545/
--
To view, visit http://gerrit.cloudera.org:8080/8690
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6c242cca209a7138b10ad517076707709b5cd204
Gerrit-Change-Number: 8690
Gerrit-PatchSet: 2
Gerrit-Owner: Zach Amsden <za...@cloudera.com>
Gerrit-Reviewer: David Knupp <dk...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Thu, 30 Nov 2017 04:18:15 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6068: Scale back fixing functional-types
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/8690 )
Change subject: IMPALA-6068: Scale back fixing functional-types
......................................................................
Patch Set 4: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/8690
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6c242cca209a7138b10ad517076707709b5cd204
Gerrit-Change-Number: 8690
Gerrit-PatchSet: 4
Gerrit-Owner: Zach Amsden <za...@cloudera.com>
Gerrit-Reviewer: David Knupp <dk...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Zach Amsden <za...@cloudera.com>
Gerrit-Comment-Date: Mon, 04 Dec 2017 23:40:07 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6068: Scale back fixing functional-types
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/8690 )
Change subject: IMPALA-6068: Scale back fixing functional-types
......................................................................
Patch Set 4:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/8690/4/testdata/datasets/functional/functional_schema_template.sql
File testdata/datasets/functional/functional_schema_template.sql:
http://gerrit.cloudera.org:8080/#/c/8690/4/testdata/datasets/functional/functional_schema_template.sql@a2115
PS4, Line 2115:
> What the heck is this actually doing? If I am reading this correctly, it i
The LOAD section populates the text version of the table (functional.table_with_header_2). In this case, a csv file where the first line is column names. The DEPENDENT_LOAD populates other encodings. This table is very hacky. The only encoding other than text is gzip encoded text (the file that determines which formats apply for which tables is testdata/datasets/functional/schema_constraints.csv). We want to preserve the fact that the first line is column names. I don't think Hive would preserve this if we were to create a gzip table and do an "insert as select * from table_with_header_2". So, we load a gzip compressed version of the same file to populate functional_gzip.table_with_header_2. This only works because the only non-text format is gzip compressed text.
--
To view, visit http://gerrit.cloudera.org:8080/8690
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6c242cca209a7138b10ad517076707709b5cd204
Gerrit-Change-Number: 8690
Gerrit-PatchSet: 4
Gerrit-Owner: Zach Amsden <za...@cloudera.com>
Gerrit-Reviewer: David Knupp <dk...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Zach Amsden <za...@cloudera.com>
Gerrit-Comment-Date: Mon, 04 Dec 2017 17:29:11 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6068: Scale back fixing functional-types
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/8690 )
Change subject: IMPALA-6068: Scale back fixing functional-types
......................................................................
Patch Set 1:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1543/
--
To view, visit http://gerrit.cloudera.org:8080/8690
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6c242cca209a7138b10ad517076707709b5cd204
Gerrit-Change-Number: 8690
Gerrit-PatchSet: 1
Gerrit-Owner: Zach Amsden <za...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Comment-Date: Thu, 30 Nov 2017 00:32:41 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6068: Scale back fixing functional-types
Posted by "Zach Amsden (Code Review)" <ge...@cloudera.org>.
Zach Amsden has posted comments on this change. ( http://gerrit.cloudera.org:8080/8690 )
Change subject: IMPALA-6068: Scale back fixing functional-types
......................................................................
Patch Set 4:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/8690/4/testdata/datasets/functional/functional_schema_template.sql
File testdata/datasets/functional/functional_schema_template.sql:
http://gerrit.cloudera.org:8080/#/c/8690/4/testdata/datasets/functional/functional_schema_template.sql@a2115
PS4, Line 2115:
What the heck is this actually doing? If I am reading this correctly, it is loading the data twice, both times with OVERWRITE, so the net effect is simply more time spent doing the data load, with no validation that the results are the same.
--
To view, visit http://gerrit.cloudera.org:8080/8690
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6c242cca209a7138b10ad517076707709b5cd204
Gerrit-Change-Number: 8690
Gerrit-PatchSet: 4
Gerrit-Owner: Zach Amsden <za...@cloudera.com>
Gerrit-Reviewer: David Knupp <dk...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Zach Amsden <za...@cloudera.com>
Gerrit-Comment-Date: Fri, 01 Dec 2017 17:48:44 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6068: Scale back fixing functional-types
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/8690 )
Change subject: IMPALA-6068: Scale back fixing functional-types
......................................................................
Patch Set 4: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/1552/
--
To view, visit http://gerrit.cloudera.org:8080/8690
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6c242cca209a7138b10ad517076707709b5cd204
Gerrit-Change-Number: 8690
Gerrit-PatchSet: 4
Gerrit-Owner: Zach Amsden <za...@cloudera.com>
Gerrit-Reviewer: David Knupp <dk...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Fri, 01 Dec 2017 08:17:21 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6068: Scale back fixing functional-types
Posted by "Zach Amsden (Code Review)" <ge...@cloudera.org>.
Hello Philip Zeyliger, David Knupp, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/8690
to look at the new patch set (#4).
Change subject: IMPALA-6068: Scale back fixing functional-types
......................................................................
IMPALA-6068: Scale back fixing functional-types
I re-created the original patch for IMPALA-6068, but only
performed what I believe to be the limited legal transformation
of data load: DEPENDENT_LOAD -> DEPENDENT_LOAD_HIVE.
Any place that directly uploads via hadoop or hdfs commands
was left alone as changing it can't be proven to be correct.
Change-Id: I6c242cca209a7138b10ad517076707709b5cd204
Testing: Doing a full data load. I mistakenly changed a variable
name causing the first two dry-runs to fail.
---
M testdata/bin/generate-schema-statements.py
M testdata/common/widetable.py
M testdata/datasets/functional/functional_schema_template.sql
3 files changed, 61 insertions(+), 38 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/8690/4
--
To view, visit http://gerrit.cloudera.org:8080/8690
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6c242cca209a7138b10ad517076707709b5cd204
Gerrit-Change-Number: 8690
Gerrit-PatchSet: 4
Gerrit-Owner: Zach Amsden <za...@cloudera.com>
Gerrit-Reviewer: David Knupp <dk...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
[Impala-ASF-CR] IMPALA-6068: Scale back fixing functional-types
Posted by "Zach Amsden (Code Review)" <ge...@cloudera.org>.
Zach Amsden has posted comments on this change. ( http://gerrit.cloudera.org:8080/8690 )
Change subject: IMPALA-6068: Scale back fixing functional-types
......................................................................
Patch Set 5: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/8690
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6c242cca209a7138b10ad517076707709b5cd204
Gerrit-Change-Number: 8690
Gerrit-PatchSet: 5
Gerrit-Owner: Zach Amsden <za...@cloudera.com>
Gerrit-Reviewer: David Knupp <dk...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Zach Amsden <za...@cloudera.com>
Gerrit-Comment-Date: Mon, 04 Dec 2017 23:46:22 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6068: Scale back fixing functional-types
Posted by "Zach Amsden (Code Review)" <ge...@cloudera.org>.
Zach Amsden has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/8690 )
Change subject: IMPALA-6068: Scale back fixing functional-types
......................................................................
IMPALA-6068: Scale back fixing functional-types
I re-created the original patch for IMPALA-6068, but only
performed what I believe to be the limited legal transformation
of data load: DEPENDENT_LOAD -> DEPENDENT_LOAD_HIVE.
Any place that directly uploads via hadoop or hdfs commands
was left alone as changing it can't be proven to be correct.
Change-Id: I6c242cca209a7138b10ad517076707709b5cd204
Testing: Doing a full data load. I mistakenly changed a variable
name causing the first two dry-runs to fail.
Reviewed-on: http://gerrit.cloudera.org:8080/8690
Reviewed-by: Zach Amsden <za...@cloudera.com>
Tested-by: Zach Amsden <za...@cloudera.com>
---
M testdata/bin/generate-schema-statements.py
M testdata/common/widetable.py
M testdata/datasets/functional/functional_schema_template.sql
3 files changed, 61 insertions(+), 38 deletions(-)
Approvals:
Zach Amsden: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/8690
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I6c242cca209a7138b10ad517076707709b5cd204
Gerrit-Change-Number: 8690
Gerrit-PatchSet: 6
Gerrit-Owner: Zach Amsden <za...@cloudera.com>
Gerrit-Reviewer: David Knupp <dk...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Zach Amsden <za...@cloudera.com>
[Impala-ASF-CR] IMPALA-6068: Scale back fixing functional-types
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/8690 )
Change subject: IMPALA-6068: Scale back fixing functional-types
......................................................................
Patch Set 4:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1562/
--
To view, visit http://gerrit.cloudera.org:8080/8690
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6c242cca209a7138b10ad517076707709b5cd204
Gerrit-Change-Number: 8690
Gerrit-PatchSet: 4
Gerrit-Owner: Zach Amsden <za...@cloudera.com>
Gerrit-Reviewer: David Knupp <dk...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Zach Amsden <za...@cloudera.com>
Gerrit-Comment-Date: Mon, 04 Dec 2017 19:26:49 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6068: Scale back fixing functional-types
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/8690 )
Change subject: IMPALA-6068: Scale back fixing functional-types
......................................................................
Patch Set 3:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1548/
--
To view, visit http://gerrit.cloudera.org:8080/8690
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6c242cca209a7138b10ad517076707709b5cd204
Gerrit-Change-Number: 8690
Gerrit-PatchSet: 3
Gerrit-Owner: Zach Amsden <za...@cloudera.com>
Gerrit-Reviewer: David Knupp <dk...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Thu, 30 Nov 2017 17:55:09 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6068: Scale back fixing functional-types
Posted by "Zach Amsden (Code Review)" <ge...@cloudera.org>.
Zach Amsden has posted comments on this change. ( http://gerrit.cloudera.org:8080/8690 )
Change subject: IMPALA-6068: Scale back fixing functional-types
......................................................................
Patch Set 4:
Finally! Anyone care to review?
--
To view, visit http://gerrit.cloudera.org:8080/8690
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6c242cca209a7138b10ad517076707709b5cd204
Gerrit-Change-Number: 8690
Gerrit-PatchSet: 4
Gerrit-Owner: Zach Amsden <za...@cloudera.com>
Gerrit-Reviewer: David Knupp <dk...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Zach Amsden <za...@cloudera.com>
Gerrit-Comment-Date: Mon, 04 Dec 2017 23:09:42 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6068: Scale back fixing functional-types
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/8690 )
Change subject: IMPALA-6068: Scale back fixing functional-types
......................................................................
Patch Set 2: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/1545/
--
To view, visit http://gerrit.cloudera.org:8080/8690
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6c242cca209a7138b10ad517076707709b5cd204
Gerrit-Change-Number: 8690
Gerrit-PatchSet: 2
Gerrit-Owner: Zach Amsden <za...@cloudera.com>
Gerrit-Reviewer: David Knupp <dk...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Thu, 30 Nov 2017 07:53:06 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6068: Scale back fixing functional-types
Posted by "Zach Amsden (Code Review)" <ge...@cloudera.org>.
Hello Philip Zeyliger, David Knupp, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/8690
to look at the new patch set (#3).
Change subject: IMPALA-6068: Scale back fixing functional-types
......................................................................
IMPALA-6068: Scale back fixing functional-types
I re-created the original patch for IMPALA-6068, but only
performed what I believe to be the limited legal transformation
of data load: DEPENDENT_LOAD -> DEPENDENT_LOAD_HIVE.
Any place that directly uploads via hadoop or hdfs commands
was left alone as changing it can't be proven to be correct.
Change-Id: I6c242cca209a7138b10ad517076707709b5cd204
Testing: Doing a full data load. I mistakenly changed a variable
name causing the first two dry-runs to fail.
---
M testdata/bin/generate-schema-statements.py
M testdata/common/widetable.py
M testdata/datasets/functional/functional_schema_template.sql
3 files changed, 61 insertions(+), 38 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/8690/3
--
To view, visit http://gerrit.cloudera.org:8080/8690
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6c242cca209a7138b10ad517076707709b5cd204
Gerrit-Change-Number: 8690
Gerrit-PatchSet: 3
Gerrit-Owner: Zach Amsden <za...@cloudera.com>
Gerrit-Reviewer: David Knupp <dk...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
[Impala-ASF-CR] IMPALA-6068: Scale back fixing functional-types
Posted by "Zach Amsden (Code Review)" <ge...@cloudera.org>.
Zach Amsden has posted comments on this change. ( http://gerrit.cloudera.org:8080/8690 )
Change subject: IMPALA-6068: Scale back fixing functional-types
......................................................................
Patch Set 4:
(1 comment)
22:54:51 FAILED (Took: 11 min 53 sec)
22:54:51 'load-data functional-query exhaustive' failed. Tail of log:
22:54:51 c1 int,
22:54:51 c2 double
22:54:51 )
22:54:51 ROW FORMAT delimited fields terminated by ',' escaped by '\\'
22:54:51 STORED AS TEXTFILE
22:54:51 LOCATION '/test-warehouse/table_with_header'
22:54:51
22:54:51 (load-functional-query-exhaustive-impala-generated-text-none-none.sql):
22:54:51 USE functional
22:54:51
22:54:51 (load-functional-query-exhaustive-impala-generated-text-none-none.sql):
22:54:51 ALTER TABLE table_with_header SET TBLPROPERTIES('skip.header.line.count'='1')
22:54:51
22:54:51 (load-functional-query-exhaustive-impala-generated-text-none-none.sql):
22:54:51 CREATE DATABASE IF NOT EXISTS functional
22:54:51
22:54:51 (load-functional-query-exhaustive-impala-generated-text-none-none.sql):
22:54:51 CREATE EXTERNAL TABLE IF NOT EXISTS functional.table_with_header_2 (
22:54:51 c1 int,
22:54:51 c2 double
22:54:51 )
22:54:51 ROW FORMAT delimited fields terminated by ',' escaped by '\\'
22:54:51 STORED AS TEXTFILE
22:54:51 LOCATION '/test-warehouse/table_with_header_2'
22:54:51
22:54:51 (load-functional-query-exhaustive-impala-generated-text-none-none.sql):
22:54:51 USE functional
22:54:51
22:54:51 (load-functional-query-exhaustive-impala-generated-text-none-none.sql):
22:54:51 ALTER TABLE table_with_header_2 SET TBLPROPERTIES('skip.header.line.count'='2')
22:54:51
22:54:51 Data Loading from Impala failed with error: ImpalaBeeswaxException:
22:54:51 INNER EXCEPTION: <class 'beeswaxd.ttypes.BeeswaxException'>
22:54:51 MESSAGE: AnalysisException: Could not resolve table reference: 'table_with_header_2'
22:54:51
22:54:51 Traceback (most recent call last):
22:54:51 File "/home/ubuntu/Impala/bin/load-data.py", line 178, in exec_impala_query_from_file
22:54:51 result = impala_client.execute(query)
22:54:51 File "/home/ubuntu/Impala/tests/beeswax/impala_beeswax.py", line 173, in execute
22:54:51 handle = self.__execute_query(query_string.strip(), user=user)
22:54:51 File "/home/ubuntu/Impala/tests/beeswax/impala_beeswax.py", line 339, in __execute_query
22:54:51 handle = self.execute_query_async(query_string, user=user)
22:54:51 File "/home/ubuntu/Impala/tests/beeswax/impala_beeswax.py", line 335, in execute_query_async
22:54:51 return self.__do_rpc(lambda: self.imp_service.query(query,))
22:54:51 File "/home/ubuntu/Impala/tests/beeswax/impala_beeswax.py", line 460, in __do_rpc
22:54:51 raise ImpalaBeeswaxException(self.__build_error_message(b), b)
22:54:51 ImpalaBeeswaxException: ImpalaBeeswaxException:
22:54:51 INNER EXCEPTION: <class 'beeswaxd.ttypes.BeeswaxException'>
22:54:51 MESSAGE: AnalysisException: Could not resolve table reference: 'table_with_header_2'
22:54:51
22:54:51 Background task Loading functional-query data (pid 69352) failed.
http://gerrit.cloudera.org:8080/#/c/8690/3/testdata/bin/generate-schema-statements.py
File testdata/bin/generate-schema-statements.py:
http://gerrit.cloudera.org:8080/#/c/8690/3/testdata/bin/generate-schema-statements.py@531
PS3, Line 531: assert not (insert and insert_hive),\
s/load_hive/insert_hive
My bad. Trying again.
--
To view, visit http://gerrit.cloudera.org:8080/8690
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6c242cca209a7138b10ad517076707709b5cd204
Gerrit-Change-Number: 8690
Gerrit-PatchSet: 4
Gerrit-Owner: Zach Amsden <za...@cloudera.com>
Gerrit-Reviewer: David Knupp <dk...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Zach Amsden <za...@cloudera.com>
Gerrit-Comment-Date: Fri, 01 Dec 2017 17:33:09 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6068: Scale back fixing functional-types
Posted by "Zach Amsden (Code Review)" <ge...@cloudera.org>.
Zach Amsden has posted comments on this change. ( http://gerrit.cloudera.org:8080/8690 )
Change subject: IMPALA-6068: Scale back fixing functional-types
......................................................................
Patch Set 4:
I still can't see anything that would actually change based on this patch, so either I'm being dense, or GVO is exposing a pre-existing problem here.
Let's try again...
--
To view, visit http://gerrit.cloudera.org:8080/8690
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6c242cca209a7138b10ad517076707709b5cd204
Gerrit-Change-Number: 8690
Gerrit-PatchSet: 4
Gerrit-Owner: Zach Amsden <za...@cloudera.com>
Gerrit-Reviewer: David Knupp <dk...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Reviewer: Zach Amsden <za...@cloudera.com>
Gerrit-Comment-Date: Mon, 04 Dec 2017 19:26:27 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6068: Scale back fixing functional-types
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/8690 )
Change subject: IMPALA-6068: Scale back fixing functional-types
......................................................................
Patch Set 3: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/1548/
--
To view, visit http://gerrit.cloudera.org:8080/8690
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6c242cca209a7138b10ad517076707709b5cd204
Gerrit-Change-Number: 8690
Gerrit-PatchSet: 3
Gerrit-Owner: Zach Amsden <za...@cloudera.com>
Gerrit-Reviewer: David Knupp <dk...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Thu, 30 Nov 2017 21:26:08 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6068: Scale back fixing functional-types
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/8690 )
Change subject: IMPALA-6068: Scale back fixing functional-types
......................................................................
Patch Set 1: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/1543/
--
To view, visit http://gerrit.cloudera.org:8080/8690
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6c242cca209a7138b10ad517076707709b5cd204
Gerrit-Change-Number: 8690
Gerrit-PatchSet: 1
Gerrit-Owner: Zach Amsden <za...@cloudera.com>
Gerrit-Reviewer: David Knupp <dk...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Philip Zeyliger <ph...@cloudera.com>
Gerrit-Comment-Date: Thu, 30 Nov 2017 04:09:45 +0000
Gerrit-HasComments: No