You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Todd Lipcon (Code Review)" <ge...@cloudera.org> on 2018/07/11 00:45:11 UTC
[Impala-ASF-CR] IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
Hello Tianyi Wang, Vuk Ercegovac,
I'd like you to do a code review. Please visit
http://gerrit.cloudera.org:8080/10913
to review the following change.
Change subject: IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
......................................................................
IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
This fixed most of the remaining Kudu tests which relied on CTAS. Now only a
few Kudu tests fail:
FAIL query_test/test_kudu.py::TestKuduOperations::()::test_kudu_col_changed
FAIL query_test/test_kudu.py::TestKuduOperations::()::test_kudu_col_null_changed
FAIL query_test/test_kudu.py::TestKuduOperations::()::test_kudu_col_not_null_changed
FAIL query_test/test_kudu.py::TestKuduOperations::()::test_kudu_col_added
The above 4 fail because they are asserting something about the caching
behavior of the old catalog implementation.
FAIL query_test/test_kudu.py::TestImpalaKuduIntegration::()::test_delete_external_kudu_table
FAIL query_test/test_kudu.py::TestImpalaKuduIntegration::()::test_delete_managed_kudu_table
These fail due to attempting to load non-existent tables referred to by a
DELETE statement. Need to investigate these further, but not related to CTAS.
Change-Id: I93937aed9b76ef6a62b1c588c59c34d3d6831a46
---
M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java
M fe/src/main/java/org/apache/impala/catalog/Db.java
M fe/src/main/java/org/apache/impala/catalog/FeDb.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalKuduTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
M fe/src/main/java/org/apache/impala/planner/TableSink.java
M fe/src/main/java/org/apache/impala/util/KuduUtil.java
10 files changed, 137 insertions(+), 28 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/13/10913/1
--
To view, visit http://gerrit.cloudera.org:8080/10913
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I93937aed9b76ef6a62b1c588c59c34d3d6831a46
Gerrit-Change-Number: 10913
Gerrit-PatchSet: 1
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Tianyi Wang <tw...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
[Impala-ASF-CR] IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
Posted by "Vuk Ercegovac (Code Review)" <ge...@cloudera.org>.
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/10913 )
Change subject: IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
......................................................................
Patch Set 3: Code-Review+1
(2 comments)
lgtm. I'll let Csaba have another look.
http://gerrit.cloudera.org:8080/#/c/10913/3/fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java
File fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java:
http://gerrit.cloudera.org:8080/#/c/10913/3/fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java@188
PS3, Line 188: the
nit: remove second 'the'
http://gerrit.cloudera.org:8080/#/c/10913/3/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
File fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java:
http://gerrit.cloudera.org:8080/#/c/10913/3/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java@114
PS3, Line 114: load them?
what would be loaded... this is a new table if I've understood it correctly?
--
To view, visit http://gerrit.cloudera.org:8080/10913
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I93937aed9b76ef6a62b1c588c59c34d3d6831a46
Gerrit-Change-Number: 10913
Gerrit-PatchSet: 3
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Tianyi Wang <tw...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Wed, 18 Jul 2018 23:54:50 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10913 )
Change subject: IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
......................................................................
Patch Set 4:
Build Started https://jenkins.impala.io/job/gerrit-code-review-checks/18/
Running initial code review checks. This is experimental - please report any issues to tarmstrong@cloudera.com or on this JIRA: IMPALA-7317
--
To view, visit http://gerrit.cloudera.org:8080/10913
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I93937aed9b76ef6a62b1c588c59c34d3d6831a46
Gerrit-Change-Number: 10913
Gerrit-PatchSet: 4
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tianyi Wang <tw...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Mon, 23 Jul 2018 23:48:45 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10913 )
Change subject: IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
......................................................................
IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
This fixed most of the remaining Kudu tests which relied on CTAS. Now only a
few Kudu tests fail:
FAIL query_test/test_kudu.py::TestKuduOperations::()::test_kudu_col_changed
FAIL query_test/test_kudu.py::TestKuduOperations::()::test_kudu_col_null_changed
FAIL query_test/test_kudu.py::TestKuduOperations::()::test_kudu_col_not_null_changed
FAIL query_test/test_kudu.py::TestKuduOperations::()::test_kudu_col_added
The above 4 fail because they are asserting something about the caching
behavior of the old catalog implementation.
FAIL query_test/test_kudu.py::TestImpalaKuduIntegration::()::test_delete_external_kudu_table
FAIL query_test/test_kudu.py::TestImpalaKuduIntegration::()::test_delete_managed_kudu_table
These fail due to attempting to load non-existent tables referred to by a
DELETE statement. Need to investigate these further, but not related to CTAS.
Change-Id: I93937aed9b76ef6a62b1c588c59c34d3d6831a46
Reviewed-on: http://gerrit.cloudera.org:8080/10913
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Todd Lipcon <to...@apache.org>
---
M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java
M fe/src/main/java/org/apache/impala/catalog/Db.java
M fe/src/main/java/org/apache/impala/catalog/FeDb.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalKuduTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
M fe/src/main/java/org/apache/impala/planner/TableSink.java
9 files changed, 135 insertions(+), 28 deletions(-)
Approvals:
Impala Public Jenkins: Verified
Todd Lipcon: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/10913
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I93937aed9b76ef6a62b1c588c59c34d3d6831a46
Gerrit-Change-Number: 10913
Gerrit-PatchSet: 7
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tianyi Wang <tw...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
[Impala-ASF-CR] IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/10913 )
Change subject: IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
......................................................................
Patch Set 3:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/10913/3/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
File fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java:
http://gerrit.cloudera.org:8080/#/c/10913/3/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java@114
PS3, Line 114: load them?
> ok, so just for future proofing.
yep. improved the TODO to be more clear what I meant.
--
To view, visit http://gerrit.cloudera.org:8080/10913
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I93937aed9b76ef6a62b1c588c59c34d3d6831a46
Gerrit-Change-Number: 10913
Gerrit-PatchSet: 3
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Tianyi Wang <tw...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Mon, 23 Jul 2018 23:01:14 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/10913 )
Change subject: IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
......................................................................
Patch Set 3:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/10913/3/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
File fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java:
http://gerrit.cloudera.org:8080/#/c/10913/3/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java@114
PS3, Line 114: load them?
> makes sense... I'd prefer the flag to guard.
nope, this would only show up temporarily in the analyzer during analysis of a CREATE TABLE AS SELECT statement -- it never gets inserted into any Db and never sticks around outside the scope of a single CTAS
--
To view, visit http://gerrit.cloudera.org:8080/10913
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I93937aed9b76ef6a62b1c588c59c34d3d6831a46
Gerrit-Change-Number: 10913
Gerrit-PatchSet: 3
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Tianyi Wang <tw...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Thu, 19 Jul 2018 00:21:01 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10913 )
Change subject: IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
......................................................................
Patch Set 5: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/2857/
--
To view, visit http://gerrit.cloudera.org:8080/10913
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I93937aed9b76ef6a62b1c588c59c34d3d6831a46
Gerrit-Change-Number: 10913
Gerrit-PatchSet: 5
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tianyi Wang <tw...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Tue, 24 Jul 2018 22:54:50 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/10913 )
Change subject: IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
......................................................................
Patch Set 5: Code-Review+1
lgtm, but I do not feel the confidence to give +2
--
To view, visit http://gerrit.cloudera.org:8080/10913
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I93937aed9b76ef6a62b1c588c59c34d3d6831a46
Gerrit-Change-Number: 10913
Gerrit-PatchSet: 5
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tianyi Wang <tw...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Tue, 24 Jul 2018 16:39:37 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10913 )
Change subject: IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
......................................................................
Patch Set 6: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/10913
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I93937aed9b76ef6a62b1c588c59c34d3d6831a46
Gerrit-Change-Number: 10913
Gerrit-PatchSet: 6
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tianyi Wang <tw...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Wed, 25 Jul 2018 05:22:15 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
Posted by "Vuk Ercegovac (Code Review)" <ge...@cloudera.org>.
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/10913 )
Change subject: IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
......................................................................
Patch Set 3:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/10913/3/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
File fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java:
http://gerrit.cloudera.org:8080/#/c/10913/3/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java@114
PS3, Line 114: load them?
> yea, the target table is sort of an odd temporary catalog entry which doesn
makes sense... I'd prefer the flag to guard.
so this can end up in a "show tables" and from there, such a load could be requested? if that's the case, seems like a hidden table scheme for tmp could make sense.
--
To view, visit http://gerrit.cloudera.org:8080/10913
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I93937aed9b76ef6a62b1c588c59c34d3d6831a46
Gerrit-Change-Number: 10913
Gerrit-PatchSet: 3
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Tianyi Wang <tw...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Thu, 19 Jul 2018 00:19:34 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10913 )
Change subject: IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
......................................................................
Patch Set 4:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/18/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/10913
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I93937aed9b76ef6a62b1c588c59c34d3d6831a46
Gerrit-Change-Number: 10913
Gerrit-PatchSet: 4
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tianyi Wang <tw...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Tue, 24 Jul 2018 00:37:14 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Hello Tianyi Wang, Csaba Ringhofer, Vuk Ercegovac,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/10913
to look at the new patch set (#4).
Change subject: IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
......................................................................
IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
This fixed most of the remaining Kudu tests which relied on CTAS. Now only a
few Kudu tests fail:
FAIL query_test/test_kudu.py::TestKuduOperations::()::test_kudu_col_changed
FAIL query_test/test_kudu.py::TestKuduOperations::()::test_kudu_col_null_changed
FAIL query_test/test_kudu.py::TestKuduOperations::()::test_kudu_col_not_null_changed
FAIL query_test/test_kudu.py::TestKuduOperations::()::test_kudu_col_added
The above 4 fail because they are asserting something about the caching
behavior of the old catalog implementation.
FAIL query_test/test_kudu.py::TestImpalaKuduIntegration::()::test_delete_external_kudu_table
FAIL query_test/test_kudu.py::TestImpalaKuduIntegration::()::test_delete_managed_kudu_table
These fail due to attempting to load non-existent tables referred to by a
DELETE statement. Need to investigate these further, but not related to CTAS.
Change-Id: I93937aed9b76ef6a62b1c588c59c34d3d6831a46
---
M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java
M fe/src/main/java/org/apache/impala/catalog/Db.java
M fe/src/main/java/org/apache/impala/catalog/FeDb.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalKuduTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
M fe/src/main/java/org/apache/impala/planner/TableSink.java
9 files changed, 135 insertions(+), 28 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/13/10913/4
--
To view, visit http://gerrit.cloudera.org:8080/10913
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I93937aed9b76ef6a62b1c588c59c34d3d6831a46
Gerrit-Change-Number: 10913
Gerrit-PatchSet: 4
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Tianyi Wang <tw...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
[Impala-ASF-CR] IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
Posted by "Vuk Ercegovac (Code Review)" <ge...@cloudera.org>.
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/10913 )
Change subject: IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
......................................................................
Patch Set 3:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/10913/3/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
File fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java:
http://gerrit.cloudera.org:8080/#/c/10913/3/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java@114
PS3, Line 114: load them?
> nope, this would only show up temporarily in the analyzer during analysis o
ok, so just for future proofing.
--
To view, visit http://gerrit.cloudera.org:8080/10913
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I93937aed9b76ef6a62b1c588c59c34d3d6831a46
Gerrit-Change-Number: 10913
Gerrit-PatchSet: 3
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Tianyi Wang <tw...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Thu, 19 Jul 2018 01:10:56 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10913 )
Change subject: IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
......................................................................
Patch Set 5:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2857/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/10913
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I93937aed9b76ef6a62b1c588c59c34d3d6831a46
Gerrit-Change-Number: 10913
Gerrit-PatchSet: 5
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tianyi Wang <tw...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Tue, 24 Jul 2018 19:51:34 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/10913 )
Change subject: IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/10913/2/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
File fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java:
http://gerrit.cloudera.org:8080/#/c/10913/2/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java@57
PS2, Line 57: import org.apache.sentry.hdfs.service.thrift.SentryHDFSService.get_all_related_paths_result;
> Where do we use this? I did not find it in the patch.
sorry, Eclipse went nuts on autocomplete here apparently. Will remove in next rev.
--
To view, visit http://gerrit.cloudera.org:8080/10913
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I93937aed9b76ef6a62b1c588c59c34d3d6831a46
Gerrit-Change-Number: 10913
Gerrit-PatchSet: 2
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Tianyi Wang <tw...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Fri, 13 Jul 2018 00:25:25 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10913 )
Change subject: IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
......................................................................
Patch Set 6:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2860/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/10913
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I93937aed9b76ef6a62b1c588c59c34d3d6831a46
Gerrit-Change-Number: 10913
Gerrit-PatchSet: 6
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tianyi Wang <tw...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Wed, 25 Jul 2018 02:07:16 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/10913 )
Change subject: IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
......................................................................
Patch Set 3:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/10913/3/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
File fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java:
http://gerrit.cloudera.org:8080/#/c/10913/3/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java@114
PS3, Line 114: load them?
> what would be loaded... this is a new table if I've understood it correctly
yea, the target table is sort of an odd temporary catalog entry which doesn't actually have an entry in the HMS. I was afraid that, if someone someone uses this method to create an instance, and then ends up calling getPartitions(), we'll end up making an HMS API call to get partitions for a table and throw an error like TableDoesNotExistException. Was thinking of either a new subclass or adding some kind of boolean isCtasTarget_ to assert against to make sure we don't do something like that. I'll elaborate.
--
To view, visit http://gerrit.cloudera.org:8080/10913
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I93937aed9b76ef6a62b1c588c59c34d3d6831a46
Gerrit-Change-Number: 10913
Gerrit-PatchSet: 3
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Tianyi Wang <tw...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Thu, 19 Jul 2018 00:11:31 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/10913 )
Change subject: IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
......................................................................
Patch Set 2:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/10913/2/fe/src/main/java/org/apache/impala/catalog/Db.java
File fe/src/main/java/org/apache/impala/catalog/Db.java:
http://gerrit.cloudera.org:8080/#/c/10913/2/fe/src/main/java/org/apache/impala/catalog/Db.java@176
PS2, Line 176: public FeKuduTable createKuduCtasTarget(org.apache.hadoop.hive.metastore.api.Table msTbl,
> nit: long line
Done
http://gerrit.cloudera.org:8080/#/c/10913/2/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
File fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java:
http://gerrit.cloudera.org:8080/#/c/10913/2/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java@57
PS2, Line 57: import org.apache.sentry.hdfs.service.thrift.SentryHDFSService.get_all_related_paths_result;
> sorry, Eclipse went nuts on autocomplete here apparently. Will remove in ne
Done
http://gerrit.cloudera.org:8080/#/c/10913/2/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java@211
PS2, Line 211: public TTableDescriptor toThriftDescriptor(int tableId, Set<Long> referencedPartitions) {
> nit: long line
Done
http://gerrit.cloudera.org:8080/#/c/10913/2/fe/src/main/java/org/apache/impala/catalog/local/LocalKuduTable.java
File fe/src/main/java/org/apache/impala/catalog/local/LocalKuduTable.java:
http://gerrit.cloudera.org:8080/#/c/10913/2/fe/src/main/java/org/apache/impala/catalog/local/LocalKuduTable.java@83
PS2, Line 83: List<KuduPartitionParam> partitionBy = KuduTable.loadPartitionByParams(
> nit: this could be one line
Done
--
To view, visit http://gerrit.cloudera.org:8080/10913
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I93937aed9b76ef6a62b1c588c59c34d3d6831a46
Gerrit-Change-Number: 10913
Gerrit-PatchSet: 2
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Tianyi Wang <tw...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Wed, 18 Jul 2018 00:59:42 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/10913 )
Change subject: IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
......................................................................
Patch Set 2:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/10913/2/fe/src/main/java/org/apache/impala/catalog/Db.java
File fe/src/main/java/org/apache/impala/catalog/Db.java:
http://gerrit.cloudera.org:8080/#/c/10913/2/fe/src/main/java/org/apache/impala/catalog/Db.java@176
PS2, Line 176: public FeKuduTable createKuduCtasTarget(org.apache.hadoop.hive.metastore.api.Table msTbl,
nit: long line
http://gerrit.cloudera.org:8080/#/c/10913/2/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
File fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java:
http://gerrit.cloudera.org:8080/#/c/10913/2/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java@57
PS2, Line 57: import org.apache.sentry.hdfs.service.thrift.SentryHDFSService.get_all_related_paths_result;
Where do we use this? I did not find it in the patch.
http://gerrit.cloudera.org:8080/#/c/10913/2/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java@211
PS2, Line 211: public TTableDescriptor toThriftDescriptor(int tableId, Set<Long> referencedPartitions) {
nit: long line
http://gerrit.cloudera.org:8080/#/c/10913/2/fe/src/main/java/org/apache/impala/catalog/local/LocalKuduTable.java
File fe/src/main/java/org/apache/impala/catalog/local/LocalKuduTable.java:
http://gerrit.cloudera.org:8080/#/c/10913/2/fe/src/main/java/org/apache/impala/catalog/local/LocalKuduTable.java@83
PS2, Line 83: List<KuduPartitionParam> partitionBy = KuduTable.loadPartitionByParams(
nit: this could be one line
--
To view, visit http://gerrit.cloudera.org:8080/10913
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I93937aed9b76ef6a62b1c588c59c34d3d6831a46
Gerrit-Change-Number: 10913
Gerrit-PatchSet: 2
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Tianyi Wang <tw...@cloudera.com>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Thu, 12 Jul 2018 16:28:52 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/10913 )
Change subject: IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
......................................................................
Patch Set 6: Code-Review+2
Adding +2 since Vuk said LGTM pending Csaba's +1
--
To view, visit http://gerrit.cloudera.org:8080/10913
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I93937aed9b76ef6a62b1c588c59c34d3d6831a46
Gerrit-Change-Number: 10913
Gerrit-PatchSet: 6
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tianyi Wang <tw...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>
Gerrit-Comment-Date: Wed, 25 Jul 2018 19:27:03 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Hello Tianyi Wang, Csaba Ringhofer, Vuk Ercegovac,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/10913
to look at the new patch set (#3).
Change subject: IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
......................................................................
IMPALA-7276. Support CREATE TABLE AS SELECT with LocalCatalog
This fixed most of the remaining Kudu tests which relied on CTAS. Now only a
few Kudu tests fail:
FAIL query_test/test_kudu.py::TestKuduOperations::()::test_kudu_col_changed
FAIL query_test/test_kudu.py::TestKuduOperations::()::test_kudu_col_null_changed
FAIL query_test/test_kudu.py::TestKuduOperations::()::test_kudu_col_not_null_changed
FAIL query_test/test_kudu.py::TestKuduOperations::()::test_kudu_col_added
The above 4 fail because they are asserting something about the caching
behavior of the old catalog implementation.
FAIL query_test/test_kudu.py::TestImpalaKuduIntegration::()::test_delete_external_kudu_table
FAIL query_test/test_kudu.py::TestImpalaKuduIntegration::()::test_delete_managed_kudu_table
These fail due to attempting to load non-existent tables referred to by a
DELETE statement. Need to investigate these further, but not related to CTAS.
Change-Id: I93937aed9b76ef6a62b1c588c59c34d3d6831a46
---
M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java
M fe/src/main/java/org/apache/impala/catalog/Db.java
M fe/src/main/java/org/apache/impala/catalog/FeDb.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalKuduTable.java
M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java
M fe/src/main/java/org/apache/impala/planner/TableSink.java
M fe/src/main/java/org/apache/impala/util/KuduUtil.java
10 files changed, 134 insertions(+), 29 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/13/10913/3
--
To view, visit http://gerrit.cloudera.org:8080/10913
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I93937aed9b76ef6a62b1c588c59c34d3d6831a46
Gerrit-Change-Number: 10913
Gerrit-PatchSet: 3
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Tianyi Wang <tw...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Vuk Ercegovac <ve...@cloudera.com>