You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Fredy Wijaya (Code Review)" <ge...@cloudera.org> on 2018/02/20 15:51:56 UTC

[Impala-ASF-CR] IMPALA-6275: Fix warn stacktrace in successful CTAS 

Fredy Wijaya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/9364


Change subject: IMPALA-6275: Fix warn stacktrace in successful CTAS
......................................................................

IMPALA-6275: Fix warn stacktrace in successful CTAS

For CTAS in HDFS tables, do not load the column statistics from HMS
since since that information will not be available in HMS.

Change-Id: I6f07a188458954802fda20e3b3b56280d96e788e
---
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
1 file changed, 11 insertions(+), 4 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/64/9364/1
-- 
To view, visit http://gerrit.cloudera.org:8080/9364
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I6f07a188458954802fda20e3b3b56280d96e788e
Gerrit-Change-Number: 9364
Gerrit-PatchSet: 1
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>

[Impala-ASF-CR] IMPALA-6275: Fix warn stacktrace in successful CTAS

Posted by "Alex Behm (Code Review)" <ge...@cloudera.org>.
Alex Behm has posted comments on this change. ( http://gerrit.cloudera.org:8080/9364 )

Change subject: IMPALA-6275: Fix warn stacktrace in successful CTAS
......................................................................


Patch Set 6: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/9364
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6f07a188458954802fda20e3b3b56280d96e788e
Gerrit-Change-Number: 9364
Gerrit-PatchSet: 6
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Comment-Date: Fri, 23 Feb 2018 05:48:09 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6275: Fix warn stacktrace in successful CTAS

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9364 )

Change subject: IMPALA-6275: Fix warn stacktrace in successful CTAS
......................................................................


Patch Set 5:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1988/


-- 
To view, visit http://gerrit.cloudera.org:8080/9364
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6f07a188458954802fda20e3b3b56280d96e788e
Gerrit-Change-Number: 9364
Gerrit-PatchSet: 5
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Comment-Date: Thu, 22 Feb 2018 21:16:36 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6275: Fix warn stacktrace in successful CTAS

Posted by "Alex Behm (Code Review)" <ge...@cloudera.org>.
Alex Behm has posted comments on this change. ( http://gerrit.cloudera.org:8080/9364 )

Change subject: IMPALA-6275: Fix warn stacktrace in successful CTAS
......................................................................


Patch Set 4:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/9364/4/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:

http://gerrit.cloudera.org:8080/#/c/9364/4/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@1254
PS4, Line 1254:         loadAllColumnStats(client);
to preserve the existing behavior exactly:
if (loadTableSchema) {
  loadSchema(msTbl);
  loadAllColumnStats(client);
}


http://gerrit.cloudera.org:8080/#/c/9364/4/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@1569
PS4, Line 1569:    * Loads table schema and column stats from Hive Metastore.
update comment


http://gerrit.cloudera.org:8080/#/c/9364/4/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@2280
PS4, Line 2280:       org.apache.hadoop.hive.metastore.api.Table msTbl) throws Exception {
What throws here? If nothing, remove throws.



-- 
To view, visit http://gerrit.cloudera.org:8080/9364
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6f07a188458954802fda20e3b3b56280d96e788e
Gerrit-Change-Number: 9364
Gerrit-PatchSet: 4
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Comment-Date: Thu, 22 Feb 2018 20:44:56 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-6275: Fix warn stacktrace in successful CTAS

Posted by "Alex Behm (Code Review)" <ge...@cloudera.org>.
Alex Behm has posted comments on this change. ( http://gerrit.cloudera.org:8080/9364 )

Change subject: IMPALA-6275: Fix warn stacktrace in successful CTAS
......................................................................


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/9364/1/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:

http://gerrit.cloudera.org:8080/#/c/9364/1/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@1577
PS1, Line 1577:   private void loadSchema(IMetaStoreClient client,
Even better, let's separate the loadSchema() part from the loadAllColumnStats() part.

* the loadSchema() part really converts an existing msTbl to our internal schema; we only need client to get the nullPartitionKeyValue_, so probably we can change loadSchema() to only take the msTbl and nothing else as param
* loadAllColumnStats() really needs a client because it tries to RPC to the HMS; this is a good indication that loadSchema() and loadAlColumnStats() should really be separate



-- 
To view, visit http://gerrit.cloudera.org:8080/9364
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6f07a188458954802fda20e3b3b56280d96e788e
Gerrit-Change-Number: 9364
Gerrit-PatchSet: 1
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Comment-Date: Wed, 21 Feb 2018 06:08:10 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-6275: Fix warn stacktrace in successful CTAS

Posted by "Fredy Wijaya (Code Review)" <ge...@cloudera.org>.
Fredy Wijaya has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/9364 )

Change subject: IMPALA-6275: Fix warn stacktrace in successful CTAS
......................................................................

IMPALA-6275: Fix warn stacktrace in successful CTAS

For CTAS in HDFS tables, do not load the column statistics from HMS
since since that information will not be available in HMS.

Change-Id: I6f07a188458954802fda20e3b3b56280d96e788e
---
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
1 file changed, 21 insertions(+), 14 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/64/9364/2
-- 
To view, visit http://gerrit.cloudera.org:8080/9364
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6f07a188458954802fda20e3b3b56280d96e788e
Gerrit-Change-Number: 9364
Gerrit-PatchSet: 2
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>

[Impala-ASF-CR] IMPALA-6275: Fix warn stacktrace in successful CTAS

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9364 )

Change subject: IMPALA-6275: Fix warn stacktrace in successful CTAS
......................................................................


Patch Set 6:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1997/


-- 
To view, visit http://gerrit.cloudera.org:8080/9364
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6f07a188458954802fda20e3b3b56280d96e788e
Gerrit-Change-Number: 9364
Gerrit-PatchSet: 6
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Comment-Date: Fri, 23 Feb 2018 05:48:46 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6275: Fix warn stacktrace in successful CTAS

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9364 )

Change subject: IMPALA-6275: Fix warn stacktrace in successful CTAS
......................................................................


Patch Set 6: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/9364
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6f07a188458954802fda20e3b3b56280d96e788e
Gerrit-Change-Number: 9364
Gerrit-PatchSet: 6
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Comment-Date: Fri, 23 Feb 2018 10:49:27 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6275: Fix warn stacktrace in successful CTAS

Posted by "Alex Behm (Code Review)" <ge...@cloudera.org>.
Alex Behm has posted comments on this change. ( http://gerrit.cloudera.org:8080/9364 )

Change subject: IMPALA-6275: Fix warn stacktrace in successful CTAS
......................................................................


Patch Set 5: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/9364
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6f07a188458954802fda20e3b3b56280d96e788e
Gerrit-Change-Number: 9364
Gerrit-PatchSet: 5
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Comment-Date: Thu, 22 Feb 2018 21:10:27 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6275: Fix warn stacktrace in successful CTAS

Posted by "Fredy Wijaya (Code Review)" <ge...@cloudera.org>.
Fredy Wijaya has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/9364 )

Change subject: IMPALA-6275: Fix warn stacktrace in successful CTAS
......................................................................

IMPALA-6275: Fix warn stacktrace in successful CTAS

For CTAS in HDFS tables, do not load the column statistics from HMS
since since that information will not be available in HMS.

Refactored CTAS for HDFS tables to use HdfsTable.createCtasTarget()
instead of HdfsTable.load().

Testing:
- Ran end-to-end query tests

Change-Id: I6f07a188458954802fda20e3b3b56280d96e788e
---
M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
2 files changed, 28 insertions(+), 20 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/64/9364/4
-- 
To view, visit http://gerrit.cloudera.org:8080/9364
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6f07a188458954802fda20e3b3b56280d96e788e
Gerrit-Change-Number: 9364
Gerrit-PatchSet: 4
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>

Re: [Impala-ASF-CR] IMPALA-6275: Fix warn stacktrace in successful CTAS

Posted by Alexander Behm <al...@cloudera.com>.
GVO hit IMPALA-6532, going to retry

On Thu, Feb 22, 2018 at 6:31 PM, Impala Public Jenkins (Code Review) <
gerrit@cloudera.org> wrote:

> Impala Public Jenkins *posted comments* on this change.
>
> View Change <http://gerrit.cloudera.org:8080/9364>
>
> Patch set 5:Verified -1
>
> Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/1988/
>
>
> To view, visit change 9364 <http://gerrit.cloudera.org:8080/9364>. To
> unsubscribe, visit settings <http://gerrit.cloudera.org:8080/settings>.
> Gerrit-Project: Impala-ASF
> Gerrit-Branch: master
> Gerrit-MessageType: comment
> Gerrit-Change-Id: I6f07a188458954802fda20e3b3b56280d96e788e
> Gerrit-Change-Number: 9364
> Gerrit-PatchSet: 5
> Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
> Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
> Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
> Gerrit-Reviewer: Impala Public Jenkins
> Gerrit-Comment-Date: Fri, 23 Feb 2018 02:31:09 +0000
> Gerrit-HasComments: No
>
> --
> You received this message because you are subscribed to the Google Groups
> "impala-cr" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to impala-cr+unsubscribe@cloudera.com.
> For more options, visit https://groups.google.com/a/cloudera.com/d/optout.
>

[Impala-ASF-CR] IMPALA-6275: Fix warn stacktrace in successful CTAS

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9364 )

Change subject: IMPALA-6275: Fix warn stacktrace in successful CTAS
......................................................................


Patch Set 5: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/1988/


-- 
To view, visit http://gerrit.cloudera.org:8080/9364
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6f07a188458954802fda20e3b3b56280d96e788e
Gerrit-Change-Number: 9364
Gerrit-PatchSet: 5
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Comment-Date: Fri, 23 Feb 2018 02:31:09 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6275: Fix warn stacktrace in successful CTAS

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/9364 )

Change subject: IMPALA-6275: Fix warn stacktrace in successful CTAS
......................................................................

IMPALA-6275: Fix warn stacktrace in successful CTAS

For CTAS in HDFS tables, do not load the column statistics from HMS
since since that information will not be available in HMS.

Refactored CTAS for HDFS tables to use HdfsTable.createCtasTarget()
instead of HdfsTable.load().

Testing:
- Ran end-to-end query tests

Change-Id: I6f07a188458954802fda20e3b3b56280d96e788e
Reviewed-on: http://gerrit.cloudera.org:8080/9364
Reviewed-by: Alex Behm <al...@cloudera.com>
Tested-by: Impala Public Jenkins
---
M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
2 files changed, 30 insertions(+), 20 deletions(-)

Approvals:
  Alex Behm: Looks good to me, approved
  Impala Public Jenkins: Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/9364
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I6f07a188458954802fda20e3b3b56280d96e788e
Gerrit-Change-Number: 9364
Gerrit-PatchSet: 7
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins

[Impala-ASF-CR] IMPALA-6275: Fix warn stacktrace in successful CTAS

Posted by "Fredy Wijaya (Code Review)" <ge...@cloudera.org>.
Fredy Wijaya has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/9364 )

Change subject: IMPALA-6275: Fix warn stacktrace in successful CTAS
......................................................................

IMPALA-6275: Fix warn stacktrace in successful CTAS

For CTAS in HDFS tables, do not load the column statistics from HMS
since since that information will not be available in HMS.

Refactored CTAS for HDFS tables to use HdfsTable.createCtasTarget()
instead of HdfsTable.load().

Testing:
- Ran end-to-end query tests

Change-Id: I6f07a188458954802fda20e3b3b56280d96e788e
---
M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
2 files changed, 30 insertions(+), 20 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/64/9364/5
-- 
To view, visit http://gerrit.cloudera.org:8080/9364
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6f07a188458954802fda20e3b3b56280d96e788e
Gerrit-Change-Number: 9364
Gerrit-PatchSet: 5
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>

[Impala-ASF-CR] IMPALA-6275: Fix warn stacktrace in successful CTAS

Posted by "Alex Behm (Code Review)" <ge...@cloudera.org>.
Alex Behm has posted comments on this change. ( http://gerrit.cloudera.org:8080/9364 )

Change subject: IMPALA-6275: Fix warn stacktrace in successful CTAS
......................................................................


Patch Set 2:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/9364/2/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java:

http://gerrit.cloudera.org:8080/#/c/9364/2/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@1252
PS2, Line 1252:         boolean isInitialCatalog =
Looking at how the CreateTableAsSelectStmt calls this load() function, I think it makes even more sense to address the TODO in CreateTableAsSelectStmt#analyze.

The TODO proposes to add a special createCtasTarget() to make the desired behavior more explicit. Relying on the catalog version to detect CTAS is wonky at best.

Sorry for the back and forth. I think it's worth spending the time to fix it cleanly.

Can you give the createCtasTarget() solution a try?



-- 
To view, visit http://gerrit.cloudera.org:8080/9364
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6f07a188458954802fda20e3b3b56280d96e788e
Gerrit-Change-Number: 9364
Gerrit-PatchSet: 2
Gerrit-Owner: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Comment-Date: Wed, 21 Feb 2018 19:43:00 +0000
Gerrit-HasComments: Yes