You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Alex Behm (Code Review)" <ge...@cloudera.org> on 2017/02/16 07:24:31 UTC
[Impala-ASF-CR] IMPALA-4282: Remove max length check for type strings.
Alex Behm has uploaded a new change for review.
http://gerrit.cloudera.org:8080/6034
Change subject: IMPALA-4282: Remove max length check for type strings.
......................................................................
IMPALA-4282: Remove max length check for type strings.
During analysis, Impala used to enforce the default HMS
limit on the number of characters of a type string.
That enforcement has the benefit of a clear error
message, but has the severe drawback of not having
any workaround because the limit is baked into the code.
Going above the 4000 limit is pretty easy with complex
types, and several users have already run into this
issue.
This patch removes the Impala-side enforcement such
that the 4000 limit can be increased by altering
the corresponding column in the Metastore's backend
database.
Change-Id: I01f5e503e14feee857bbdf80b3ef4f5a8b57fb2d
---
M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
2 files changed, 0 insertions(+), 42 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/34/6034/1
--
To view, visit http://gerrit.cloudera.org:8080/6034
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I01f5e503e14feee857bbdf80b3ef4f5a8b57fb2d
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Alex Behm <al...@cloudera.com>
[Impala-ASF-CR] IMPALA-4282: Remove max length check for type strings.
Posted by "Alex Behm (Code Review)" <ge...@cloudera.org>.
Alex Behm has posted comments on this change.
Change subject: IMPALA-4282: Remove max length check for type strings.
......................................................................
Patch Set 2:
Ahh right. Thanks. Done.
--
To view, visit http://gerrit.cloudera.org:8080/6034
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I01f5e503e14feee857bbdf80b3ef4f5a8b57fb2d
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4282: Remove max length check for type strings.
Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.
Bharath Vissapragada has posted comments on this change.
Change subject: IMPALA-4282: Remove max length check for type strings.
......................................................................
Patch Set 2: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/6034
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I01f5e503e14feee857bbdf80b3ef4f5a8b57fb2d
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4282: Remove max length check for type strings.
Posted by "Alex Behm (Code Review)" <ge...@cloudera.org>.
Alex Behm has posted comments on this change.
Change subject: IMPALA-4282: Remove max length check for type strings.
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/6034/2/fe/src/main/java/org/apache/impala/analysis/ColumnDef.java
File fe/src/main/java/org/apache/impala/analysis/ColumnDef.java:
Line 185: comment_.length() > MetaStoreUtil.CREATE_MAX_COMMENT_LENGTH) {
> why still do this, rather than rely on the lower layer to error like the ty
We enforce a variety of other HMS constraints on our side, see MetaStoreUtil. I generally think we should not enforce the constraints ourselves and bite the bullet of bad error messages in favor of having an HMS-side workaround. We should really fix the bad error messages in Hive.
However, the type-string length constraint is particularly painful, and I don't want to gate its progress on removing all of our enforcements.
I'm happy to fix the other ones later.
--
To view, visit http://gerrit.cloudera.org:8080/6034
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I01f5e503e14feee857bbdf80b3ef4f5a8b57fb2d
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4282: Remove max length check for type strings.
Posted by "Alex Behm (Code Review)" <ge...@cloudera.org>.
Alex Behm has uploaded a new patch set (#2).
Change subject: IMPALA-4282: Remove max length check for type strings.
......................................................................
IMPALA-4282: Remove max length check for type strings.
During analysis, Impala used to enforce the default HMS
limit on the number of characters of a type string.
That enforcement has the benefit of a clear error
message, but has the severe drawback of not having
any workaround because the limit is baked into the code.
Going above the 4000 limit is pretty easy with complex
types, and several users have already run into this
issue.
This patch removes the Impala-side enforcement such
that the 4000 limit can be increased by altering
the corresponding columns in the Metastore's backend
database.
Change-Id: I01f5e503e14feee857bbdf80b3ef4f5a8b57fb2d
---
M fe/src/main/java/org/apache/impala/analysis/ColumnDef.java
M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
3 files changed, 1 insertion(+), 50 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/34/6034/2
--
To view, visit http://gerrit.cloudera.org:8080/6034
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I01f5e503e14feee857bbdf80b3ef4f5a8b57fb2d
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
[Impala-ASF-CR] IMPALA-4282: Remove max length check for type strings.
Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.
Bharath Vissapragada has posted comments on this change.
Change subject: IMPALA-4282: Remove max length check for type strings.
......................................................................
Patch Set 1:
Looks like you forgot to git-add changes from ColumnDef.java.
--
To view, visit http://gerrit.cloudera.org:8080/6034
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I01f5e503e14feee857bbdf80b3ef4f5a8b57fb2d
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4282: Remove max length check for type strings.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged.
Change subject: IMPALA-4282: Remove max length check for type strings.
......................................................................
IMPALA-4282: Remove max length check for type strings.
During analysis, Impala used to enforce the default HMS
limit on the number of characters of a type string.
That enforcement has the benefit of a clear error
message, but has the severe drawback of not having
any workaround because the limit is baked into the code.
Going above the 4000 limit is pretty easy with complex
types, and several users have already run into this
issue.
This patch removes the Impala-side enforcement such
that the 4000 limit can be increased by altering
the corresponding columns in the Metastore's backend
database.
Change-Id: I01f5e503e14feee857bbdf80b3ef4f5a8b57fb2d
Reviewed-on: http://gerrit.cloudera.org:8080/6034
Reviewed-by: Alex Behm <al...@cloudera.com>
Tested-by: Impala Public Jenkins
---
M fe/src/main/java/org/apache/impala/analysis/ColumnDef.java
M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
3 files changed, 1 insertion(+), 50 deletions(-)
Approvals:
Impala Public Jenkins: Verified
Alex Behm: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/6034
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I01f5e503e14feee857bbdf80b3ef4f5a8b57fb2d
Gerrit-PatchSet: 4
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-4282: Remove max length check for type strings.
Posted by "Dan Hecht (Code Review)" <ge...@cloudera.org>.
Dan Hecht has posted comments on this change.
Change subject: IMPALA-4282: Remove max length check for type strings.
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/6034/2//COMMIT_MSG
Commit Message:
Line 12: message, but has the severe drawback of not having
what happens now when the type string is too long? do we still get an error, but more cryptic?
--
To view, visit http://gerrit.cloudera.org:8080/6034
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I01f5e503e14feee857bbdf80b3ef4f5a8b57fb2d
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4282: Remove max length check for type strings.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change.
Change subject: IMPALA-4282: Remove max length check for type strings.
......................................................................
Patch Set 3: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/6034
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I01f5e503e14feee857bbdf80b3ef4f5a8b57fb2d
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4282: Remove max length check for type strings.
Posted by "Alex Behm (Code Review)" <ge...@cloudera.org>.
Alex Behm has posted comments on this change.
Change subject: IMPALA-4282: Remove max length check for type strings.
......................................................................
Patch Set 3: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/6034
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I01f5e503e14feee857bbdf80b3ef4f5a8b57fb2d
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4282: Remove max length check for type strings.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change.
Change subject: IMPALA-4282: Remove max length check for type strings.
......................................................................
Patch Set 3:
Build started: http://jenkins.impala.io:8080/job/gerrit-verify-dryrun/281/
--
To view, visit http://gerrit.cloudera.org:8080/6034
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I01f5e503e14feee857bbdf80b3ef4f5a8b57fb2d
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4282: Remove max length check for type strings.
Posted by "Alex Behm (Code Review)" <ge...@cloudera.org>.
Alex Behm has posted comments on this change.
Change subject: IMPALA-4282: Remove max length check for type strings.
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/6034/2//COMMIT_MSG
Commit Message:
Line 12: message, but has the severe drawback of not having
> what happens now when the type string is too long? do we still get an erro
Yes. The Metastore backend DB will reject the INSERT due to a constraint violation, and that gets bubbled up through the ORM layer and finally makes it to Impala in a very cryptic form. You can see a flavor of the error in the JIRA.
--
To view, visit http://gerrit.cloudera.org:8080/6034
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I01f5e503e14feee857bbdf80b3ef4f5a8b57fb2d
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4282: Remove max length check for type strings.
Posted by "Dan Hecht (Code Review)" <ge...@cloudera.org>.
Dan Hecht has posted comments on this change.
Change subject: IMPALA-4282: Remove max length check for type strings.
......................................................................
Patch Set 2: Code-Review+1
(1 comment)
http://gerrit.cloudera.org:8080/#/c/6034/2/fe/src/main/java/org/apache/impala/analysis/ColumnDef.java
File fe/src/main/java/org/apache/impala/analysis/ColumnDef.java:
Line 185: comment_.length() > MetaStoreUtil.CREATE_MAX_COMMENT_LENGTH) {
why still do this, rather than rely on the lower layer to error like the type field? especially given the comment above CREATE_MAX_COMMENT_LENGTH that this is conservative.
--
To view, visit http://gerrit.cloudera.org:8080/6034
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I01f5e503e14feee857bbdf80b3ef4f5a8b57fb2d
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-HasComments: Yes