You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@impala.apache.org by "Huaisi Xu (Code Review)" <ge...@cloudera.org> on 2016/06/07 22:52:49 UTC
[Impala-CR](cdh5-trunk) IMPALA-3687: Use type from avro schema during schema reconcilation.
Huaisi Xu has uploaded a new change for review.
http://gerrit.cloudera.org:8080/3331
Change subject: IMPALA-3687: Use type from avro schema during schema reconcilation.
......................................................................
IMPALA-3687: Use type from avro schema during schema reconcilation.
There is no effect that we change the name of a string/char/varchar
column in schema definition because impala uses column defnition
for these types. This patch changes the behavior to that impala
uses type from schema definition. This is to preserve support for
CHAR/VARCHAR column type.
Change-Id: Ia3e43b2885853c2b4f207a45a873c9d7f31379cd
---
M fe/src/main/java/com/cloudera/impala/util/AvroSchemaUtils.java
M testdata/workloads/functional-query/queries/QueryTest/avro-schema-changes.test
2 files changed, 58 insertions(+), 8 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/31/3331/1
--
To view, visit http://gerrit.cloudera.org:8080/3331
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ia3e43b2885853c2b4f207a45a873c9d7f31379cd
Gerrit-PatchSet: 1
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Huaisi Xu <hx...@cloudera.com>
[Impala-CR](cdh5-trunk) IMPALA-3687: Use type from avro schema during schema reconcilation.
Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.
Bharath Vissapragada has posted comments on this change.
Change subject: IMPALA-3687: Use type from avro schema during schema reconcilation.
......................................................................
Patch Set 2:
(8 comments)
http://gerrit.cloudera.org:8080/#/c/3331/2//COMMIT_MSG
Commit Message:
PS2, Line 7: avro
nit: Avro (capitalization)
PS2, Line 7: reconcilation
nit:typo
PS2, Line 9: string/char/varchar
nit:capitalize for consitency.
PS2, Line 9: here is no effect that we change the
I think the commit message can be reworded to indicate the issue we are trying to fix, i.e, having different column name for string types in Avro schema and hive col defs can raise an error ....So we just pick the name from the Avro schema and retain the type from hive coldef...
http://gerrit.cloudera.org:8080/#/c/3331/2/fe/src/main/java/com/cloudera/impala/util/AvroSchemaUtils.java
File fe/src/main/java/com/cloudera/impala/util/AvroSchemaUtils.java:
PS2, Line 116:
:
:
:
Why did you remove this?
http://gerrit.cloudera.org:8080/#/c/3331/1/testdata/workloads/functional-query/queries/QueryTest/avro-schema-changes.test
File testdata/workloads/functional-query/queries/QueryTest/avro-schema-changes.test:
Line 72: CREATE EXTERNAL TABLE alltypesagg_changed_column_name_comment (
> I would rather prefer to do this in Hive. you know how?
Why in hive?
http://gerrit.cloudera.org:8080/#/c/3331/2/testdata/workloads/functional-query/queries/QueryTest/avro-schema-changes.test
File testdata/workloads/functional-query/queries/QueryTest/avro-schema-changes.test:
PS2, Line 71: # IMPALA-3687
I think avro-schema-resolution.test is a better place for this.
PS2, Line 71: avro
nit: Avro
--
To view, visit http://gerrit.cloudera.org:8080/3331
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia3e43b2885853c2b4f207a45a873c9d7f31379cd
Gerrit-PatchSet: 2
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Huaisi Xu <hx...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-CR](cdh5-trunk) IMPALA-3687: Prefer Avro field name during schema reconciliation
Posted by "Huaisi Xu (Code Review)" <ge...@cloudera.org>.
Hello Bharath Vissapragada,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/3331
to look at the new patch set (#9).
Change subject: IMPALA-3687: Prefer Avro field name during schema reconciliation
......................................................................
IMPALA-3687: Prefer Avro field name during schema reconciliation
Since it is possible to create an Avro table with both column
definitions and an Avro schema, Impala attempts to reconcile
inconsistencies in the two schema definitions, generally preferring the
Avro schema. The only exception to this rule was with
CHAR/VARCHAR/STRING columns, where the column definition was preferred
in order to support tables with CHAR/VARCHAR columns although Avro only
supports STRING. This exception is confusing because the name for such a
column will be taken from the column definition (and not from the Avro
schema).
This patch prefers name, comment from Avro schema definition and
uses column type from column definition for CHAR/VARCHAR/STRING
columns.
Change-Id: Ia3e43b2885853c2b4f207a45a873c9d7f31379cd
---
M fe/src/main/java/com/cloudera/impala/util/AvroSchemaConverter.java
M fe/src/main/java/com/cloudera/impala/util/AvroSchemaUtils.java
M testdata/workloads/functional-query/queries/QueryTest/avro-schema-changes.test
M testdata/workloads/functional-query/queries/QueryTest/avro-schema-resolution.test
M tests/query_test/test_avro_schema_resolution.py
5 files changed, 130 insertions(+), 17 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/31/3331/9
--
To view, visit http://gerrit.cloudera.org:8080/3331
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia3e43b2885853c2b4f207a45a873c9d7f31379cd
Gerrit-PatchSet: 9
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Dimitris Tsirogiannis <dt...@cloudera.com>
Gerrit-Reviewer: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Juan Yu <jy...@cloudera.com>
[Impala-CR](cdh5-trunk) IMPALA-3687: Prefer Avro field name during schema reconciliation
Posted by "Internal Jenkins (Code Review)" <ge...@cloudera.org>.
Internal Jenkins has posted comments on this change.
Change subject: IMPALA-3687: Prefer Avro field name during schema reconciliation
......................................................................
Patch Set 13: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/3331
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia3e43b2885853c2b4f207a45a873c9d7f31379cd
Gerrit-PatchSet: 13
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Dimitris Tsirogiannis <dt...@cloudera.com>
Gerrit-Reviewer: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Internal Jenkins
Gerrit-Reviewer: Juan Yu <jy...@cloudera.com>
Gerrit-HasComments: No
[Impala-CR](cdh5-trunk) IMPALA-3687: Use type from Avro schema during schema reconciliation.
Posted by "Alex Behm (Code Review)" <ge...@cloudera.org>.
Alex Behm has posted comments on this change.
Change subject: IMPALA-3687: Use type from Avro schema during schema reconciliation.
......................................................................
Patch Set 2:
(8 comments)
http://gerrit.cloudera.org:8080/#/c/3331/4//COMMIT_MSG
Commit Message:
Line 7: IMPALA-3687: Use type from avro schema during schema reconcilation.
Always prefer the Avro field name during schema reconciliation.
Line 9: There is no effect that we change the name of a string/char/varchar
This blurb is very difficult to follow. It would be good to summarize the new behavior, i.e., in case of inconsistencies we generally prefer the name, type, and comment from the Avro schema, except for CHAR/VARCHAR where we retain the CHAR/VARCHAR type but keep the name and comment from the Avro column.
http://gerrit.cloudera.org:8080/#/c/3331/2/fe/src/main/java/com/cloudera/impala/util/AvroSchemaUtils.java
File fe/src/main/java/com/cloudera/impala/util/AvroSchemaUtils.java:
PS2, Line 116:
:
:
:
> I know but the code does not "reconcile" that at all right?
Bharath is correct that the comment had an original intention. The comment was put here to describe what will happen in cases like "create table t (ts timestamp) stored as avro". The resulting table will have a STRING type. But the 'root cause' of this behavior is actually in how we synthesize an Avro schema from the column definitions, so the comment is somewhat out of place here. Let's move this comment into AvroSchemaConverter.java and rephrase as appropriate.
http://gerrit.cloudera.org:8080/#/c/3331/4/fe/src/main/java/com/cloudera/impala/util/AvroSchemaUtils.java
File fe/src/main/java/com/cloudera/impala/util/AvroSchemaUtils.java:
PS4, Line 113: A
grammar; also could be rephrased for clarity, e.g.:
A CHAR/VARCHAR column definition maps to an Avro STRING. The reconciled column will preserve the CHAR/VARCHAR type but prefer the column name and comment from the Avro column.
also move this blurb to L134
http://gerrit.cloudera.org:8080/#/c/3331/2/testdata/workloads/functional-query/queries/QueryTest/avro-schema-changes.test
File testdata/workloads/functional-query/queries/QueryTest/avro-schema-changes.test:
PS2, Line 71: # IMPALA-3687
> I am fine with anything.. what do you think @Alex?
Agree that avro-schema-resolution.test is more appropriate.
http://gerrit.cloudera.org:8080/#/c/3331/4/testdata/workloads/functional-query/queries/QueryTest/avro-schema-changes.test
File testdata/workloads/functional-query/queries/QueryTest/avro-schema-changes.test:
Line 71: # IMPALA-3687. Create a table that has different avro schema from its column definition
Create a table with mismatched column definitions and Avro schema.
Line 86: LOCATION '$FILESYSTEM_PREFIX/test-warehouse/alltypesaggmultifilesnopart_avro_snap'
remove this clause - it only adds confusion
Line 95: {"type": ["float", "null"], "name": "float_col_changed", "doc": "float comment changed"},
typo: changed
--
To view, visit http://gerrit.cloudera.org:8080/3331
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia3e43b2885853c2b4f207a45a873c9d7f31379cd
Gerrit-PatchSet: 2
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Huaisi Xu <hx...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-CR](cdh5-trunk) IMPALA-3687: Prefer Avro field name during schema reconciliation
Posted by "Internal Jenkins (Code Review)" <ge...@cloudera.org>.
Internal Jenkins has submitted this change and it was merged.
Change subject: IMPALA-3687: Prefer Avro field name during schema reconciliation
......................................................................
IMPALA-3687: Prefer Avro field name during schema reconciliation
Since it is possible to create an Avro table with both column
definitions and an Avro schema, Impala attempts to reconcile
inconsistencies in the two schema definitions, generally preferring the
Avro schema. The only exception to this rule was with
CHAR/VARCHAR/STRING columns, where the column definition was preferred
in order to support tables with CHAR/VARCHAR columns although Avro only
supports STRING. This exception is confusing because the name for such a
column will be taken from the column definition (and not from the Avro
schema).
This patch prefers name, comment from Avro schema definition and
uses column type from column definition for CHAR/VARCHAR/STRING
columns.
Change-Id: Ia3e43b2885853c2b4f207a45a873c9d7f31379cd
Reviewed-on: http://gerrit.cloudera.org:8080/3331
Reviewed-by: Huaisi Xu <hx...@cloudera.com>
Tested-by: Internal Jenkins
---
M fe/src/main/java/com/cloudera/impala/util/AvroSchemaConverter.java
M fe/src/main/java/com/cloudera/impala/util/AvroSchemaUtils.java
M testdata/workloads/functional-query/queries/QueryTest/avro-schema-changes.test
M testdata/workloads/functional-query/queries/QueryTest/avro-schema-resolution.test
M tests/query_test/test_avro_schema_resolution.py
5 files changed, 131 insertions(+), 17 deletions(-)
Approvals:
Huaisi Xu: Looks good to me, approved
Internal Jenkins: Verified
--
To view, visit http://gerrit.cloudera.org:8080/3331
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Ia3e43b2885853c2b4f207a45a873c9d7f31379cd
Gerrit-PatchSet: 14
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Dimitris Tsirogiannis <dt...@cloudera.com>
Gerrit-Reviewer: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Internal Jenkins
Gerrit-Reviewer: Juan Yu <jy...@cloudera.com>
[Impala-CR](cdh5-trunk) IMPALA-3687: Prefer Avro field name during schema reconciliation
Posted by "Dan Hecht (Code Review)" <ge...@cloudera.org>.
Dan Hecht has posted comments on this change.
Change subject: IMPALA-3687: Prefer Avro field name during schema reconciliation
......................................................................
Patch Set 5:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/3331/5//COMMIT_MSG
Commit Message:
PS5, Line 9: schema definition from schema definition
what does that mean?
--
To view, visit http://gerrit.cloudera.org:8080/3331
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia3e43b2885853c2b4f207a45a873c9d7f31379cd
Gerrit-PatchSet: 5
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Huaisi Xu <hx...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-CR](cdh5-trunk) IMPALA-3687: Prefer Avro field name during schema reconciliation
Posted by "Huaisi Xu (Code Review)" <ge...@cloudera.org>.
Huaisi Xu has posted comments on this change.
Change subject: IMPALA-3687: Prefer Avro field name during schema reconciliation
......................................................................
Patch Set 7:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/3331/6//COMMIT_MSG
Commit Message:
Line 9: Currently Impala prefers schema definition from schema definition
> How about something like the following blurb.
will do
http://gerrit.cloudera.org:8080/#/c/3331/6/fe/src/main/java/com/cloudera/impala/util/AvroSchemaUtils.java
File fe/src/main/java/com/cloudera/impala/util/AvroSchemaUtils.java:
Line 136: // as a CHAR/VARCHAR/STRING in the reconciled schema. Column name and comment
> I agree with Juan, and I think we all agree with her. This reconciliation l
I agree that we should not allow this things.
but I disagree that this change breaks compatibility. I think it was a bug that we treat string column the same as char/varchar column.
Maybe a purer bug fix is to only fix STRING column? what do you think?
Line 143: reconciledColDef.analyze();
> analysis should never fail, might as well try catch with a Preconditions ch
will do
http://gerrit.cloudera.org:8080/#/c/3331/6/testdata/workloads/functional-query/queries/QueryTest/avro-schema-resolution.test
File testdata/workloads/functional-query/queries/QueryTest/avro-schema-resolution.test:
Line 95: {"type": ["int", "null"], "name": "id_mismatch", "doc": "int comment mismatch"},
> Dimitris' request to include the original test case seems reasonable. We ca
Done
--
To view, visit http://gerrit.cloudera.org:8080/3331
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia3e43b2885853c2b4f207a45a873c9d7f31379cd
Gerrit-PatchSet: 7
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Dimitris Tsirogiannis <dt...@cloudera.com>
Gerrit-Reviewer: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Juan Yu <jy...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-CR](cdh5-trunk) IMPALA-3687: Prefer Avro field name during schema reconciliation
Posted by "Huaisi Xu (Code Review)" <ge...@cloudera.org>.
Huaisi Xu has posted comments on this change.
Change subject: IMPALA-3687: Prefer Avro field name during schema reconciliation
......................................................................
Patch Set 10:
just saw the comment.. will update a new patch
--
To view, visit http://gerrit.cloudera.org:8080/3331
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia3e43b2885853c2b4f207a45a873c9d7f31379cd
Gerrit-PatchSet: 10
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Dimitris Tsirogiannis <dt...@cloudera.com>
Gerrit-Reviewer: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Juan Yu <jy...@cloudera.com>
Gerrit-HasComments: No
[Impala-CR](cdh5-trunk) IMPALA-3687: Prefer Avro field name during schema reconciliation
Posted by "Huaisi Xu (Code Review)" <ge...@cloudera.org>.
Huaisi Xu has posted comments on this change.
Change subject: IMPALA-3687: Prefer Avro field name during schema reconciliation
......................................................................
Patch Set 13: Code-Review+2
carry +2
--
To view, visit http://gerrit.cloudera.org:8080/3331
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia3e43b2885853c2b4f207a45a873c9d7f31379cd
Gerrit-PatchSet: 13
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Dimitris Tsirogiannis <dt...@cloudera.com>
Gerrit-Reviewer: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Internal Jenkins
Gerrit-Reviewer: Juan Yu <jy...@cloudera.com>
Gerrit-HasComments: No
[Impala-CR](cdh5-trunk) IMPALA-3687: Prefer Avro field name during schema reconciliation
Posted by "Huaisi Xu (Code Review)" <ge...@cloudera.org>.
Hello Bharath Vissapragada, Alex Behm,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/3331
to look at the new patch set (#10).
Change subject: IMPALA-3687: Prefer Avro field name during schema reconciliation
......................................................................
IMPALA-3687: Prefer Avro field name during schema reconciliation
Since it is possible to create an Avro table with both column
definitions and an Avro schema, Impala attempts to reconcile
inconsistencies in the two schema definitions, generally preferring the
Avro schema. The only exception to this rule was with
CHAR/VARCHAR/STRING columns, where the column definition was preferred
in order to support tables with CHAR/VARCHAR columns although Avro only
supports STRING. This exception is confusing because the name for such a
column will be taken from the column definition (and not from the Avro
schema).
This patch prefers name, comment from Avro schema definition and
uses column type from column definition for CHAR/VARCHAR/STRING
columns.
Change-Id: Ia3e43b2885853c2b4f207a45a873c9d7f31379cd
---
M fe/src/main/java/com/cloudera/impala/util/AvroSchemaConverter.java
M fe/src/main/java/com/cloudera/impala/util/AvroSchemaUtils.java
M testdata/workloads/functional-query/queries/QueryTest/avro-schema-changes.test
M testdata/workloads/functional-query/queries/QueryTest/avro-schema-resolution.test
M tests/query_test/test_avro_schema_resolution.py
5 files changed, 132 insertions(+), 17 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/31/3331/10
--
To view, visit http://gerrit.cloudera.org:8080/3331
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia3e43b2885853c2b4f207a45a873c9d7f31379cd
Gerrit-PatchSet: 10
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Dimitris Tsirogiannis <dt...@cloudera.com>
Gerrit-Reviewer: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Juan Yu <jy...@cloudera.com>
[Impala-CR](cdh5-trunk) IMPALA-3687: Use type from Avro schema during schema reconciliation.
Posted by "Huaisi Xu (Code Review)" <ge...@cloudera.org>.
Huaisi Xu has posted comments on this change.
Change subject: IMPALA-3687: Use type from Avro schema during schema reconciliation.
......................................................................
Patch Set 4:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/3331/2/fe/src/main/java/com/cloudera/impala/util/AvroSchemaUtils.java
File fe/src/main/java/com/cloudera/impala/util/AvroSchemaUtils.java:
PS2, Line 116:
:
:
:
> based on my understanding, this comment was put here so that the reader und
I know but the code does not "reconcile" that at all right?
--
To view, visit http://gerrit.cloudera.org:8080/3331
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia3e43b2885853c2b4f207a45a873c9d7f31379cd
Gerrit-PatchSet: 4
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Huaisi Xu <hx...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-CR](cdh5-trunk) IMPALA-3687: Prefer Avro field name during schema reconciliation
Posted by "Huaisi Xu (Code Review)" <ge...@cloudera.org>.
Hello Bharath Vissapragada, Alex Behm,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/3331
to look at the new patch set (#11).
Change subject: IMPALA-3687: Prefer Avro field name during schema reconciliation
......................................................................
IMPALA-3687: Prefer Avro field name during schema reconciliation
Since it is possible to create an Avro table with both column
definitions and an Avro schema, Impala attempts to reconcile
inconsistencies in the two schema definitions, generally preferring the
Avro schema. The only exception to this rule was with
CHAR/VARCHAR/STRING columns, where the column definition was preferred
in order to support tables with CHAR/VARCHAR columns although Avro only
supports STRING. This exception is confusing because the name for such a
column will be taken from the column definition (and not from the Avro
schema).
This patch prefers name, comment from Avro schema definition and
uses column type from column definition for CHAR/VARCHAR/STRING
columns.
Change-Id: Ia3e43b2885853c2b4f207a45a873c9d7f31379cd
---
M fe/src/main/java/com/cloudera/impala/util/AvroSchemaConverter.java
M fe/src/main/java/com/cloudera/impala/util/AvroSchemaUtils.java
M testdata/workloads/functional-query/queries/QueryTest/avro-schema-changes.test
M testdata/workloads/functional-query/queries/QueryTest/avro-schema-resolution.test
M tests/query_test/test_avro_schema_resolution.py
5 files changed, 133 insertions(+), 17 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/31/3331/11
--
To view, visit http://gerrit.cloudera.org:8080/3331
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia3e43b2885853c2b4f207a45a873c9d7f31379cd
Gerrit-PatchSet: 11
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Dimitris Tsirogiannis <dt...@cloudera.com>
Gerrit-Reviewer: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Juan Yu <jy...@cloudera.com>
[Impala-CR](cdh5-trunk) IMPALA-3687: Prefer Avro field name during schema reconciliation
Posted by "Internal Jenkins (Code Review)" <ge...@cloudera.org>.
Internal Jenkins has posted comments on this change.
Change subject: IMPALA-3687: Prefer Avro field name during schema reconciliation
......................................................................
Patch Set 12: Verified-1
Build failed: http://sandbox.jenkins.cloudera.com/job/impala-external-gerrit-verify-merge/2581/
--
To view, visit http://gerrit.cloudera.org:8080/3331
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia3e43b2885853c2b4f207a45a873c9d7f31379cd
Gerrit-PatchSet: 12
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Dimitris Tsirogiannis <dt...@cloudera.com>
Gerrit-Reviewer: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Internal Jenkins
Gerrit-Reviewer: Juan Yu <jy...@cloudera.com>
Gerrit-HasComments: No
[Impala-CR](cdh5-trunk) IMPALA-3687: Prefer Avro field name during schema reconciliation
Posted by "Huaisi Xu (Code Review)" <ge...@cloudera.org>.
Huaisi Xu has posted comments on this change.
Change subject: IMPALA-3687: Prefer Avro field name during schema reconciliation
......................................................................
Patch Set 5:
(5 comments)
> Also, can you plz add the test that follows the reproduction steps
> from the JIRA? i.e. create table, alter column in schema file,
> invalidate metadata and select.
> Also, can you plz add the test that follows the reproduction steps
> from the JIRA? i.e. create table, alter column in schema file,
> invalidate metadata and select.
that is a little "intrusive" to our code base as it needs to load a changed schema file. avro.schema.literal has the same code path as avro.schema.url.
I can change the reproduction step in the Jira though...
http://gerrit.cloudera.org:8080/#/c/3331/5/fe/src/main/java/com/cloudera/impala/util/AvroSchemaUtils.java
File fe/src/main/java/com/cloudera/impala/util/AvroSchemaUtils.java:
PS5, Line 114:
> "from the column definition"
good to say "avro schema".
PS5, Line 112: Mismatched name/type -> Prefer Avro column, except:
: * A CHAR/VARCHAR column definition maps to an Avro STRING. The reconciled column
: * will preserve the CHAR/VARCHAR type but prefer the column name and comment from
: * the Avro column.
> Maybe it will be more clear if you have:
Did in another way.
PS5, Line 119: AnalysisException
> You need to comment when this function throws an AnalysisException.
not sure how to comment.. analyze() can throw but I think it should have a comment in the first place...
Line 136: // as a CHAR/VARCHAR in the reconciled schema.
> Comment that the name and comment come from the Avro schema.
Done
http://gerrit.cloudera.org:8080/#/c/3331/5/testdata/workloads/functional-query/queries/QueryTest/avro-schema-resolution.test
File testdata/workloads/functional-query/queries/QueryTest/avro-schema-resolution.test:
PS5, Line 80: INT
> Can we add a type mismatch as well? e.g. make this float
Done
--
To view, visit http://gerrit.cloudera.org:8080/3331
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia3e43b2885853c2b4f207a45a873c9d7f31379cd
Gerrit-PatchSet: 5
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Dimitris Tsirogiannis <dt...@cloudera.com>
Gerrit-Reviewer: Huaisi Xu <hx...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-CR](cdh5-trunk) IMPALA-3687: Use type from avro schema during schema reconcilation.
Posted by "Huaisi Xu (Code Review)" <ge...@cloudera.org>.
Huaisi Xu has uploaded a new patch set (#2).
Change subject: IMPALA-3687: Use type from avro schema during schema reconcilation.
......................................................................
IMPALA-3687: Use type from avro schema during schema reconcilation.
There is no effect that we change the name of a string/char/varchar
column in schema definition because impala uses column defnition
for these types. This patch changes the behavior to that impala
only uses type from column definition. This is to preserve support
for CHAR/VARCHAR column type.
Change-Id: Ia3e43b2885853c2b4f207a45a873c9d7f31379cd
---
M fe/src/main/java/com/cloudera/impala/util/AvroSchemaUtils.java
M testdata/workloads/functional-query/queries/QueryTest/avro-schema-changes.test
2 files changed, 58 insertions(+), 8 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/31/3331/2
--
To view, visit http://gerrit.cloudera.org:8080/3331
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia3e43b2885853c2b4f207a45a873c9d7f31379cd
Gerrit-PatchSet: 2
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
[Impala-CR](cdh5-trunk) IMPALA-3687: Prefer Avro field name during schema reconciliation
Posted by "Huaisi Xu (Code Review)" <ge...@cloudera.org>.
Huaisi Xu has posted comments on this change.
Change subject: IMPALA-3687: Prefer Avro field name during schema reconciliation
......................................................................
Patch Set 11:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/3331/11/testdata/workloads/functional-query/queries/QueryTest/avro-schema-resolution.test
File testdata/workloads/functional-query/queries/QueryTest/avro-schema-resolution.test:
PS11, Line 165: ions except for CHAR/VARCHAR,
will update this..
--
To view, visit http://gerrit.cloudera.org:8080/3331
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia3e43b2885853c2b4f207a45a873c9d7f31379cd
Gerrit-PatchSet: 11
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Dimitris Tsirogiannis <dt...@cloudera.com>
Gerrit-Reviewer: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Juan Yu <jy...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-CR](cdh5-trunk) IMPALA-3687: Use type from Avro schema during schema reconciliation.
Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.
Bharath Vissapragada has posted comments on this change.
Change subject: IMPALA-3687: Use type from Avro schema during schema reconciliation.
......................................................................
Patch Set 4: Code-Review+1
(1 comment)
http://gerrit.cloudera.org:8080/#/c/3331/2/fe/src/main/java/com/cloudera/impala/util/AvroSchemaUtils.java
File fe/src/main/java/com/cloudera/impala/util/AvroSchemaUtils.java:
PS2, Line 116:
:
:
:
> because we do not have any logic for treating timestamp differently?
based on my understanding, this comment was put here so that the reader understands that timestamp follows the codepath of a string type which we deal differently. Its better to retain this. (Please confirm with Alex)
--
To view, visit http://gerrit.cloudera.org:8080/3331
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia3e43b2885853c2b4f207a45a873c9d7f31379cd
Gerrit-PatchSet: 4
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Huaisi Xu <hx...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-CR](cdh5-trunk) IMPALA-3687: Prefer Avro field name during schema reconciliation
Posted by "Alex Behm (Code Review)" <ge...@cloudera.org>.
Alex Behm has posted comments on this change.
Change subject: IMPALA-3687: Prefer Avro field name during schema reconciliation
......................................................................
Patch Set 9: Code-Review+2
(4 comments)
http://gerrit.cloudera.org:8080/#/c/3331/6/fe/src/main/java/com/cloudera/impala/util/AvroSchemaUtils.java
File fe/src/main/java/com/cloudera/impala/util/AvroSchemaUtils.java:
Line 136: // are taken from the Avro schema.
> I agree that we should not allow this things.
Agreed that the current change is a bug fix. Based on my understanding and the comments in the code, it was never intended for the field name to be picked up from the column definitions. Only the type. Let's take your patch as is.
http://gerrit.cloudera.org:8080/#/c/3331/9/fe/src/main/java/com/cloudera/impala/util/AvroSchemaUtils.java
File fe/src/main/java/com/cloudera/impala/util/AvroSchemaUtils.java:
Line 145: Preconditions.checkNotNull(null, "reconciledColDef.Analyze should never throw");
reconciledColDef.analyze() should never throw
http://gerrit.cloudera.org:8080/#/c/3331/9/testdata/workloads/functional-query/queries/QueryTest/avro-schema-resolution.test
File testdata/workloads/functional-query/queries/QueryTest/avro-schema-resolution.test:
PS9, Line 78: IMPALA
use colon, i.e. IMPALA-3687:
Line 128: # IMPALA-3687. Create a table with column definition and later alter its Avro schema to
use colon, i.e. IMPALA-3687:
... column definitions ...
--
To view, visit http://gerrit.cloudera.org:8080/3331
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia3e43b2885853c2b4f207a45a873c9d7f31379cd
Gerrit-PatchSet: 9
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Dimitris Tsirogiannis <dt...@cloudera.com>
Gerrit-Reviewer: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Juan Yu <jy...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-CR](cdh5-trunk) IMPALA-3687: Use type from avro schema during schema reconcilation.
Posted by "Huaisi Xu (Code Review)" <ge...@cloudera.org>.
Huaisi Xu has posted comments on this change.
Change subject: IMPALA-3687: Use type from avro schema during schema reconcilation.
......................................................................
Patch Set 2:
(1 comment)
I think we have to continue supporting char/varchar for avro?
http://gerrit.cloudera.org:8080/#/c/3331/1/testdata/workloads/functional-query/queries/QueryTest/avro-schema-changes.test
File testdata/workloads/functional-query/queries/QueryTest/avro-schema-changes.test:
Line 72: CREATE EXTERNAL TABLE alltypesagg_changed_column_name_comment (
I would rather prefer to do this in Hive. you know how?
--
To view, visit http://gerrit.cloudera.org:8080/3331
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia3e43b2885853c2b4f207a45a873c9d7f31379cd
Gerrit-PatchSet: 2
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Huaisi Xu <hx...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-CR](cdh5-trunk) IMPALA-3687: Prefer Avro field name during schema reconciliation
Posted by "Huaisi Xu (Code Review)" <ge...@cloudera.org>.
Hello Bharath Vissapragada,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/3331
to look at the new patch set (#7).
Change subject: IMPALA-3687: Prefer Avro field name during schema reconciliation
......................................................................
IMPALA-3687: Prefer Avro field name during schema reconciliation
Currently Impala prefers schema definition from schema definition
except for CHAR/VARCHAR/STRING columns. As a result, Impala ignores
schema changes made in avro.schema.url or avro.schema.literal.
This patch prefers name, comment from Avro schema definition and
uses column type from column definition for CHAR/VARCHAR/STRING
columns.
Change-Id: Ia3e43b2885853c2b4f207a45a873c9d7f31379cd
---
M fe/src/main/java/com/cloudera/impala/util/AvroSchemaConverter.java
M fe/src/main/java/com/cloudera/impala/util/AvroSchemaUtils.java
M testdata/workloads/functional-query/queries/QueryTest/avro-schema-changes.test
M testdata/workloads/functional-query/queries/QueryTest/avro-schema-resolution.test
M tests/query_test/test_avro_schema_resolution.py
5 files changed, 127 insertions(+), 18 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/31/3331/7
--
To view, visit http://gerrit.cloudera.org:8080/3331
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia3e43b2885853c2b4f207a45a873c9d7f31379cd
Gerrit-PatchSet: 7
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Dimitris Tsirogiannis <dt...@cloudera.com>
Gerrit-Reviewer: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Juan Yu <jy...@cloudera.com>
[Impala-CR](cdh5-trunk) IMPALA-3687: Prefer Avro field name during schema reconciliation
Posted by "Dimitris Tsirogiannis (Code Review)" <ge...@cloudera.org>.
Dimitris Tsirogiannis has posted comments on this change.
Change subject: IMPALA-3687: Prefer Avro field name during schema reconciliation
......................................................................
Patch Set 5:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/3331/5/fe/src/main/java/com/cloudera/impala/util/AvroSchemaUtils.java
File fe/src/main/java/com/cloudera/impala/util/AvroSchemaUtils.java:
PS5, Line 119: AnalysisException
> not sure how to comment.. analyze() can throw but I think it should have a
In the context of this particular function, what is interesting is that an AnalysisException can now be thrown if we fail to analyze a reconciled column definition for a String type, no?
--
To view, visit http://gerrit.cloudera.org:8080/3331
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia3e43b2885853c2b4f207a45a873c9d7f31379cd
Gerrit-PatchSet: 5
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Dimitris Tsirogiannis <dt...@cloudera.com>
Gerrit-Reviewer: Huaisi Xu <hx...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-CR](cdh5-trunk) IMPALA-3687: Prefer Avro field name during schema reconciliation
Posted by "Dimitris Tsirogiannis (Code Review)" <ge...@cloudera.org>.
Dimitris Tsirogiannis has posted comments on this change.
Change subject: IMPALA-3687: Prefer Avro field name during schema reconciliation
......................................................................
Patch Set 6:
(1 comment)
I still think that we're missing some testing for this patch. What was the original use for which this jira was filed? If it the one mentioned in the JIRA, I don't see why we can generate a test case for it? Can't we simply have two files pointing to two slightly different Avro schemas and apply the steps from the JIRA. Dan what do you think?
http://gerrit.cloudera.org:8080/#/c/3331/6/fe/src/main/java/com/cloudera/impala/util/AvroSchemaUtils.java
File fe/src/main/java/com/cloudera/impala/util/AvroSchemaUtils.java:
PS6, Line 111: columns
does it mean "schema"?
--
To view, visit http://gerrit.cloudera.org:8080/3331
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia3e43b2885853c2b4f207a45a873c9d7f31379cd
Gerrit-PatchSet: 6
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Dimitris Tsirogiannis <dt...@cloudera.com>
Gerrit-Reviewer: Huaisi Xu <hx...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-CR](cdh5-trunk) IMPALA-3687: Use type from avro schema during schema reconcilation.
Posted by "Huaisi Xu (Code Review)" <ge...@cloudera.org>.
Huaisi Xu has posted comments on this change.
Change subject: IMPALA-3687: Use type from avro schema during schema reconcilation.
......................................................................
Patch Set 3:
(8 comments)
http://gerrit.cloudera.org:8080/#/c/3331/2//COMMIT_MSG
Commit Message:
PS2, Line 7: avro
> nit: Avro (capitalization)
Done
PS2, Line 7: reconcilation
> nit:typo
Done
PS2, Line 9: string/char/varchar
> nit:capitalize for consitency.
Done
PS2, Line 9: here is no effect that we change the
> I think the commit message can be reworded to indicate the issue we are try
> "having different column name for string types in Avro schema and hive col defs can raise an error"
I think there is no error? which error were you referring to? it is the way which we choose? the end result is "there is no effect" since we chose column definition?
http://gerrit.cloudera.org:8080/#/c/3331/2/fe/src/main/java/com/cloudera/impala/util/AvroSchemaUtils.java
File fe/src/main/java/com/cloudera/impala/util/AvroSchemaUtils.java:
PS2, Line 116:
:
:
:
> Why did you remove this?
because we do not have any logic for treating timestamp differently?
http://gerrit.cloudera.org:8080/#/c/3331/1/testdata/workloads/functional-query/queries/QueryTest/avro-schema-changes.test
File testdata/workloads/functional-query/queries/QueryTest/avro-schema-changes.test:
Line 72: CREATE EXTERNAL TABLE alltypesagg_changed_column_name_comment (
> Why in hive?
if it is done in impala then frontend analysis will reconcile that but if in Hive, catalog will reconcile this. they use the same code though. I am good with either way..
http://gerrit.cloudera.org:8080/#/c/3331/2/testdata/workloads/functional-query/queries/QueryTest/avro-schema-changes.test
File testdata/workloads/functional-query/queries/QueryTest/avro-schema-changes.test:
PS2, Line 71: Avro
> nit: Avro
Done
PS2, Line 71: # IMPALA-3687
> I think avro-schema-resolution.test is a better place for this.
I am fine with anything.. what do you think @Alex?
--
To view, visit http://gerrit.cloudera.org:8080/3331
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia3e43b2885853c2b4f207a45a873c9d7f31379cd
Gerrit-PatchSet: 3
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Huaisi Xu <hx...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-CR](cdh5-trunk) IMPALA-3687: Prefer Avro field name during schema reconciliation
Posted by "Huaisi Xu (Code Review)" <ge...@cloudera.org>.
Huaisi Xu has posted comments on this change.
Change subject: IMPALA-3687: Prefer Avro field name during schema reconciliation
......................................................................
Patch Set 7:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/3331/7/testdata/workloads/functional-query/queries/QueryTest/avro-schema-resolution.test
File testdata/workloads/functional-query/queries/QueryTest/avro-schema-resolution.test:
Line 128: # IMPALA-3687. Create a table with mismatched column definitions and Avro schema.
will update.
--
To view, visit http://gerrit.cloudera.org:8080/3331
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia3e43b2885853c2b4f207a45a873c9d7f31379cd
Gerrit-PatchSet: 7
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Dimitris Tsirogiannis <dt...@cloudera.com>
Gerrit-Reviewer: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Juan Yu <jy...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-CR](cdh5-trunk) IMPALA-3687: Use type from Avro schema during schema reconciliation.
Posted by "Huaisi Xu (Code Review)" <ge...@cloudera.org>.
Huaisi Xu has uploaded a new patch set (#4).
Change subject: IMPALA-3687: Use type from Avro schema during schema reconciliation.
......................................................................
IMPALA-3687: Use type from Avro schema during schema reconciliation.
There is no effect that we change the name of a STRING/CHAR/VARCHAR
column in schema definition because impala uses column defnition
for these types. This patch changes the behavior to that impala
only uses type from column definition. This is to preserve support
for CHAR/VARCHAR column type.
Change-Id: Ia3e43b2885853c2b4f207a45a873c9d7f31379cd
---
M fe/src/main/java/com/cloudera/impala/util/AvroSchemaUtils.java
M testdata/workloads/functional-query/queries/QueryTest/avro-schema-changes.test
2 files changed, 59 insertions(+), 9 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/31/3331/4
--
To view, visit http://gerrit.cloudera.org:8080/3331
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia3e43b2885853c2b4f207a45a873c9d7f31379cd
Gerrit-PatchSet: 4
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Huaisi Xu <hx...@cloudera.com>
[Impala-CR](cdh5-trunk) IMPALA-3687: Prefer Avro field name during schema reconciliation
Posted by "Huaisi Xu (Code Review)" <ge...@cloudera.org>.
Huaisi Xu has posted comments on this change.
Change subject: IMPALA-3687: Prefer Avro field name during schema reconciliation
......................................................................
Patch Set 12: Code-Review+2
Carry Alex's +2. will gvm after lunch. in case Alex wants to have a final look!
--
To view, visit http://gerrit.cloudera.org:8080/3331
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia3e43b2885853c2b4f207a45a873c9d7f31379cd
Gerrit-PatchSet: 12
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Dimitris Tsirogiannis <dt...@cloudera.com>
Gerrit-Reviewer: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Juan Yu <jy...@cloudera.com>
Gerrit-HasComments: No
[Impala-CR](cdh5-trunk) IMPALA-3687: Prefer Avro field name during schema reconciliation
Posted by "Dimitris Tsirogiannis (Code Review)" <ge...@cloudera.org>.
Dimitris Tsirogiannis has posted comments on this change.
Change subject: IMPALA-3687: Prefer Avro field name during schema reconciliation
......................................................................
Patch Set 5:
Also, can you plz add the test that follows the reproduction steps from the JIRA? i.e. create table, alter column in schema file, invalidate metadata and select.
--
To view, visit http://gerrit.cloudera.org:8080/3331
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia3e43b2885853c2b4f207a45a873c9d7f31379cd
Gerrit-PatchSet: 5
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Dimitris Tsirogiannis <dt...@cloudera.com>
Gerrit-Reviewer: Huaisi Xu <hx...@cloudera.com>
Gerrit-HasComments: No
[Impala-CR](cdh5-trunk) IMPALA-3687: Use type from Avro schema during schema reconciliation.
Posted by "Huaisi Xu (Code Review)" <ge...@cloudera.org>.
Huaisi Xu has posted comments on this change.
Change subject: IMPALA-3687: Use type from Avro schema during schema reconciliation.
......................................................................
Patch Set 4:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/3331/4//COMMIT_MSG
Commit Message:
Line 7: IMPALA-3687: Use type from Avro schema during schema reconciliation.
> Always prefer the Avro field name during schema reconciliation.
It is unclear to me what "field name" refers to... I think we should explicitly say "from schema definition"..? maybe I missed something so I will use your suggestion..
Line 9: There is no effect that we change the name of a STRING/CHAR/VARCHAR
> This blurb is very difficult to follow. It would be good to summarize the n
Done. I think "but keep the name and comment from the Avro column." should be "but keep the name and comment from Avro schema definition."
http://gerrit.cloudera.org:8080/#/c/3331/2/fe/src/main/java/com/cloudera/impala/util/AvroSchemaUtils.java
File fe/src/main/java/com/cloudera/impala/util/AvroSchemaUtils.java:
PS2, Line 116:
:
:
:
> Bharath is correct that the comment had an original intention. The comment
Done
http://gerrit.cloudera.org:8080/#/c/3331/4/fe/src/main/java/com/cloudera/impala/util/AvroSchemaUtils.java
File fe/src/main/java/com/cloudera/impala/util/AvroSchemaUtils.java:
PS4, Line 113: A
> grammar; also could be rephrased for clarity, e.g.:
Done
http://gerrit.cloudera.org:8080/#/c/3331/4/testdata/workloads/functional-query/queries/QueryTest/avro-schema-changes.test
File testdata/workloads/functional-query/queries/QueryTest/avro-schema-changes.test:
Line 71: # IMPALA-3687. Create a table that has different Avro schema from its column definition
> Create a table with mismatched column definitions and Avro schema.
Done
Line 86: LOCATION '$FILESYSTEM_PREFIX/test-warehouse/alltypesaggmultifilesnopart_avro_snap'
> remove this clause - it only adds confusion
Done
Line 95: {"type": ["float", "null"], "name": "float_col_changed", "doc": "float comment changed"},
> typo: changed
since we put this in avro-schema-resolution, I changed this to *_mismatch
--
To view, visit http://gerrit.cloudera.org:8080/3331
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia3e43b2885853c2b4f207a45a873c9d7f31379cd
Gerrit-PatchSet: 4
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Huaisi Xu <hx...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-CR](cdh5-trunk) IMPALA-3687: Prefer Avro field name during schema reconciliation
Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.
Bharath Vissapragada has posted comments on this change.
Change subject: IMPALA-3687: Prefer Avro field name during schema reconciliation
......................................................................
Patch Set 6:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/3331/6/fe/src/main/java/com/cloudera/impala/util/AvroSchemaUtils.java
File fe/src/main/java/com/cloudera/impala/util/AvroSchemaUtils.java:
Line 136: // as a CHAR/VARCHAR/STRING in the reconciled schema. Column name and comment
> Just my two cents. Why don't we just throw an error if they are inconsisten
I discussed this with Alex in the past. From what I understand all this reconciliation logic was done to be consistent with Hive (although there are quite a few rough edges). May be he can give more context once he is back from vacation.
--
To view, visit http://gerrit.cloudera.org:8080/3331
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia3e43b2885853c2b4f207a45a873c9d7f31379cd
Gerrit-PatchSet: 6
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Dimitris Tsirogiannis <dt...@cloudera.com>
Gerrit-Reviewer: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Juan Yu <jy...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-CR](cdh5-trunk) IMPALA-3687: Prefer Avro field name during schema reconciliation
Posted by "Huaisi Xu (Code Review)" <ge...@cloudera.org>.
Huaisi Xu has posted comments on this change.
Change subject: IMPALA-3687: Prefer Avro field name during schema reconciliation
......................................................................
Patch Set 6:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/3331/6/testdata/workloads/functional-query/queries/QueryTest/avro-schema-resolution.test
File testdata/workloads/functional-query/queries/QueryTest/avro-schema-resolution.test:
PS6, Line 18: unctional_avro_snap
because we wont be woking under this database for this test.
http://gerrit.cloudera.org:8080/#/c/3331/6/tests/query_test/test_avro_schema_resolution.py
File tests/query_test/test_avro_schema_resolution.py:
PS6, Line 23: unique_databas
I added this in order not to create if not exist and drop that table because Bharath and Alex suggest me to move the test to avro-schema-resolution.test.
--
To view, visit http://gerrit.cloudera.org:8080/3331
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia3e43b2885853c2b4f207a45a873c9d7f31379cd
Gerrit-PatchSet: 6
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Dimitris Tsirogiannis <dt...@cloudera.com>
Gerrit-Reviewer: Huaisi Xu <hx...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-CR](cdh5-trunk) IMPALA-3687: Prefer Avro field name during schema reconciliation
Posted by "Huaisi Xu (Code Review)" <ge...@cloudera.org>.
Hello Bharath Vissapragada, Alex Behm,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/3331
to look at the new patch set (#12).
Change subject: IMPALA-3687: Prefer Avro field name during schema reconciliation
......................................................................
IMPALA-3687: Prefer Avro field name during schema reconciliation
Since it is possible to create an Avro table with both column
definitions and an Avro schema, Impala attempts to reconcile
inconsistencies in the two schema definitions, generally preferring the
Avro schema. The only exception to this rule was with
CHAR/VARCHAR/STRING columns, where the column definition was preferred
in order to support tables with CHAR/VARCHAR columns although Avro only
supports STRING. This exception is confusing because the name for such a
column will be taken from the column definition (and not from the Avro
schema).
This patch prefers name, comment from Avro schema definition and
uses column type from column definition for CHAR/VARCHAR/STRING
columns.
Change-Id: Ia3e43b2885853c2b4f207a45a873c9d7f31379cd
---
M fe/src/main/java/com/cloudera/impala/util/AvroSchemaConverter.java
M fe/src/main/java/com/cloudera/impala/util/AvroSchemaUtils.java
M testdata/workloads/functional-query/queries/QueryTest/avro-schema-changes.test
M testdata/workloads/functional-query/queries/QueryTest/avro-schema-resolution.test
M tests/query_test/test_avro_schema_resolution.py
5 files changed, 131 insertions(+), 17 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/31/3331/12
--
To view, visit http://gerrit.cloudera.org:8080/3331
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia3e43b2885853c2b4f207a45a873c9d7f31379cd
Gerrit-PatchSet: 12
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Dimitris Tsirogiannis <dt...@cloudera.com>
Gerrit-Reviewer: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Juan Yu <jy...@cloudera.com>
[Impala-CR](cdh5-trunk) IMPALA-3687: Prefer Avro field name during schema reconciliation
Posted by "Huaisi Xu (Code Review)" <ge...@cloudera.org>.
Hello Bharath Vissapragada,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/3331
to look at the new patch set (#6).
Change subject: IMPALA-3687: Prefer Avro field name during schema reconciliation
......................................................................
IMPALA-3687: Prefer Avro field name during schema reconciliation
Currently Impala prefers schema definition from schema definition
except for CHAR/VARCHAR/STRING columns. As a result, Impala ignores
schema changes made in avro.schema.url or avro.schema.literal.
This patch prefers name, comment from Avro schema definition and
uses column type from column definition for CHAR/VARCHAR/STRING
columns.
Change-Id: Ia3e43b2885853c2b4f207a45a873c9d7f31379cd
---
M fe/src/main/java/com/cloudera/impala/util/AvroSchemaConverter.java
M fe/src/main/java/com/cloudera/impala/util/AvroSchemaUtils.java
M testdata/workloads/functional-query/queries/QueryTest/avro-schema-changes.test
M testdata/workloads/functional-query/queries/QueryTest/avro-schema-resolution.test
M tests/query_test/test_avro_schema_resolution.py
5 files changed, 74 insertions(+), 18 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/31/3331/6
--
To view, visit http://gerrit.cloudera.org:8080/3331
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia3e43b2885853c2b4f207a45a873c9d7f31379cd
Gerrit-PatchSet: 6
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Dimitris Tsirogiannis <dt...@cloudera.com>
Gerrit-Reviewer: Huaisi Xu <hx...@cloudera.com>
[Impala-CR](cdh5-trunk) IMPALA-3687: Prefer Avro field name during schema reconciliation
Posted by "Huaisi Xu (Code Review)" <ge...@cloudera.org>.
Hello Bharath Vissapragada,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/3331
to look at the new patch set (#8).
Change subject: IMPALA-3687: Prefer Avro field name during schema reconciliation
......................................................................
IMPALA-3687: Prefer Avro field name during schema reconciliation
Since it is possible to create an Avro table with both column
definitions and an Avro schema, Impala attempts to reconcile
inconsistencies in the two schema definitions, generally preferring the
Avro schema. The only exception to this rule was with
CHAR/VARCHAR/STRING columns, where the column definition was preferred
in order to support tables with CHAR/VARCHAR columns although Avro only
supports STRING. This exception is confusing because the name for such a
column will be taken from the column definition (and not from the Avro
schema).
This patch prefers name, comment from Avro schema definition and
uses column type from column definition for CHAR/VARCHAR/STRING
columns.
Change-Id: Ia3e43b2885853c2b4f207a45a873c9d7f31379cd
---
M fe/src/main/java/com/cloudera/impala/util/AvroSchemaConverter.java
M fe/src/main/java/com/cloudera/impala/util/AvroSchemaUtils.java
M testdata/workloads/functional-query/queries/QueryTest/avro-schema-changes.test
M testdata/workloads/functional-query/queries/QueryTest/avro-schema-resolution.test
M tests/query_test/test_avro_schema_resolution.py
5 files changed, 129 insertions(+), 17 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/31/3331/8
--
To view, visit http://gerrit.cloudera.org:8080/3331
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia3e43b2885853c2b4f207a45a873c9d7f31379cd
Gerrit-PatchSet: 8
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Dimitris Tsirogiannis <dt...@cloudera.com>
Gerrit-Reviewer: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Juan Yu <jy...@cloudera.com>
[Impala-CR](cdh5-trunk) IMPALA-3687: Prefer Avro field name during schema reconciliation
Posted by "Juan Yu (Code Review)" <ge...@cloudera.org>.
Juan Yu has posted comments on this change.
Change subject: IMPALA-3687: Prefer Avro field name during schema reconciliation
......................................................................
Patch Set 6:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/3331/6/fe/src/main/java/com/cloudera/impala/util/AvroSchemaUtils.java
File fe/src/main/java/com/cloudera/impala/util/AvroSchemaUtils.java:
Line 136: // as a CHAR/VARCHAR/STRING in the reconciled schema. Column name and comment
Just my two cents. Why don't we just throw an error if they are inconsistent, and ask user fix the table schema by alter column definition?
I am not sure trying to fix it silently is a good thing. What if user indeed prefer the column definition.
--
To view, visit http://gerrit.cloudera.org:8080/3331
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia3e43b2885853c2b4f207a45a873c9d7f31379cd
Gerrit-PatchSet: 6
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Dimitris Tsirogiannis <dt...@cloudera.com>
Gerrit-Reviewer: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Juan Yu <jy...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-CR](cdh5-trunk) IMPALA-3687: Prefer Avro field name during schema reconciliation
Posted by "Huaisi Xu (Code Review)" <ge...@cloudera.org>.
Huaisi Xu has posted comments on this change.
Change subject: IMPALA-3687: Prefer Avro field name during schema reconciliation
......................................................................
Patch Set 6:
(3 comments)
> (1 comment)
>
> I still think that we're missing some testing for this patch. What
> was the original use for which this jira was filed? If it the one
> mentioned in the JIRA, I don't see why we can generate a test case
> for it? Can't we simply have two files pointing to two slightly
> different Avro schemas and apply the steps from the JIRA. Dan what
> do you think?
but avro.schema.url and avro.schema.literal are the same... ?
http://gerrit.cloudera.org:8080/#/c/3331/5/fe/src/main/java/com/cloudera/impala/util/AvroSchemaUtils.java
File fe/src/main/java/com/cloudera/impala/util/AvroSchemaUtils.java:
PS5, Line 119: AnalysisException
> In the context of this particular function, what is interesting is that an
Done
http://gerrit.cloudera.org:8080/#/c/3331/6/fe/src/main/java/com/cloudera/impala/util/AvroSchemaUtils.java
File fe/src/main/java/com/cloudera/impala/util/AvroSchemaUtils.java:
PS6, Line 111: columns
> does it mean "schema"?
will do
Line 136: // as a CHAR/VARCHAR/STRING in the reconciled schema. Column name and comment
> I discussed this with Alex in the past. From what I understand all this rec
another reason is that we did this before so it is hard to drop it. ideally we should not support char/varchar, which is not supported in avro's standard. hive does not support this case as well. but it is hard for us to "go back".
--
To view, visit http://gerrit.cloudera.org:8080/3331
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia3e43b2885853c2b4f207a45a873c9d7f31379cd
Gerrit-PatchSet: 6
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Dimitris Tsirogiannis <dt...@cloudera.com>
Gerrit-Reviewer: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Juan Yu <jy...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-CR](cdh5-trunk) IMPALA-3687: Prefer Avro field name during schema reconciliation
Posted by "Huaisi Xu (Code Review)" <ge...@cloudera.org>.
Huaisi Xu has posted comments on this change.
Change subject: IMPALA-3687: Prefer Avro field name during schema reconciliation
......................................................................
Patch Set 5:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/3331/5//COMMIT_MSG
Commit Message:
PS5, Line 9: schema definition from schema definition
> what does that mean?
sorry... :-) I can delete the first "schema definition"
If a table is:
create table aa (c int) <-column definition
stored as AVRO
tblproperties('avro.schema.url'='xxxx')<- schema definition I referred to
or
tblproperties('avro.schema.literal'='xxxx') <- schema definition I referred to
So impala preferred the later case.
--
To view, visit http://gerrit.cloudera.org:8080/3331
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia3e43b2885853c2b4f207a45a873c9d7f31379cd
Gerrit-PatchSet: 5
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Huaisi Xu <hx...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-CR](cdh5-trunk) IMPALA-3687: Prefer Avro field name during schema reconciliation
Posted by "Huaisi Xu (Code Review)" <ge...@cloudera.org>.
Hello Bharath Vissapragada,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/3331
to look at the new patch set (#5).
Change subject: IMPALA-3687: Prefer Avro field name during schema reconciliation
......................................................................
IMPALA-3687: Prefer Avro field name during schema reconciliation
Currently Impala prefers schema definition from schema definition
except for CHAR/VARCHAR/STRING columns. As a result, Impala ignores
schema changes made in avro.schema.url or avro.schema.literal.
This patch prefers name, comment from Avro schema definition and
uses column type from column definition for CHAR/VARCHAR/STRING
columns.
Change-Id: Ia3e43b2885853c2b4f207a45a873c9d7f31379cd
---
M fe/src/main/java/com/cloudera/impala/util/AvroSchemaConverter.java
M fe/src/main/java/com/cloudera/impala/util/AvroSchemaUtils.java
M testdata/workloads/functional-query/queries/QueryTest/avro-schema-changes.test
M testdata/workloads/functional-query/queries/QueryTest/avro-schema-resolution.test
4 files changed, 65 insertions(+), 10 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/31/3331/5
--
To view, visit http://gerrit.cloudera.org:8080/3331
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia3e43b2885853c2b4f207a45a873c9d7f31379cd
Gerrit-PatchSet: 5
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Huaisi Xu <hx...@cloudera.com>
[Impala-CR](cdh5-trunk) IMPALA-3687: Prefer Avro field name during schema reconciliation
Posted by "Alex Behm (Code Review)" <ge...@cloudera.org>.
Alex Behm has posted comments on this change.
Change subject: IMPALA-3687: Prefer Avro field name during schema reconciliation
......................................................................
Patch Set 12: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/3331
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia3e43b2885853c2b4f207a45a873c9d7f31379cd
Gerrit-PatchSet: 12
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Dimitris Tsirogiannis <dt...@cloudera.com>
Gerrit-Reviewer: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Juan Yu <jy...@cloudera.com>
Gerrit-HasComments: No
[Impala-CR](cdh5-trunk) IMPALA-3687: Prefer Avro field name during schema reconciliation
Posted by "Dimitris Tsirogiannis (Code Review)" <ge...@cloudera.org>.
Dimitris Tsirogiannis has posted comments on this change.
Change subject: IMPALA-3687: Prefer Avro field name during schema reconciliation
......................................................................
Patch Set 5:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/3331/5/fe/src/main/java/com/cloudera/impala/util/AvroSchemaUtils.java
File fe/src/main/java/com/cloudera/impala/util/AvroSchemaUtils.java:
PS5, Line 114:
"from the column definition"
PS5, Line 112: Mismatched name/type -> Prefer Avro column, except:
: * A CHAR/VARCHAR column definition maps to an Avro STRING. The reconciled column
: * will preserve the CHAR/VARCHAR type but prefer the column name and comment from
: * the Avro column.
Maybe it will be more clear if you have:
Mismatched type -> Prefer ... except
Mismatched name/comment -> Prefer
PS5, Line 119: AnalysisException
You need to comment when this function throws an AnalysisException.
Line 136: // as a CHAR/VARCHAR in the reconciled schema.
Comment that the name and comment come from the Avro schema.
http://gerrit.cloudera.org:8080/#/c/3331/5/testdata/workloads/functional-query/queries/QueryTest/avro-schema-resolution.test
File testdata/workloads/functional-query/queries/QueryTest/avro-schema-resolution.test:
PS5, Line 80: INT
Can we add a type mismatch as well? e.g. make this float
--
To view, visit http://gerrit.cloudera.org:8080/3331
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia3e43b2885853c2b4f207a45a873c9d7f31379cd
Gerrit-PatchSet: 5
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Dimitris Tsirogiannis <dt...@cloudera.com>
Gerrit-Reviewer: Huaisi Xu <hx...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-CR](cdh5-trunk) IMPALA-3687: Prefer Avro field name during schema reconciliation
Posted by "Huaisi Xu (Code Review)" <ge...@cloudera.org>.
Huaisi Xu has posted comments on this change.
Change subject: IMPALA-3687: Prefer Avro field name during schema reconciliation
......................................................................
Patch Set 10:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/3331/9/fe/src/main/java/com/cloudera/impala/util/AvroSchemaUtils.java
File fe/src/main/java/com/cloudera/impala/util/AvroSchemaUtils.java:
Line 145: Preconditions.checkNotNull(null, "reconciledColDef.Analyze should never throw");
> reconciledColDef.analyze() should never throw
Done. this will make this two lines though..
http://gerrit.cloudera.org:8080/#/c/3331/9/testdata/workloads/functional-query/queries/QueryTest/avro-schema-resolution.test
File testdata/workloads/functional-query/queries/QueryTest/avro-schema-resolution.test:
PS9, Line 78: IMPALA
> use colon, i.e. IMPALA-3687:
Done
Line 128: # IMPALA-3687. Create a table with column definition and later alter its Avro schema to
> use colon, i.e. IMPALA-3687:
Done
PS9, Line 128: column definition
changed this as well.
--
To view, visit http://gerrit.cloudera.org:8080/3331
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia3e43b2885853c2b4f207a45a873c9d7f31379cd
Gerrit-PatchSet: 10
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Dimitris Tsirogiannis <dt...@cloudera.com>
Gerrit-Reviewer: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Juan Yu <jy...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-CR](cdh5-trunk) IMPALA-3687: Use type from avro schema during schema reconcilation.
Posted by "Huaisi Xu (Code Review)" <ge...@cloudera.org>.
Huaisi Xu has uploaded a new patch set (#3).
Change subject: IMPALA-3687: Use type from avro schema during schema reconcilation.
......................................................................
IMPALA-3687: Use type from avro schema during schema reconcilation.
There is no effect that we change the name of a string/char/varchar
column in schema definition because impala uses column defnition
for these types. This patch changes the behavior to that impala
only uses type from column definition. This is to preserve support
for CHAR/VARCHAR column type.
Change-Id: Ia3e43b2885853c2b4f207a45a873c9d7f31379cd
---
M fe/src/main/java/com/cloudera/impala/util/AvroSchemaUtils.java
M testdata/workloads/functional-query/queries/QueryTest/avro-schema-changes.test
2 files changed, 58 insertions(+), 8 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/31/3331/3
--
To view, visit http://gerrit.cloudera.org:8080/3331
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia3e43b2885853c2b4f207a45a873c9d7f31379cd
Gerrit-PatchSet: 3
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Huaisi Xu <hx...@cloudera.com>
[Impala-CR](cdh5-trunk) IMPALA-3687: Prefer Avro field name during schema reconciliation
Posted by "Huaisi Xu (Code Review)" <ge...@cloudera.org>.
Huaisi Xu has posted comments on this change.
Change subject: IMPALA-3687: Prefer Avro field name during schema reconciliation
......................................................................
Patch Set 5:
bump
--
To view, visit http://gerrit.cloudera.org:8080/3331
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia3e43b2885853c2b4f207a45a873c9d7f31379cd
Gerrit-PatchSet: 5
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Huaisi Xu <hx...@cloudera.com>
Gerrit-HasComments: No
[Impala-CR](cdh5-trunk) IMPALA-3687: Prefer Avro field name during schema reconciliation
Posted by "Alex Behm (Code Review)" <ge...@cloudera.org>.
Alex Behm has posted comments on this change.
Change subject: IMPALA-3687: Prefer Avro field name during schema reconciliation
......................................................................
Patch Set 6:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/3331/6//COMMIT_MSG
Commit Message:
Line 9: Currently Impala prefers schema definition from schema definition
How about something like the following blurb.
Since it is possible to create an Avro table with both column definitions and an Avro schema, Impala attempts to reconcile inconsistencies in the two schema definitions, generally preferring the Avro schema. The only exception to this rule was with CHAR/VARCHAR/STRING columns, where the column definition was preferred in order to support tables with CHAR/VARCHAR columns although Avro only supports STRING. This exception is confusing because the name for such a column will be taken from the column definition (and not from the Avro schema).
http://gerrit.cloudera.org:8080/#/c/3331/6/fe/src/main/java/com/cloudera/impala/util/AvroSchemaUtils.java
File fe/src/main/java/com/cloudera/impala/util/AvroSchemaUtils.java:
Line 136: // as a CHAR/VARCHAR/STRING in the reconciled schema. Column name and comment
> another reason is that we did this before so it is hard to drop it. ideally
I agree with Juan, and I think we all agree with her. This reconciliation logic is very subtle and brittle, so it needs to go away. Ideally, we would only accept tables with column definitions xor an Avro schema definition. Allowing two schemas simultaneously is crazy (but Hive, for example, does allow it).
Given how often these kinds of Avro issues pop up, I'm very afraid that rejecting inconsistencies will cause many users' existing tables to break. Further, I think it will require manual intervention at the HMS backend DB level to fix such tables that have become unusable due to our "fix" to reject tables with inconsistent schemas definitions.
Technically, even the current patch breaks compatibility, but it seems far less dangerous than flat out rejecting tables with inconsistencies.
I'm in favor of proceeding with the current solution - but we definitely need to keep the correct fix in mind (not allowing two schemas as all) for a future release.
Line 143: reconciledColDef.analyze();
analysis should never fail, might as well try catch with a Preconditions check in the catch. Then we don't have to throw in this function
http://gerrit.cloudera.org:8080/#/c/3331/6/testdata/workloads/functional-query/queries/QueryTest/avro-schema-resolution.test
File testdata/workloads/functional-query/queries/QueryTest/avro-schema-resolution.test:
Line 95: {"type": ["int", "null"], "name": "id_mismatch", "doc": "int comment mismatch"},
Dimitris' request to include the original test case seems reasonable. We can just alter the table and change the schema literal.
--
To view, visit http://gerrit.cloudera.org:8080/3331
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia3e43b2885853c2b4f207a45a873c9d7f31379cd
Gerrit-PatchSet: 6
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Dimitris Tsirogiannis <dt...@cloudera.com>
Gerrit-Reviewer: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Juan Yu <jy...@cloudera.com>
Gerrit-HasComments: Yes