You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@impala.apache.org by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org> on 2016/06/22 16:29:11 UTC
[Impala-CR](cdh5-2.6.0 5.8.0) CDH-41511: Fix the preconditions check in HdfsTable#setAvroSchema()
Bharath Vissapragada has uploaded a new change for review.
http://gerrit.cloudera.org:8080/3446
Change subject: CDH-41511: Fix the preconditions check in HdfsTable#setAvroSchema()
......................................................................
CDH-41511: Fix the preconditions check in HdfsTable#setAvroSchema()
In HdfsTable#setAvroSchema(), we rely on an incorrect precondition
check(!nonPartFieldSchemas_.isEmpty()) to make sure loadSchema() is
called. However nonPartFieldSchemas_ can still be empty for Avro
tables created using older versions of Hive. We take care of this
in the schema reconciliation step where we read the Avro schema to
populate the column list. This incorrect check prevents the Catalog
from reaching the schema reconciliation code path.
Fix: Added a new flag isSchemaLoaded_ and setAvroSchema() uses it
to check if the schema has been loaded properly.
Testing: I couldn't reproduce the issue directly by creating Avro
tables without any columns. So, I manually deleted the entries from
"COLUMNS_V2" table in the hms db to make sure to hit the incorrect
preconditions check. Applying this patch fixed it and Impala loaded
the table correctly and I can query it now.
Change-Id: I02b6852385d4f4b470f763308817991e60557060
---
M fe/src/main/java/com/cloudera/impala/catalog/HdfsTable.java
1 file changed, 6 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/46/3446/1
--
To view, visit http://gerrit.cloudera.org:8080/3446
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I02b6852385d4f4b470f763308817991e60557060
Gerrit-PatchSet: 1
Gerrit-Project: Impala
Gerrit-Branch: cdh5-2.6.0_5.8.0
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
[Impala-CR](cdh5-2.6.0 5.8.0) CDH-41511: Fix the preconditions check in HdfsTable#setAvroSchema()
Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.
Bharath Vissapragada has posted comments on this change.
Change subject: CDH-41511: Fix the preconditions check in HdfsTable#setAvroSchema()
......................................................................
Patch Set 1: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/3446
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I02b6852385d4f4b470f763308817991e60557060
Gerrit-PatchSet: 1
Gerrit-Project: Impala
Gerrit-Branch: cdh5-2.6.0_5.8.0
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Marcel Kornacker <ma...@cloudera.com>
Gerrit-HasComments: No
[Impala-CR](cdh5-2.6.0 5.8.0) CDH-41511: Fix the preconditions check in HdfsTable#setAvroSchema()
Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.
Bharath Vissapragada has posted comments on this change.
Change subject: CDH-41511: Fix the preconditions check in HdfsTable#setAvroSchema()
......................................................................
Patch Set 2:
Core tests passed: http://sandbox.jenkins.cloudera.com/job/impala-private-build-and-test/3481/
--
To view, visit http://gerrit.cloudera.org:8080/3446
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I02b6852385d4f4b470f763308817991e60557060
Gerrit-PatchSet: 2
Gerrit-Project: Impala
Gerrit-Branch: cdh5-2.6.0_5.8.0
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Marcel Kornacker <ma...@cloudera.com>
Gerrit-HasComments: No
[Impala-CR](cdh5-2.6.0 5.8.0) CDH-41511: Fix the preconditions check in HdfsTable#setAvroSchema()
Posted by "Marcel Kornacker (Code Review)" <ge...@cloudera.org>.
Marcel Kornacker has posted comments on this change.
Change subject: CDH-41511: Fix the preconditions check in HdfsTable#setAvroSchema()
......................................................................
Patch Set 1: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/3446
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I02b6852385d4f4b470f763308817991e60557060
Gerrit-PatchSet: 1
Gerrit-Project: Impala
Gerrit-Branch: cdh5-2.6.0_5.8.0
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Marcel Kornacker <ma...@cloudera.com>
Gerrit-HasComments: No
[Impala-CR](cdh5-2.6.0 5.8.0) CDH-41511: Fix the preconditions check in HdfsTable#setAvroSchema()
Posted by "Dan Hecht (Code Review)" <ge...@cloudera.org>.
Dan Hecht has posted comments on this change.
Change subject: CDH-41511: Fix the preconditions check in HdfsTable#setAvroSchema()
......................................................................
Patch Set 1: Code-Review+2
Gatekeeper approval for 2.6.
--
To view, visit http://gerrit.cloudera.org:8080/3446
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I02b6852385d4f4b470f763308817991e60557060
Gerrit-PatchSet: 1
Gerrit-Project: Impala
Gerrit-Branch: cdh5-2.6.0_5.8.0
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Marcel Kornacker <ma...@cloudera.com>
Gerrit-HasComments: No
[Impala-CR](cdh5-2.6.0 5.8.0) CDH-41511: Fix the preconditions check in HdfsTable#setAvroSchema()
Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.
Bharath Vissapragada has submitted this change and it was merged.
Change subject: CDH-41511: Fix the preconditions check in HdfsTable#setAvroSchema()
......................................................................
CDH-41511: Fix the preconditions check in HdfsTable#setAvroSchema()
In HdfsTable#setAvroSchema(), we rely on an incorrect precondition
check(!nonPartFieldSchemas_.isEmpty()) to make sure loadSchema() is
called. However nonPartFieldSchemas_ can still be empty for Avro
tables created using older versions of Hive. We take care of this
in the schema reconciliation step where we read the Avro schema to
populate the column list. This incorrect check prevents the Catalog
from reaching the schema reconciliation code path.
Fix: Added a new flag isSchemaLoaded_ and setAvroSchema() uses it
to check if the schema has been loaded properly.
Testing: I couldn't reproduce the issue directly by creating Avro
tables without any columns. So, I manually deleted the entries from
"COLUMNS_V2" table in the hms db to make sure to hit the incorrect
preconditions check. Applying this patch fixed it and Impala loaded
the table correctly and I can query it now.
Change-Id: I02b6852385d4f4b470f763308817991e60557060
Reviewed-on: http://gerrit.cloudera.org:8080/3446
Reviewed-by: Bharath Vissapragada <bh...@cloudera.com>
Reviewed-by: Marcel Kornacker <ma...@cloudera.com>
Reviewed-by: Dan Hecht <dh...@cloudera.com>
Tested-by: Bharath Vissapragada <bh...@cloudera.com>
---
M fe/src/main/java/com/cloudera/impala/catalog/HdfsTable.java
1 file changed, 6 insertions(+), 1 deletion(-)
Approvals:
Marcel Kornacker: Looks good to me, approved
Bharath Vissapragada: Looks good to me, but someone else must approve; Verified
Dan Hecht: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/3446
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I02b6852385d4f4b470f763308817991e60557060
Gerrit-PatchSet: 2
Gerrit-Project: Impala
Gerrit-Branch: cdh5-2.6.0_5.8.0
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Marcel Kornacker <ma...@cloudera.com>
[Impala-CR](cdh5-2.6.0 5.8.0) CDH-41511: Fix the preconditions check in HdfsTable#setAvroSchema()
Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.
Bharath Vissapragada has posted comments on this change.
Change subject: CDH-41511: Fix the preconditions check in HdfsTable#setAvroSchema()
......................................................................
Patch Set 1: Code-Review+1
Carrying +1 from Dan.
--
To view, visit http://gerrit.cloudera.org:8080/3446
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I02b6852385d4f4b470f763308817991e60557060
Gerrit-PatchSet: 1
Gerrit-Project: Impala
Gerrit-Branch: cdh5-2.6.0_5.8.0
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-HasComments: No