[jira] [Commented] (IMPALA-11210) Impala can only handle lowercase schema elements of Iceberg table

["ASF subversion and git services (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-11210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17515605#comment-17515605 ] 

ASF subversion and git services commented on IMPALA-11210:
----------------------------------------------------------

Commit 952f2af0ca5afd48bac828b66db467502da76ff2 in impala's branch refs/heads/master from Zoltan Borok-Nagy
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=952f2af ]

IMPALA-11210: Impala can only handle lowercase schema elements of Iceberg table

When Impala/Hive creates a table they lowercase the schema elements.
When Spark creates an Iceberg table it doesn't lowercase the names
of the columns in the Iceberg metadata. This triggers a precondition
check in Impala which makes such Iceberg tables unloadable.

This patch converts column names to lowercase when converting Iceberg
schemas to Hive/Impala schemas.

Testing:
 * added e2e test

Change-Id: Iffd910f76844fbf34db805dda6c3053c5ad1cf79
Reviewed-on: http://gerrit.cloudera.org:8080/18368
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>


> Impala can only handle lowercase schema elements of Iceberg table
> -----------------------------------------------------------------
>
>                 Key: IMPALA-11210
>                 URL: https://issues.apache.org/jira/browse/IMPALA-11210
>             Project: IMPALA
>          Issue Type: Bug
>          Components: Catalog
>            Reporter: Zoltán Borók-Nagy
>            Priority: Major
>              Labels: impala-iceberg
>
> When Impala/Hive creates a table they lowercase the schema elements.
> Impala even has precondition checks that the schema elements are lower case:
> [https://github.com/apache/impala/blob/b1c1be12f3cceed48e93eddae8b9512737e3e0d2/fe/src/main/java/org/apache/impala/catalog/Column.java#L56]
> But when an Iceberg table is created via Spark it doesn't lowercase the column names, so the metadata.json file looks like this:
> {noformat}
> ...
>   "schema" : {
>     "type" : "struct",
>     "schema-id" : 0,
>     "fields" : [ {
>       "id" : 1,
>       "name" : "Region",
>       "required" : false,
>       "type" : "string"
>     }, {
> ...
> {noformat}
> Hence when Impala tries to load such tables it runs to the above precondition check which raises an IllegalStateException.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org