[jira] [Commented] (IMPALA-7087) Impala is unable to read Parquet decimal columns with lower precision/scale than table metadata

["ASF subversion and git services (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-7087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17367851#comment-17367851 ] 

ASF subversion and git services commented on IMPALA-7087:
---------------------------------------------------------

Commit c591b846c7956ec8eb0b1479237ac251a9d8ffed in impala's branch refs/heads/master from Zoltan Borok-Nagy
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=c591b84 ]

IMPALA-10166 (part 2): Enable DROP and CHANGE column

IMPALA-10166 (part 1) already added the necessary code for
DROP and CHANGE COLUMN, but disabled those stmts because to correctly
support schema evolution we had to wait for column resolution
by Iceberg field id.

Since then IMPALA-10361 and IMPALA-10485 added support for field-id
based column resolution for Parquet and ORC as well.

Hence this patch enables DROP and CHANGE column ALTER TABLE
statements. We still disallow REPLACE COLUMNS because it doesn't
really make sense for Iceberg tables as it basically makes all
existing data inaccessible.

Changing DECIMAL columns are still disabled due to IMPALA-7087.

Testing:
 * added e2e tests

Change-Id: I9b0d1a55bf0ed718724a69b51392ed53680ffa90
Reviewed-on: http://gerrit.cloudera.org:8080/17593
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Zoltan Borok-Nagy <bo...@cloudera.com>


> Impala is unable to read Parquet decimal columns with lower precision/scale than table metadata
> -----------------------------------------------------------------------------------------------
>
>                 Key: IMPALA-7087
>                 URL: https://issues.apache.org/jira/browse/IMPALA-7087
>             Project: IMPALA
>          Issue Type: Sub-task
>          Components: Backend
>            Reporter: Tim Armstrong
>            Priority: Major
>              Labels: decimal, parquet, ramp-up
>         Attachments: binary_decimal_precision_and_scale_widening.parquet
>
>
> This is similar to IMPALA-2515, except relates to a different precision/scale in the file metadata rather than just a mismatch in the bytes used to store the data. In a lot of cases we should be able to convert the decimal type on the fly to the higher-precision type.
> {noformat}
> ERROR: File '/hdfs/path/000000_0_x_2' column 'alterd_decimal' has an invalid type length. Expecting: 11 len in file: 8
> {noformat}
> It would be convenient to allow reading parquet files where the precision/scale in the file can be converted to the precision/scale in the table metadata without loss of precision.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org