[jira] [Resolved] (IMPALA-14646) StorageDescriptor normalization should deal with parameters

["Quanlong Huang (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/IMPALA-14646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Quanlong Huang resolved IMPALA-14646.
-------------------------------------
    Fix Version/s: Impala 5.0.0
       Resolution: Fixed

> StorageDescriptor normalization should deal with parameters
> -----------------------------------------------------------
>
>                 Key: IMPALA-14646
>                 URL: https://issues.apache.org/jira/browse/IMPALA-14646
>             Project: IMPALA
>          Issue Type: Bug
>          Components: Catalog
>            Reporter: Quanlong Huang
>            Assignee: Quanlong Huang
>            Priority: Critical
>             Fix For: Impala 5.0.0
>
>
> When checking whether an ALTER_TABLE event has trivial changes in StorageDescriptor, we normalize fields that are unrelated to file metadata, e.g. cols, bucketCols, sortCols, etc. See code in [normalizeStorageDescriptor()|https://github.com/apache/impala/blob/85d77b908b12ae3d3f48ed5d49f38fb3832edc4e/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java#L2334-L2347].
> However, the parameters map of StorageDescriptor is not normalized, which causes null and empty map be considered as different:
> {code:bash}
> I1217 21:53:42.995173 279445 MetastoreEvents.java:825] EventId: 365355029 EventType: ALTER_TABLE Non-trivial change in table storage descriptor (SD) detected for table my_db.my_tbl. So file metadata should be reloaded. SD before: StorageDescriptor(cols:[...], location:hdfs://table_path, inputFormat:org.apache.iceberg.mr.hive.HiveIcebergInputFormat, outputFormat:org.apache.iceberg.mr.hive.HiveIcebergOutputFormat, compressed:false, numBuckets:0, serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.iceberg.mr.hive.HiveIcebergSerDe, parameters:{}), bucketCols:[], sortCols:[], parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[], skewedColValueLocationMaps:{}), storedAsSubDirectories:false), SD after: StorageDescriptor(cols:[...], location:hdfs://table_path, inputFormat:org.apache.iceberg.mr.hive.HiveIcebergInputFormat, outputFormat:org.apache.iceberg.mr.hive.HiveIcebergOutputFormat, compressed:false, numBuckets:0, serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.iceberg.mr.hive.HiveIcebergSerDe, parameters:{}), bucketCols:null, sortCols:null, parameters:null){code}
> In the above log, the old SD has an empty parameters map: {}, the new SD has a null parameters map. This should be considered as trivial change.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)