[jira] [Created] (IMPALA-14646) StorageDescriptor normalization should deal with parameters

["Quanlong Huang (Jira)" <ji...@apache.org>]

Quanlong Huang created IMPALA-14646:
---------------------------------------

             Summary: StorageDescriptor normalization should deal with parameters
                 Key: IMPALA-14646
                 URL: https://issues.apache.org/jira/browse/IMPALA-14646
             Project: IMPALA
          Issue Type: Bug
          Components: Catalog
            Reporter: Quanlong Huang


When checking whether an ALTER_TABLE event has trivial changes in StorageDescriptor, we normalize fields that are unrelated to file metadata, e.g. cols, bucketCols, sortCols, etc. See code in [normalizeStorageDescriptor()|https://github.com/apache/impala/blob/85d77b908b12ae3d3f48ed5d49f38fb3832edc4e/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java#L2334-L2347].

However, the parameters map of StorageDescriptor is not normalized, which causes null and empty map be considered as different:
{code:bash}
I1217 21:53:42.995173 279445 MetastoreEvents.java:825] EventId: 365355029 EventType: ALTER_TABLE Non-trivial change in table storage descriptor (SD) detected for table my_db.my_tbl. So file metadata should be reloaded. SD before: StorageDescriptor(cols:[...], location:hdfs://table_path, inputFormat:org.apache.iceberg.mr.hive.HiveIcebergInputFormat, outputFormat:org.apache.iceberg.mr.hive.HiveIcebergOutputFormat, compressed:false, numBuckets:0, serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.iceberg.mr.hive.HiveIcebergSerDe, parameters:{}), bucketCols:[], sortCols:[], parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[], skewedColValueLocationMaps:{}), storedAsSubDirectories:false), SD after: StorageDescriptor(cols:[...], location:hdfs://table_path, inputFormat:org.apache.iceberg.mr.hive.HiveIcebergInputFormat, outputFormat:org.apache.iceberg.mr.hive.HiveIcebergOutputFormat, compressed:false, numBuckets:0, serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.iceberg.mr.hive.HiveIcebergSerDe, parameters:{}), bucketCols:null, sortCols:null, parameters:null){code}
In the above log, the old SD has an empty parameters map: {}, the new SD has a null parameters map. This should be considered as trivial change.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org