[jira] [Created] (IMPALA-12099) Catalogd NPE when refreshing a partitioned table

["Wenzhe Zhou (Jira)" <ji...@apache.org>]

Wenzhe Zhou created IMPALA-12099:
------------------------------------

             Summary: Catalogd NPE when refreshing a partitioned table
                 Key: IMPALA-12099
                 URL: https://issues.apache.org/jira/browse/IMPALA-12099
             Project: IMPALA
          Issue Type: Bug
          Components: Catalog
            Reporter: Wenzhe Zhou


The following NPE occurs in the catalogd when it tries to refresh certain partitioned tables

{code:java}
I0331 13:33:04.345813 256 TAcceptQueueServer.cpp:355] New connection to server CatalogService from client <Host: 127.0.0.6 Port: 50285>
E0331 13:33:04.372188 31256 JniCatalog.java:424] Error in getting PartialCatalogObject of TABLE:ml_foodforecast.presence_history_archive_ahdp_5584. Time spent: 15ms.
I0331 13:33:04.372326 31256 jni-util.cc:286] java.lang.NullPointerException
at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:878)
at org.apache.impala.compat.MetastoreShim.getPartitionsForRefreshingFileMetadata(MetastoreShim.java:602)
at org.apache.impala.util.AcidUtils.getPartitionsForRefreshingFileMetadata(AcidUtils.java:778)
at org.apache.impala.catalog.CatalogServiceCatalog.getOrLoadTable(CatalogServiceCatalog.java:2252)
at org.apache.impala.catalog.CatalogServiceCatalog.doGetPartialCatalogObject(CatalogServiceCatalog.java:3528)
at org.apache.impala.catalog.CatalogServiceCatalog.getPartialCatalogObject(CatalogServiceCatalog.java:3454)
at org.apache.impala.catalog.CatalogServiceCatalog.getPartialCatalogObject(CatalogServiceCatalog.java:3421)
at org.apache.impala.service.JniCatalog.getPartialCatalogObject(JniCatalog.java:419)
I0331 13:33:04.372351 31256 status.cc:129] NullPointerException: null
    @ 0xdbeae3
    @ 0x166ade9
    @ 0xd88ec3
    @ 0xd6e013
    @ 0xd44e32
    @ 0xcae4e3
    @ 0xd19ab5
    @ 0x11d1b69
    @ 0x11c51b8
    @ 0x1748f74
    @ 0x174a3ba
    @ 0x1fec2d0
    @ 0x7fad37596ea4
    @ 0x7fad33fd7b0c
E0331 13:33:04.372615 31256 catalog-server.cc:239] NullPointerException: null
{code}

According to [1], Impala assumes that if a table is partitioned then all CompactionInfoStruct entries should not have null partitionname or id. While id is required, partitionname is defined as optional by Hive [2].

This assumption is false, however. As according to [3], it is possible for a compaction queue entry for a partitioned table to have null partitionname.

[1]https://github.com/apache/impala/blob/master/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java#L692-L694
[2]https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift#L1303-L1323
[3]https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java#L345

Function checkLatestCompaction() has similar code. It should also handle the null case.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org