You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2023/04/27 18:28:00 UTC
[jira] [Commented] (IMPALA-12099) Catalogd NPE when refreshing a partitioned table
[ https://issues.apache.org/jira/browse/IMPALA-12099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17717335#comment-17717335 ]
ASF subversion and git services commented on IMPALA-12099:
----------------------------------------------------------
Commit 112bab64b77d6ed966b1c67bd503ed632da6f208 in impala's branch refs/heads/master from wzhou-code
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=112bab64b ]
IMPALA-12099: Catalogd throws NPE when refreshing a partitioned table
Impala Catalogd assumes that if a table is partitioned then all
CompactionInfoStruct entries should not have null partitionname.
This assumption is not true. As according to [1], it is possible
for a compaction queue entry for a partitioned table to have null
partitionname.
This patch handles the null case.
Testing:
- Ran core test.
[1] https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java#L345
Change-Id: If37559f3c74ac517d1403ddb82ca28708015d6bf
Reviewed-on: http://gerrit.cloudera.org:8080/19804
Reviewed-by: Quanlong Huang <hu...@gmail.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
> Catalogd NPE when refreshing a partitioned table
> ------------------------------------------------
>
> Key: IMPALA-12099
> URL: https://issues.apache.org/jira/browse/IMPALA-12099
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Reporter: Wenzhe Zhou
> Assignee: Wenzhe Zhou
> Priority: Major
> Fix For: Impala 4.3.0
>
>
> The following NPE occurs in the catalogd when it tries to refresh certain partitioned tables
> {code:java}
> I0331 13:33:04.345813 256 TAcceptQueueServer.cpp:355] New connection to server CatalogService from client <Host: 127.0.0.6 Port: 50285>
> E0331 13:33:04.372188 31256 JniCatalog.java:424] Error in getting PartialCatalogObject of TABLE:ml_foodforecast.presence_history_archive_ahdp_5584. Time spent: 15ms.
> I0331 13:33:04.372326 31256 jni-util.cc:286] java.lang.NullPointerException
> at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:878)
> at org.apache.impala.compat.MetastoreShim.getPartitionsForRefreshingFileMetadata(MetastoreShim.java:602)
> at org.apache.impala.util.AcidUtils.getPartitionsForRefreshingFileMetadata(AcidUtils.java:778)
> at org.apache.impala.catalog.CatalogServiceCatalog.getOrLoadTable(CatalogServiceCatalog.java:2252)
> at org.apache.impala.catalog.CatalogServiceCatalog.doGetPartialCatalogObject(CatalogServiceCatalog.java:3528)
> at org.apache.impala.catalog.CatalogServiceCatalog.getPartialCatalogObject(CatalogServiceCatalog.java:3454)
> at org.apache.impala.catalog.CatalogServiceCatalog.getPartialCatalogObject(CatalogServiceCatalog.java:3421)
> at org.apache.impala.service.JniCatalog.getPartialCatalogObject(JniCatalog.java:419)
> I0331 13:33:04.372351 31256 status.cc:129] NullPointerException: null
> @ 0xdbeae3
> @ 0x166ade9
> @ 0xd88ec3
> @ 0xd6e013
> @ 0xd44e32
> @ 0xcae4e3
> @ 0xd19ab5
> @ 0x11d1b69
> @ 0x11c51b8
> @ 0x1748f74
> @ 0x174a3ba
> @ 0x1fec2d0
> @ 0x7fad37596ea4
> @ 0x7fad33fd7b0c
> E0331 13:33:04.372615 31256 catalog-server.cc:239] NullPointerException: null
> {code}
> According to [1], Impala assumes that if a table is partitioned then all CompactionInfoStruct entries should not have null partitionname or id. While id is required, partitionname is defined as optional by Hive [2].
> This assumption is false, however. As according to [3], it is possible for a compaction queue entry for a partitioned table to have null partitionname.
> [1]https://github.com/apache/impala/blob/master/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java#L692-L694
> [2]https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift#L1303-L1323
> [3]https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java#L345
> Function checkLatestCompaction() has similar code. It should also handle the null case.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org