You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Sai Hemanth Gantasala (Jira)" <ji...@apache.org> on 2024/11/06 17:39:00 UTC
[jira] [Updated] (IMPALA-12277) metadata reload of INSERT failed by NullPointerException: Invalid partition name
[ https://issues.apache.org/jira/browse/IMPALA-12277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sai Hemanth Gantasala updated IMPALA-12277:
-------------------------------------------
Epic Link: IMPALA-11533
> metadata reload of INSERT failed by NullPointerException: Invalid partition name
> --------------------------------------------------------------------------------
>
> Key: IMPALA-12277
> URL: https://issues.apache.org/jira/browse/IMPALA-12277
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Reporter: Quanlong Huang
> Assignee: Sai Hemanth Gantasala
> Priority: Major
> Fix For: Impala 4.5.0
>
>
> INSERT into a partition that exists in catalogd but doesn't exist in HMS will fail in metadata reloading on the partition. The cause is that updateCatalog doesn't create the partition in HMS (since catalogd is not aware of the non-existence of the partition in HMS):
> [https://github.com/apache/impala/blob/d0fe4c604f72d41019832513ebf65cfe8f469953/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java#L6697-L6699]
> When reloading the partition, catalogd first removes it since it doesn't exist in HMS:
> [https://github.com/apache/impala/blob/d0fe4c604f72d41019832513ebf65cfe8f469953/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java#L1530-L1531]
> It then try to reload it, which hits NullPointerException at:
> [https://github.com/apache/impala/blob/d0fe4c604f72d41019832513ebf65cfe8f469953/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java#L1566]
> To reproduce the issue, launch Impala with event processing disabled so catalogd can be unsynced with HMS. Create a partitioned table in Impala with one partition:
> {noformat}
> bin/start-impala-cluster.py --catalogd_args=--hms_event_polling_interval_s=0
> impala> create table my_part2 (id int) partitioned by (p int) stored as textfile;
> impala> insert into my_part2 partition(p=0) values (0); {noformat}
> Drop the partition in Hive:
> {code:sql}
> hive> alter table my_part2 drop partition (p=0);{code}
> Then insert the partition again in Impala
> {code:sql}
> impala> insert into my_part2 partition(p=0) values (1);
> ERROR: TableLoadingException: Failed to load metadata for table: default.my_part2
> CAUSED BY: NullPointerException: Invalid partition name: p=0
> {code}
> The exception:
> {noformat}
> E0710 19:34:43.569339 4413 JniUtil.java:183] bb4452d18eafe116:eaf16c4000000000] Error in Update catalog for default.my_part2. Time spent: 1s186ms
> I0710 19:34:43.569918 4413 jni-util.cc:288] bb4452d18eafe116:eaf16c4000000000] org.apache.impala.catalog.TableLoadingException: Failed to load metadata for table: default.my_part2
> at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1308)
> at org.apache.impala.service.CatalogOpExecutor.loadTableMetadata(CatalogOpExecutor.java:1521)
> at org.apache.impala.service.CatalogOpExecutor.updateCatalog(CatalogOpExecutor.java:6863)
> at org.apache.impala.service.JniCatalog.lambda$updateCatalog$16(JniCatalog.java:471)
> at org.apache.impala.service.JniCatalogOp.lambda$execAndSerialize$1(JniCatalogOp.java:90)
> at org.apache.impala.service.JniCatalogOp.execOp(JniCatalogOp.java:58)
> at org.apache.impala.service.JniCatalogOp.execAndSerialize(JniCatalogOp.java:89)
> at org.apache.impala.service.JniCatalogOp.execAndSerialize(JniCatalogOp.java:100)
> at org.apache.impala.service.JniCatalog.execAndSerialize(JniCatalog.java:230)
> at org.apache.impala.service.JniCatalog.updateCatalog(JniCatalog.java:470)
> Caused by: java.lang.NullPointerException: Invalid partition name: p=0
> at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:907)
> at org.apache.impala.catalog.HdfsTable.getPartitionsForNames(HdfsTable.java:1766)
> at org.apache.impala.catalog.HdfsTable$PartitionDeltaUpdater.apply(HdfsTable.java:1566)
> at org.apache.impala.catalog.HdfsTable.updatePartitionsFromHms(HdfsTable.java:1447)
> at org.apache.impala.catalog.HdfsTable.load(HdfsTable.java:1282)
> ... 9 more{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org