You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2024/06/04 23:14:00 UTC
[jira] [Commented] (IMPALA-12680) NullPointerException in addHmsPartitions() during MetastoreEventsProcessor switch state from PAUSED to ACTIVE
[ https://issues.apache.org/jira/browse/IMPALA-12680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17852216#comment-17852216 ]
ASF subversion and git services commented on IMPALA-12680:
----------------------------------------------------------
Commit 880690c7bd46e6af058d8a669bea41af4a131049 in impala's branch refs/heads/master from Sai Hemanth Gantasala
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=880690c7b ]
IMPALA-12680: Fix NullPointerException during AlterTableAddPartitions
When global INVALIDATE METADATA is run at the same time while
AlterTableAddPartition statement is being run, a precondition check in
addHmsPartitions() could lead to NullPointerException. This happens
due to Map<String, Long> partitionToEventId being initialized to null
when event processor is not active.
We should always initialize 'partitionToEventId' to empty hash map
regardless of the state of event processor. If the event processor is
not active, then addHmsPartitions() adds partitions that are directly
fetched from metastore.
Note: Also, addressed the same issue that could potentially happen in
AlterTableRecoverPartitions.
Testing:
- Verified manually that NullPointerException scenario is avoided.
- Added a unit test to verify the above use case.
Change-Id: I730fed311ebc09762dccc152d9583d5394b0b9b3
Reviewed-on: http://gerrit.cloudera.org:8080/21430
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
> NullPointerException in addHmsPartitions() during MetastoreEventsProcessor switch state from PAUSED to ACTIVE
> -------------------------------------------------------------------------------------------------------------
>
> Key: IMPALA-12680
> URL: https://issues.apache.org/jira/browse/IMPALA-12680
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Reporter: Quanlong Huang
> Assignee: Sai Hemanth Gantasala
> Priority: Critical
> Attachments: alterTableAddPartitionProfile.txt, catalogd.INFO.gz, impalad.INFO.gz
>
>
> Event processing is paused during a global INVALIDATE METADATA.
> catalog_.isEventProcessingActive() returns false in this state.
>
> If an AlterTableAddPartition statement is running during that time, we could pass in a null value for the 'partitionToEventId' map in here:
> {code:java}
> Map<String, Long> partitionToEventId = catalog_.isEventProcessingActive() ?
> Maps.newHashMap() : null;
> List<Partition> addedHmsPartitions = addHmsPartitionsInTransaction(msClient,
> tbl, allHmsPartitionsToAdd, partitionToEventId, ifNotExists); {code}
> https://github.com/apache/impala/blob/fcda98ad99c13324e3ab09f2e92d331d0304bb8e/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java#L4404-L4407
> If the global INVALIDATE METADATA finishes and event processing is back to the ACTIVE state before the AlterTableAddPartition statement runs into addHmsPartitions(), we will have an non-empty 'partitionToEventSubMap' here:
> {code:java}
> List<NotificationEvent> events = getNextMetastoreEventsIfEnabled(eventId,
> event -> AddPartitionEvent.ADD_PARTITION_EVENT_TYPE
> .equals(event.getEventType())
> && msTbl.getDbName().equalsIgnoreCase(event.getDbName())
> && msTbl.getTableName().equalsIgnoreCase(event.getTableName()));
> Map<Partition, Long> partitionToEventSubMap = Maps.newHashMap();
> getPartitionsFromEvent(events, partitionToEventSubMap);
> // set the eventId to last one which we received so the we fetch the next
> // set of events correctly
> if (!events.isEmpty()) {
> eventId = events.get(events.size() - 1).getEventId();
> }
> if (partitionToEventSubMap.isEmpty()) {
> // if partitions couldn't be fetched from events, use the one returned by
> // add_partitions call above.
> addedHmsPartitions.addAll(addedPartitions);
> } else {
> Preconditions.checkNotNull(partitionToEventId); // <-- This will fail{code}
> https://github.com/apache/impala/blob/fcda98ad99c13324e3ab09f2e92d331d0304bb8e/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java#L5052-L5069
> Then the AlterTableAddPartition statement fails with NullPointerException:
> {code:java}
> I0104 02:46:32.075830 1010 jni-util.cc:302] 4a4eae34f60ba947:b6b2bcfc00000000] java.lang.NullPointerException
> at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:889)
> at org.apache.impala.service.CatalogOpExecutor.addHmsPartitions(CatalogOpExecutor.java:5051)
> at org.apache.impala.service.CatalogOpExecutor.addHmsPartitionsInTransaction(CatalogOpExecutor.java:5082)
> at org.apache.impala.service.CatalogOpExecutor.alterTableAddPartitions(CatalogOpExecutor.java:4388)
> at org.apache.impala.service.CatalogOpExecutor.alterTable(CatalogOpExecutor.java:1136)
> at org.apache.impala.service.CatalogOpExecutor.execDdlRequest(CatalogOpExecutor.java:450)
> at org.apache.impala.service.JniCatalog.lambda$execDdl$3(JniCatalog.java:304)
> at org.apache.impala.service.JniCatalogOp.lambda$execAndSerialize$1(JniCatalogOp.java:90)
> at org.apache.impala.service.JniCatalogOp.execOp(JniCatalogOp.java:58)
> at org.apache.impala.service.JniCatalogOp.execAndSerialize(JniCatalogOp.java:89)
> at org.apache.impala.service.JniCatalogOp.execAndSerialize(JniCatalogOp.java:100)
> at org.apache.impala.service.JniCatalog.execAndSerialize(JniCatalog.java:233)
> at org.apache.impala.service.JniCatalog.execAndSerialize(JniCatalog.java:247)
> at org.apache.impala.service.JniCatalog.execDdl(JniCatalog.java:303){code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org