[jira] [Created] (IMPALA-12213) Not a self-event because service id of this catalog does not match with one in event

["Ezra Zerihun (Jira)" <ji...@apache.org>]

Ezra Zerihun created IMPALA-12213:
-------------------------------------

             Summary: Not a self-event because service id of this catalog does not match with one in event
                 Key: IMPALA-12213
                 URL: https://issues.apache.org/jira/browse/IMPALA-12213
             Project: IMPALA
          Issue Type: Bug
          Components: Catalog
    Affects Versions: Impala 4.0.0
            Reporter: Ezra Zerihun


Customer started a few Impala Virtual Warehouses that all connected to the same Database Catalog in DW. After it finished starting, they noticed tables seen in their Hive VW were missing from their Impala VW. Looking in Catalog pod logs we Event Processor was stuck in Error state:
{quote}W0528 07:11:42.398236   228 MetastoreEventsProcessor.java:844] Event processing is skipped since status is ERROR. Last synced event id is 33348458{quote}
Running "invalidate metadata \{table}" or global invalidate metadata shows the tables.

 

Looking at old Catalog pod logs when the error first appeared shows this unique stacktrace that led to the ERROR state
{quote}_____________________
I0528 07:11:40.397091   228 MetastoreEvents.java:638] EventId: 33348458 EventType: OTHER Ignoring unknown event type OPEN_TXN
I0528 07:11:40.397158   228 CatalogServiceCatalog.java:1022] Not a self-event because service id of this catalog c1aad0ff73c54b79:96249bcba6589cb8 does not match with one in event 751641d1aed04da4:8929d897001acc0f.
I0528 07:11:40.397374   228 MetastoreEventsProcessor.java:1026] Time elapsed in processing event batch: 1.015ms E0528 07:11:40.397637   228 MetastoreEventsProcessor.java:865] Unexpected exception received while processing event
Java exception follows:
java.lang.NullPointerException
        at org.apache.impala.catalog.events.MetastoreEvents$ReloadEvent.isOlderEvent(MetastoreEvents.java:2526)
        at org.apache.impala.catalog.events.MetastoreEvents$ReloadEvent.process(MetastoreEvents.java:2501)
        at org.apache.impala.catalog.events.MetastoreEvents$MetastoreEvent.processIfEnabled(MetastoreEvents.java:521)
        at org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:1013)
        at org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:851)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
E0528 07:11:40.397707   228 MetastoreEventsProcessor.java:1040] Notification event is null
I0528 07:11:41.177731   341 catalog-server.cc:411] A catalog update with 2 entries is assembled. Catalog version: 113555 Last sent catalog version: 113554
I0528 07:11:41.177889   295 JniUtil.java:162] getCatalogDelta request: Getting catalog delta from version 113555
I0528 07:11:41.187924   295 catalog-server.cc:824] Collected update: 2:TABLE:ddmlcnp1_gbrs_snp.concept_timeline, version=113559, original size=70, compressed size=72
I0528 07:11:41.233108 20814 JniUtil.java:162] getPartialCatalogObject request: Getting partial catalog object of DATABASE:rsa_edocuvp1
I0528 07:11:41.234481 20814 JniUtil.java:169] Finished getPartialCatalogObject request: Getting partial catalog object of DATABASE:rsa_edocuvp1. Time spent: 1ms
I0528 07:11:41.298694   295 catalog-server.cc:824] Collected update: 2:TABLE:rsa_edocuvp1.docubridge_document, version=113558, original size=68, compressed size=68
I0528 07:11:41.299316   295 catalog-server.cc:824] Collected update: 2:TABLE:rsa_edocuvp1.docubridge_db_settings, version=113557, original size=71, compressed size=71
I0528 07:11:41.324290   295 catalog-server.cc:824] Collected update: 2:CATALOG_SERVICE_ID, version=113559, original size=60, compressed size=59
I0528 07:11:41.324410   295 JniUtil.java:169] Finished getCatalogDelta request: Getting catalog delta from version 113555. Time spent: 147ms
W0528 07:11:41.397955   228 MetastoreEventsProcessor.java:844] Event processing is skipped since status is ERROR. Last synced event id is 33348458
W0528 07:11:42.398236   228 MetastoreEventsProcessor.java:844] Event processing is skipped since status is ERROR. Last synced event id is 33348458
_____________________{quote}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org