You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Sai Hemanth Gantasala (Jira)" <ji...@apache.org> on 2023/12/21 00:58:00 UTC
[jira] [Assigned] (IMPALA-12637) Failed to process CREATE_TABLE events if the db is unloaded
[ https://issues.apache.org/jira/browse/IMPALA-12637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sai Hemanth Gantasala reassigned IMPALA-12637:
----------------------------------------------
Assignee: Sai Hemanth Gantasala
> Failed to process CREATE_TABLE events if the db is unloaded
> -----------------------------------------------------------
>
> Key: IMPALA-12637
> URL: https://issues.apache.org/jira/browse/IMPALA-12637
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Reporter: Quanlong Huang
> Assignee: Sai Hemanth Gantasala
> Priority: Critical
>
> Applying a CREATE_TABLE event could fail by DatabaseNotFoundException is the db is unloaded. The exception is thrown by CatalogOpExecutor#addTableIfNotRemovedLater():
> [https://github.com/apache/impala/blob/3af193022916e42c33d6eafafb6f9560a0789895/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java#L826-L831]
> In fact, there are cases that the db exists in HMS but is unloaded in catalogd (even if HMS event processing is enabled). For instance, when HMS is restarted during the launching period of catalogd, some dbs might be failed in loading:
> {noformat}
> W1215 08:27:27.811750 26987 CatalogServiceCatalog.java:2014] Encountered an exception while invalidating database: default. Ignoring further load of this db.
> Java exception follows:
> MetaException(message:Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
> at org.apache.thrift.transport.TSocket.open(TSocket.java:255)
> at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:755)
> at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.reconnect(HiveMetaStoreClient.java:448)
> at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient$1.run(RetryingMetaStoreClient.java:191)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
> at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:187)
> at com.sun.proxy.$Proxy11.getFunctions(Unknown Source)
> at org.apache.impala.catalog.CatalogServiceCatalog.invalidateDb(CatalogServiceCatalog.java:1943)
> at org.apache.impala.catalog.CatalogServiceCatalog.reset(CatalogServiceCatalog.java:2103)
> at org.apache.impala.service.JniCatalog.<init>(JniCatalog.java:175)
> Caused by: java.net.ConnectException: Connection refused (Connection refused)
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
> at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
> at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.net.Socket.connect(Socket.java:607)
> at org.apache.thrift.transport.TSocket.open(TSocket.java:250)
> ... 11 more
> )
> at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:826)
> at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.reconnect(HiveMetaStoreClient.java:448)
> at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient$1.run(RetryingMetaStoreClient.java:191)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
> at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:187)
> at com.sun.proxy.$Proxy11.getFunctions(Unknown Source)
> at org.apache.impala.catalog.CatalogServiceCatalog.invalidateDb(CatalogServiceCatalog.java:1943)
> at org.apache.impala.catalog.CatalogServiceCatalog.reset(CatalogServiceCatalog.java:2103)
> at org.apache.impala.service.JniCatalog.<init>(JniCatalog.java:175){noformat}
> I can reproduce the issue by adding a breakpoint in CatalogServiceCatalog#invalidateDb() and attach to the process by IntelliJ during catalogd is launching. Then kill HMS and resume catalogd. After seeing this log, restart HMS and resume catalogd again.
> The event processor will go into the ERROR state when processing any CREATE_TABLE event in that db:
> {noformat}
> E1215 08:29:54.121959 27281 MetastoreEventsProcessor.java:979] Unexpected exception received while processing event
> Java exception follows:
> org.apache.impala.catalog.events.MetastoreNotificationException: EventId: 8411895 EventType: CREATE_TABLE Unable to process event
> at org.apache.impala.catalog.events.MetastoreEvents$CreateTableEvent.process(MetastoreEvents.java:1213)
> at org.apache.impala.catalog.events.MetastoreEvents$MetastoreEvent.processIfEnabled(MetastoreEvents.java:528)
> at org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:1153)
> at org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:961)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:750)
> Caused by: org.apache.impala.catalog.DatabaseNotFoundException: default not found
> at org.apache.impala.service.CatalogOpExecutor.addTableIfNotRemovedLater(CatalogOpExecutor.java:831)
> at org.apache.impala.catalog.events.MetastoreEvents$CreateTableEvent.process(MetastoreEvents.java:1200)
> ... 10 more
> E1215 08:29:54.137053 27281 MetastoreEventsProcessor.java:1237] Event id: 8411895
> Event Type: CREATE_TABLE
> Event time: 1702600193
> Database name: default
> Table name: newtbl
> Event message: H4sIAAAAAAAAAL1V23LaMBD9Fz87xjK5AG+0uNO0DGTAmelMyTACL1itkB1JTkoY/j1rySTYuGn70LxgsUfeyzm7652jQD6AdHqOTiRb6V6rxdMl5UmqdK/rd9qOa66wJdxIJpYsoxwvozVe4DOGFc25xr+aLjigRcCjXvCDIdpmhTH8FoWTUX84j/ofhuEBHC9+fFGpQHw3c8jM6eFDaYmHWelm5uzdmRNUoTKmxdpV7D6ngqdibcFzC7J24YFc+cGl75Nuu4AujiG/sFyeWK6sRcLSHsoUucLYve8l4BL3NHvWmDgTRdL7uwYoiVeqQn3gX/h+S4PSZ49UQpLmClrHpNQKT+XaoxldJuAlNE7TzNvQTELsRfBLX4ss159SuaG6QszvX07YA3j33GOp9xmP12uRSvgK28LbONc1dyWZetXI5RlpIjP4qxSw9WIIPE6ftt4Qf6Zsk3GYghxAhQYs1mpi/LmHp9EGfTDK2RPVLBXeqsy7Z0RDMfaFm05NWvu6b6TqNspuMeK/Gd93d1ZuQv7QS8cBgxpmDiVWr9di1XimIhJURGms8riS7puFXBRERpP+aDrsR+FgHo3nh5k2VEaT29Bc15IKxUDoOadKD2IesQ1Ytl/mz1xc5MufoJlYz3H9KFTGXAoMhk0GUlDumTXhZblcQzVMQ/CS6FpfVTeP7RnSbhbjpHZLrmmi18XyStx5w+Rfj6bhJKqM//EU/MO6qg8WQQlP403DYfjxHePd3gywAd4v3gDr+2/x7l6G5by+wchVbUHjNiq7p3PiNjjsu8vCD+l0SYDei88cNr/SdJMdf35cZ8U4KGyku/0zLOV3nYAHAAA=
> W1215 08:29:54.979072 27281 MetastoreEventsProcessor.java:950] Event processing is skipped since status is ERROR. Last synced event id is 8411894{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org