You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Quanlong Huang (Jira)" <ji...@apache.org> on 2025/03/07 12:04:00 UTC
[jira] [Created] (HIVE-28808) DB-Notification-Cleaner thread dies in startup due to OOM
Quanlong Huang created HIVE-28808:
-------------------------------------
Summary: DB-Notification-Cleaner thread dies in startup due to OOM
Key: HIVE-28808
URL: https://issues.apache.org/jira/browse/HIVE-28808
Project: Hive
Issue Type: Bug
Reporter: Quanlong Huang
Saw this when launching HMS on a huge NOTIFICATION_LOG table.
{noformat}
2025-03-07 19:37:18: Starting Hive Metastore Server
Listening for transport dt_socket at address: 30010
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/quanlong/workspace/Impala/toolchain/cdp_components-58457853/apache-hive-3.1.3000.7.3.1.0-160-bin/lib/log4j-slf4j-impl-2.18.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/quanlong/workspace/Impala/toolchain/cdp_components-58457853/hadoop-3.1.1.7.3.1.0-160/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Exception in thread "DB-Notification-Cleaner" java.lang.OutOfMemoryError: Java heap space
at java.lang.StringCoding.decode(StringCoding.java:215)
at java.lang.String.<init>(String.java:463)
at org.postgresql.core.Encoding.decode(Encoding.java:284)
at org.postgresql.core.Encoding.decode(Encoding.java:295)
at org.postgresql.jdbc.PgResultSet.getString(PgResultSet.java:2256)
at com.zaxxer.hikari.pool.HikariProxyResultSet.getString(HikariProxyResultSet.java)
at org.datanucleus.store.rdbms.mapping.column.LongVarcharColumnMapping.getString(LongVarcharColumnMapping.java:102)
at org.datanucleus.store.rdbms.mapping.java.SingleFieldMapping.getString(SingleFieldMapping.java:188)
at org.datanucleus.store.rdbms.fieldmanager.ResultSetGetter.fetchStringField(ResultSetGetter.java:133)
at org.datanucleus.state.StateManagerImpl.replacingStringField(StateManagerImpl.java:1986)
at org.apache.hadoop.hive.metastore.model.MNotificationLog.dnReplaceField(MNotificationLog.java)
at org.apache.hadoop.hive.metastore.model.MNotificationLog.dnReplaceFields(MNotificationLog.java)
at org.datanucleus.state.StateManagerImpl.replaceFields(StateManagerImpl.java:4352)
at org.datanucleus.store.rdbms.query.PersistentClassROF$1.fetchFields(PersistentClassROF.java:528)
at org.datanucleus.state.StateManagerImpl.loadFieldValues(StateManagerImpl.java:3743)
at org.datanucleus.state.StateManagerImpl.initialiseForHollow(StateManagerImpl.java:383)
at org.datanucleus.state.ObjectProviderFactoryImpl.newForHollow(ObjectProviderFactoryImpl.java:99)
at org.datanucleus.ExecutionContextImpl.findObject(ExecutionContextImpl.java:3199)
at org.datanucleus.store.rdbms.query.PersistentClassROF.findObjectWithIdAndLoadFields(PersistentClassROF.java:523)
at org.datanucleus.store.rdbms.query.PersistentClassROF.getObject(PersistentClassROF.java:456)
at org.datanucleus.store.rdbms.query.ForwardQueryResult.nextResultSetElement(ForwardQueryResult.java:181)
at org.datanucleus.store.rdbms.query.ForwardQueryResult$QueryResultIterator.next(ForwardQueryResult.java:409)
at org.datanucleus.store.rdbms.query.ForwardQueryResult.processNumberOfResults(ForwardQueryResult.java:137)
at org.datanucleus.store.rdbms.query.ForwardQueryResult.advanceToEndOfResultSet(ForwardQueryResult.java:165)
at org.datanucleus.store.rdbms.query.ForwardQueryResult.getSizeUsingMethod(ForwardQueryResult.java:519)
at org.datanucleus.store.query.AbstractQueryResult.size(AbstractQueryResult.java:256)
at org.apache.hadoop.hive.metastore.ObjectStore.doCleanNotificationEvents(ObjectStore.java:12213)
at org.apache.hadoop.hive.metastore.ObjectStore.cleanOlderEvents(ObjectStore.java:12175)
at org.apache.hadoop.hive.metastore.ObjectStore.cleanNotificationEvents(ObjectStore.java:12161)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) {noformat}
This is a downstream build. The related hive code is
https://github.com/apache/hive/blob/d0372808177a823d63383e311c5909aa46b9a961/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java#L11577
It seems we should optimize the code of cleaning old events.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)