[jira] [Created] (IMPALA-6436) Impala Catalog generates a core file / mini dump when the HMS is not available

["Luis E Martinez-Poblete (JIRA)" <ji...@apache.org>]

Luis E Martinez-Poblete created IMPALA-6436:
-----------------------------------------------

             Summary: Impala Catalog generates a core file / mini dump when the HMS is not available
                 Key: IMPALA-6436
                 URL: https://issues.apache.org/jira/browse/IMPALA-6436
             Project: IMPALA
          Issue Type: Bug
          Components: Catalog
    Affects Versions: Impala 2.10.0
            Reporter: Luis E Martinez-Poblete


Synopsis:
=========
Impala Catalog generates a core file / mini dump when the HMS is not available

Problem:
========
Customer reported that the Catalog server created multiple Catalog core files. During the investigation it was determine that the cause of the core files was because the Hive Meta Store was not available and the option "Enable Core Dump" was enabled when starting the Impala service.

Below is the back trace of the core file:

#0 0x00007f72e93ee5d7 in raise () from /root/191729/slib/lib64/libc.so.6
#1 0x00007f72e93efcc8 in abort () from /root/191729/slib/lib64/libc.so.6
#2 0x0000000001ba5754 in google::DumpStackTraceAndExit() ()
#3 0x0000000001b9c1cd in google::LogMessage::Fail() ()
#4 0x0000000001b9da72 in google::LogMessage::SendToLog() ()
#5 0x0000000001b9bba7 in google::LogMessage::Flush() ()
#6 0x0000000001b9f16e in google::LogMessageFatal::~LogMessageFatal() ()
#7 0x000000000083067e in impala::Catalog::(GetCatalogVersion (this=0x0, version=0x7ffc2aa6b750) at /usr/src/debug/impala-2.10.0-cdh5.13.1/be/src/catalog/catalog.cc:88
#8 0x00000000008143c9 in impala::CatalogServer::Start() () at /usr/src/debug/impala-2.10.0-cdh5.13.1/be/src/catalog/catalog-server.cc:175

The corresponding entries in the Catalog server log show the following fatal error:

F0111 09:48:05.017491 14571 catalog.cc:76] java.lang.IllegalStateException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
at org.apache.impala.catalog.MetaStoreClientPool$MetaStoreClient.<init>(MetaStoreClientPool.java:99)
at org.apache.impala.catalog.MetaStoreClientPool$MetaStoreClient.<init>(MetaStoreClientPool.java:72)
at org.apache.impala.catalog.MetaStoreClientPool.initClients(MetaStoreClientPool.java:168)
at org.apache.impala.catalog.Catalog.<init>(Catalog.java:103)
at org.apache.impala.catalog.CatalogServiceCatalog.<init>(CatalogServiceCatalog.java:163)
at org.apache.impala.service.JniCatalog.<init>(JniCatalog.java:104)
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient

I was able to reproduce this issue. When the option "Enable Core Dump" is enable and the Hive Meta Store is not available, the system generates a core file. If the option "Enable Core Dump" is disabled, the system generates a mini dump.

Crashing due to an error is not expected. Impala should fail in a more user friendly way.

Reproduction case:
==================
1) Enable the option "Enable Core Dump" for the Impala service in CM.
2) Stop Hive and Impala services.
3) Start Impala Catalog server



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)