You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (JIRA)" <ji...@apache.org> on 2018/11/05 21:58:00 UTC
[jira] [Commented] (IMPALA-6436) Impala Catalog generates a core
file / mini dump when the HMS is not available
[ https://issues.apache.org/jira/browse/IMPALA-6436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16675801#comment-16675801 ]
ASF subversion and git services commented on IMPALA-6436:
---------------------------------------------------------
Commit f08642bf43102cf326f4f00e9b9e3536d6906b2c in impala's branch refs/heads/master from [~tarmstrong@cloudera.com]
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=f08642b ]
IMPALA-6436: exit instead of abort for catalog startup failure
Rename EXIT_WITH_EXC to ABORT_WITH_EXC to make the behaviour more
obvious at callsites.
Handle exceptions from Catalog constructor by logging the backtrace and
exiting cleanly, rather than aborting. This will prevent generation of a
coredump or minidump.
Testing:
Tested starting the catalogd locally without the HMS running and a
low connection timeout:
start-impala-cluster.py --catalogd_args=--initial_hms_cnxn_timeout_s=2
Confirmed that the backtrace was logged to catalogd.ERROR and that no
core or minidump was generated.
Change-Id: I4026dccb39843b847426112fc0fe9ba897e48dcc
Reviewed-on: http://gerrit.cloudera.org:8080/11871
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
> Impala Catalog generates a core file / mini dump when the HMS is not available
> ------------------------------------------------------------------------------
>
> Key: IMPALA-6436
> URL: https://issues.apache.org/jira/browse/IMPALA-6436
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Affects Versions: Impala 2.10.0
> Reporter: Luis E Martinez-Poblete
> Assignee: Tim Armstrong
> Priority: Critical
> Labels: supportability
>
> Synopsis:
> =========
> Impala Catalog generates a core file / mini dump when the HMS is not available
> Problem:
> ========
> Catalog server created multiple Catalog core files. During the investigation it was determine that the cause of the core files was because the Hive Meta Store was not available and the option "Enable Core Dump" was enabled when starting the Impala service.
> Below is the back trace of the core file:
> #0 0x00007f72e93ee5d7 in raise () from /root/191729/slib/lib64/libc.so.6
> #1 0x00007f72e93efcc8 in abort () from /root/191729/slib/lib64/libc.so.6
> #2 0x0000000001ba5754 in google::DumpStackTraceAndExit() ()
> #3 0x0000000001b9c1cd in google::LogMessage::Fail() ()
> #4 0x0000000001b9da72 in google::LogMessage::SendToLog() ()
> #5 0x0000000001b9bba7 in google::LogMessage::Flush() ()
> #6 0x0000000001b9f16e in google::LogMessageFatal::~LogMessageFatal() ()
> #7 0x000000000083067e in impala::Catalog::(GetCatalogVersion (this=0x0, version=0x7ffc2aa6b750) at /usr/src/debug/impala-2.10.0-cdh5.13.1/be/src/catalog/catalog.cc:88
> #8 0x00000000008143c9 in impala::CatalogServer::Start() () at /usr/src/debug/impala-2.10.0-cdh5.13.1/be/src/catalog/catalog-server.cc:175
> The corresponding entries in the Catalog server log show the following fatal error:
> F0111 09:48:05.017491 14571 catalog.cc:76] java.lang.IllegalStateException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
> at org.apache.impala.catalog.MetaStoreClientPool$MetaStoreClient.<init>(MetaStoreClientPool.java:99)
> at org.apache.impala.catalog.MetaStoreClientPool$MetaStoreClient.<init>(MetaStoreClientPool.java:72)
> at org.apache.impala.catalog.MetaStoreClientPool.initClients(MetaStoreClientPool.java:168)
> at org.apache.impala.catalog.Catalog.<init>(Catalog.java:103)
> at org.apache.impala.catalog.CatalogServiceCatalog.<init>(CatalogServiceCatalog.java:163)
> at org.apache.impala.service.JniCatalog.<init>(JniCatalog.java:104)
> Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
> I was able to reproduce this issue. When the option "Enable Core Dump" is enable and the Hive Meta Store is not available, the system generates a core file. If the option "Enable Core Dump" is disabled, the system generates a mini dump.
> Crashing due to an error is not expected. Impala should fail in a more user friendly way.
> Reproduction case:
> ==================
> 1) Enable the option "Enable Core Dump" for the Impala service in CM.
> 2) Stop Hive and Impala services.
> 3) Start Impala Catalog server
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org