You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2021/03/03 00:39:00 UTC
[jira] [Commented] (IMPALA-10450) Catalogd crashes in serializing
thrift debug string
[ https://issues.apache.org/jira/browse/IMPALA-10450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17294079#comment-17294079 ]
ASF subversion and git services commented on IMPALA-10450:
----------------------------------------------------------
Commit a47700ed790c2415e52a85e40063bed53a7cb9e8 in impala's branch refs/heads/master from Vihang Karajgaonkar
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=a47700e ]
IMPALA-10450: Catalogd crashes due to exception in ThriftDebugString
This patch adds a wrapper around ThriftDebugString method provided
in the Thrift library. The thrift's method can throw exceptions
like (bad_alloc or TProtocolException) when the object cannot be
serialized into a string representation. This exception is not
caught on the catalogd side and it crashes the catalogd.
The error was specifically seen in the catalogd's debug UI
which provides a way to display a Table object. An exception
thrown when rendering the table on the UI would have crashed
the catalogd before the patch. In order to simulate this crash a new debug
action called EXCEPTION was added. A new custom cluster test
was added which simulates a exception thrown in this method and
makes sure that fetching the table from catalogd's debug UI
does not crash the catalogd.
Tests:
1. Added a new custom cluster test which reproduces the crash.
2. Created a large table which has ~270K partitions and reduced
the memory of the catalogd to 16GB. This configuration throws
bad_alloc exception in the ThriftDebugString method and crashes
the catalogd. After the patch the crash is averted and we see
a error message on the debug UI instead. I also looped around
the catalog web UI call for more than an hour to see if there
are any other stability issues. I could not see any problems.
Change-Id: I42cee6186a3d5bacc1117bae5961ac60ac9f7a66
Reviewed-on: http://gerrit.cloudera.org:8080/17110
Reviewed-by: Vihang Karajgaonkar <vi...@cloudera.com>
Tested-by: Vihang Karajgaonkar <vi...@cloudera.com>
> Catalogd crashes in serializing thrift debug string
> ---------------------------------------------------
>
> Key: IMPALA-10450
> URL: https://issues.apache.org/jira/browse/IMPALA-10450
> Project: IMPALA
> Issue Type: Bug
> Reporter: Quanlong Huang
> Assignee: Vihang Karajgaonkar
> Priority: Major
> Fix For: Impala 4.0
>
>
> A crash of catalogd is reported from a CDH-6.3.0 user:
> {code:java}
> Operating system: Linux
> 0.0.0 Linux 3.10.0-957.21.3.el7.x86_64 #1 SMP Tue Jun 18 16:35:19 UTC 2019 x86_64
> CPU: amd64
> family 6 model 79 stepping 1
> 1 CPU
> GPU: UNKNOWN
> Crash reason: SIGABRT
> Crash address: 0x3d40000db31
> Process uptime: not available
> Thread 161 (crashed)
> 0 libc-2.17.so + 0x36207
> 1 libc-2.17.so + 0x378f8
> 2 libstdc++.so.6!std::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >::basic_string<wchar_t const*>(wchar_t const*, wchar_t const*, std::allocator<wchar_t> const&) + 0x142
> 3 libstdc++.so.6!std::basic_string<wchar_t, std::char_traits<wchar_t>, std::allocator<wchar_t> >::basic_string<wchar_t const*>(wchar_t const*, wchar_t const*, std::allocator<wchar_t> const&) + 0x142
> 4 libstdc++.so.6!__gnu_cxx::__verbose_terminate_handler() + 0x15d
> 5 libstdc++.so.6!std::rethrow_exception(std::__exception_ptr::exception_ptr) + 0x76
> 6 libstdc++.so.6!std::terminate() + 0x11
> 7 libstdc++.so.6!__cxa_throw + 0x69
> 8 impalad!apache::thrift::transport::TMemoryBuffer::ensureCanWrite(unsigned int) + 0x179
> 9 impalad!apache::thrift::transport::TMemoryBuffer::writeSlow(unsigned char const*, unsigned int) + 0x13
> 10 impalad!apache::thrift::protocol::TDebugProtocol::writePlain(std::string const&) + 0x1f
> 11 impalad!apache::thrift::protocol::TDebugProtocol::writeItem(std::string const&) + 0x1c
> 12 impalad!apache::thrift::protocol::TDebugProtocol::writeString(std::string const&) + 0x3c8
> 13 impalad!impala::THdfsFileDesc::write(apache::thrift::protocol::TProtocol*) const + 0x62
> 14 impalad!impala::THdfsPartition::write(apache::thrift::protocol::TProtocol*) const + 0x3a3
> 15 impalad!impala::THdfsTable::write(apache::thrift::protocol::TProtocol*) const + 0x1a1
> 16 impalad!impala::TTable::write(apache::thrift::protocol::TProtocol*) const + 0x421
> 17 impalad!impala::TCatalogObject::write(apache::thrift::protocol::TProtocol*) const + 0x2bc
> 18 impalad!std::string apache::thrift::ThriftDebugString<impala::TCatalogObject>(impala::TCatalogObject const&) + 0x9b
> 19 impalad!impala::CatalogServer::CatalogObjectsUrlCallback(std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator>, rapidjson::CrtAllocator>*) + 0xea8
> 20 impalad!impala::Webserver::RenderUrlWithTemplate(std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&, impala::Webserver::UrlHandler const&, std::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >*, impala::ContentType*) + 0x170
> 21 impalad!impala::Webserver::BeginRequestCallback(sq_connection*, sq_request_info*) + 0x2b4
> 22 impalad!handle_request + 0x740
> 23 impalad!process_new_connection + 0x16d
> 24 impalad!worker_thread + 0x2e5
> 25 libpthread-2.17.so + 0x7dd5
> 26 libc-2.17.so + 0xfdead {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org