You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@impala.apache.org by "Tianyi Wang (JIRA)" <ji...@apache.org> on 2017/11/06 18:56:00 UTC
[jira] [Resolved] (IMPALA-6118) Assertion failure in mem-tracker
when releasing runtime filter memory
[ https://issues.apache.org/jira/browse/IMPALA-6118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tianyi Wang resolved IMPALA-6118.
---------------------------------
Resolution: Fixed
Fix Version/s: Impala 2.11.0
IMPALA-6118: Fix assertion failure in coordinator bloom filter updating
Coordinator::UpdateFilter wrongly assumed std::string::capacity() to be
0 after std::string::clear(), resulting in an DCHECK failure in memory
tracking. This patch tracks size() instead of capacity() to fix the bug.
Change-Id: Ia28f9eb8d0b3e795d798949a1cfcfeaaec7040ed
Reviewed-on: http://gerrit.cloudera.org:8080/8410
Reviewed-by: Tim Armstrong <ta...@cloudera.com>
Reviewed-by: Alex Behm <al...@cloudera.com>
Tested-by: Impala Public Jenkins
> Assertion failure in mem-tracker when releasing runtime filter memory
> ---------------------------------------------------------------------
>
> Key: IMPALA-6118
> URL: https://issues.apache.org/jira/browse/IMPALA-6118
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 2.11.0
> Reporter: Philip Zeyliger
> Assignee: Tianyi Wang
> Priority: Blocker
> Labels: crash
> Fix For: Impala 2.11.0
>
> Attachments: impalad.ip-172-31-13-161.ubuntu.log.ERROR.20171025-012412.38084.gz
>
>
> In a build on jenkins.impala.io, I hit an assertion in mem-tracker.h. The log file is https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/506/artifact/Impala/logs_static/logs/ee_tests/impalad.FATAL/*view*/ and https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/506/artifact/Impala/logs_static/logs/ee_tests/impalad.ip-172-31-13-161.ubuntu.log.ERROR.20171025-012412.38084/*view*/ . The ERROR log is attached to this ticket.
> The stack trace was:
> {code}
> F1025 02:20:43.786911 82485 mem-tracker.h:231] Check failed: tracker->consumption_->current_value() >= 0 (-1052615 vs. 0)
> Runtime Filter (Coordinator): Total=-1.00 MB Peak=1.00 MB
> *** Check failure stack trace: ***
> @ 0x2f1e11d google::LogMessage::Fail()
> @ 0x2f1f9c2 google::LogMessage::SendToLog()
> @ 0x2f1daf7 google::LogMessage::Flush()
> @ 0x2f210be google::LogMessageFatal::~LogMessageFatal()
> @ 0x17425fb impala::MemTracker::Release()
> @ 0x1fa7e8b impala::Coordinator::UpdateFilter()
> @ 0x186e3cf impala::ImpalaServer::UpdateFilter()
> @ 0x18d824f impala::ImpalaInternalService::UpdateFilter()
> @ 0x1dda35a impala::ImpalaInternalServiceProcessor::process_UpdateFilter()
> @ 0x1dd8308 impala::ImpalaInternalServiceProcessor::dispatchCall()
> @ 0x15410ea apache::thrift::TDispatchProcessor::process()
> @ 0x171042b apache::thrift::server::TAcceptQueueServer::Task::run()
> @ 0x170c307 impala::ThriftThread::RunRunnable()
> @ 0x170da13 boost::_mfi::mf2<>::operator()()
> @ 0x170d8a9 boost::_bi::list3<>::operator()<>()
> @ 0x170d5f5 boost::_bi::bind_t<>::operator()()
> @ 0x170d508 boost::detail::function::void_function_obj_invoker0<>::invoke()
> @ 0x171bdfc boost::function0<>::operator()()
> @ 0x19f3393 impala::Thread::SuperviseThread()
> @ 0x19fbf26 boost::_bi::list4<>::operator()<>()
> @ 0x19fbe69 boost::_bi::bind_t<>::operator()()
> @ 0x19fbe2c boost::detail::thread_data<>::run()
> @ 0x20a7c9a thread_proxy
> @ 0x7fe6536186ba start_thread
> @ 0x7fe65334e3dd clone
> r.java:81)
> {code}
> [~tarmstrong] suggested:
> {quote}
> Yeah it's probably another consequence of
> https://issues.apache.org/jira/browse/IMPALA-5789. Maybe your patch changed
> the timing enough to trigger it.
> I think the bug might be related to using directory.capacity() as the
> argument to Release(). Calling directory.clear() after releasing the memory
> in FitlerState::Disable() won't necessarily deallocate the memory so we
> could end up releasing it twice.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)