You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@impala.apache.org by "Joe McDonnell (JIRA)" <ji...@apache.org> on 2017/06/09 16:22:18 UTC
[jira] [Resolved] (IMPALA-5386)
disk-io-mgr-handle-cache.inline.h:124] Check failed: release_it !=
range.second
[ https://issues.apache.org/jira/browse/IMPALA-5386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joe McDonnell resolved IMPALA-5386.
-----------------------------------
Resolution: Fixed
Fix Version/s: Impala 2.10.0
commit 5f9f704bde6816c8469e31b1d655f3df5732dac4
Author: Joe McDonnell <jo...@cloudera.com>
Date: Tue May 30 12:59:37 2017 -0700
IMPALA-5386: Fix ReopenCachedHdfsFileHandle failure case
This fixes three issues with the file handle cache.
The first issue is that ReopenCachedHdfsFileHandle can
destroy the passed in file handle without removing the
reference to it. The old file handle then refers to
a piece of memory that is not a handle in the cache,
so future use of the handle fails with an assert. The
fix is to always overwrite the reference to the file
handle when it has been destroyed.
The second issue is that query_test/test_hdfs_fd_caching.py
should run on anything that supports the hdfs commandline
and tolerate query failure. It's logic is not specific to
file handle caching, so it has been renamed to
query_test/test_hdfs_file_mods.py.
Finally, custom_cluster/test_hdfs_fd_caching.py should not
be running on remote files (S3, ADLS, Isilon, remote
clusters). The file handle cache semantics won't apply on
those platforms.
Change-Id: Iee982fa5e964f6c8969b2eb7e5f3eca89e793b3a
Reviewed-on: http://gerrit.cloudera.org:8080/7020
Reviewed-by: Joe McDonnell <jo...@cloudera.com>
Reviewed-by: Dan Hecht <dh...@cloudera.com>
Tested-by: Impala Public Jenkins
> disk-io-mgr-handle-cache.inline.h:124] Check failed: release_it != range.second
> -------------------------------------------------------------------------------
>
> Key: IMPALA-5386
> URL: https://issues.apache.org/jira/browse/IMPALA-5386
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 2.9.0
> Reporter: Michael Ho
> Assignee: Joe McDonnell
> Priority: Blocker
> Labels: broken-build
> Fix For: Impala 2.10.0
>
>
> Running latest bit on asf-master hits the following DCHECK on Isilon hardware. [~joemcdonnell], can you please take a look ?
> {noformat}
> F0529 10:00:13.853025 15540 disk-io-mgr-handle-cache.inline.h:124] Check failed: release_it != range.second
> CORE: ./core.1496077215.15169.impalad
> BINARY: ./be/build/latest/service/impalad
> Core was generated by `/data/jenkins/workspace/impala-umbrella-build-and-test-isilon/repos/Impala/be/b'.
> Program terminated with signal 6, Aborted.
> #0 0x0000003069e328e5 in raise () from /lib64/libc.so.6
> To enable execution of this file add
> add-auto-load-safe-path /data/jenkins/workspace/impala-umbrella-build-and-test-isilon/Impala-Toolchain/gcc-4.9.2/lib64/libstdc++.so.6.0.20-gdb.py
> line to your configuration file "/var/lib/jenkins/.gdbinit".
> To completely disable this security protection add
> set auto-load safe-path /
> line to your configuration file "/var/lib/jenkins/.gdbinit".
> For more information about this security protection see the
> "Auto-loading safe path" section in the GDB manual. E.g., run from the shell:
> info "(gdb)Auto-loading safe path"
> #0 0x0000003069e328e5 in raise () from /lib64/libc.so.6
> #1 0x0000003069e340c5 in abort () from /lib64/libc.so.6
> #2 0x0000000002946f24 in google::DumpStackTraceAndExit() ()
> #3 0x000000000293d99d in google::LogMessage::Fail() ()
> #4 0x000000000293f242 in google::LogMessage::SendToLog() ()
> #5 0x000000000293d377 in google::LogMessage::Flush() ()
> #6 0x000000000294093e in google::LogMessageFatal::~LogMessageFatal() ()
> #7 0x000000000143ea49 in impala::FileHandleCache<16ul>::ReleaseFileHandle (this=0x7ef6b40, fname=0xa8cf9a8, fh=0xa758200, destroy_handle=false) at /data/jenkins/workspace/impala-umbrella-build-and-test-isilon/repos/Impala/be/src/runtime/disk-io-mgr-handle-cache.inline.h:124
> #8 0x000000000143ad1c in impala::DiskIoMgr::ReleaseCachedHdfsFileHandle (this=0x7ef6a00, fname=0xa8cf9a8, fid=0xa758200) at /data/jenkins/workspace/impala-umbrella-build-and-test-isilon/repos/Impala/be/src/runtime/disk-io-mgr.cc:1266
> #9 0x000000000144dcbc in impala::DiskIoMgr::ScanRange::Close (this=0xa8cf980) at /data/jenkins/workspace/impala-umbrella-build-and-test-isilon/repos/Impala/be/src/runtime/disk-io-mgr-scan-range.cc:352
> #10 0x000000000144bca3 in impala::DiskIoMgr::ScanRange::Cancel (this=0xa8cf980, status=...) at /data/jenkins/workspace/impala-umbrella-build-and-test-isilon/repos/Impala/be/src/runtime/disk-io-mgr-scan-range.cc:169
> #11 0x0000000001438ac3 in impala::DiskIoMgr::HandleReadFinished (this=0x7ef6a00, disk_queue=0x8d8cf80, reader=0xa01a3c0, buffer=0xf288df0) at /data/jenkins/workspace/impala-umbrella-build-and-test-isilon/repos/Impala/be/src/runtime/disk-io-mgr.cc:1000
> #12 0x0000000001439478 in impala::DiskIoMgr::ReadRange (this=0x7ef6a00, disk_queue=0x8d8cf80, reader=0xa01a3c0, range=0xa8cf980) at /data/jenkins/workspace/impala-umbrella-build-and-test-isilon/repos/Impala/be/src/runtime/disk-io-mgr.cc:1105
> #13 0x0000000001438e05 in impala::DiskIoMgr::WorkLoop (this=0x7ef6a00, disk_queue=0x8d8cf80) at /data/jenkins/workspace/impala-umbrella-build-and-test-isilon/repos/Impala/be/src/runtime/disk-io-mgr.cc:1047
> #14 0x00000000014451be in boost::_mfi::mf1<void, impala::DiskIoMgr, impala::DiskIoMgr::DiskQueue*>::operator() (this=0xa034380, p=0x7ef6a00, a1=0x8d8cf80) at /data/jenkins/workspace/impala-umbrella-build-and-test-isilon/Impala-Toolchain/boost-1.57.0-p1/include/boost/bind/mem_fn_template.hpp:165
> #15 0x0000000001444c61 in boost::_bi::list2<boost::_bi::value<impala::DiskIoMgr*>, boost::_bi::value<impala::DiskIoMgr::DiskQueue*> >::operator()<boost::_mfi::mf1<void, impala::DiskIoMgr, impala::DiskIoMgr::DiskQueue*>, boost::_bi::list0> (this=0xa034390, f=..., a=...) at /data/jenkins/workspace/impala-umbrella-build-and-test-isilon/Impala-Toolchain/boost-1.57.0-p1/include/boost/bind/bind.hpp:313
> #16 0x0000000001444409 in boost::_bi::bind_t<void, boost::_mfi::mf1<void, impala::DiskIoMgr, impala::DiskIoMgr::DiskQueue*>, boost::_bi::list2<boost::_bi::value<impala::DiskIoMgr*>, boost::_bi::value<impala::DiskIoMgr::DiskQueue*> > >::operator() (this=0xa034380) at /data/jenkins/workspace/impala-umbrella-build-and-test-isilon/Impala-Toolchain/boost-1.57.0-p1/include/boost/bind/bind_template.hpp:20
> #17 0x00000000014438c8 in boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf1<void, impala::DiskIoMgr, impala::DiskIoMgr::DiskQueue*>, boost::_bi::list2<boost::_bi::value<impala::DiskIoMgr*>, boost::_bi::value<impala::DiskIoMgr::DiskQueue*> > >, void>::invoke (function_obj_ptr=...) at /data/jenkins/workspace/impala-umbrella-build-and-test-isilon/Impala-Toolchain/boost-1.57.0-p1/include/boost/function/function_template.hpp:153
> #18 0x00000000013ba85c in boost::function0<void>::operator() (this=0x7ffc087eec20) at /data/jenkins/workspace/impala-umbrella-build-and-test-isilon/Impala-Toolchain/boost-1.57.0-p1/include/boost/function/function_template.hpp:767
> #19 0x00000000016862bf in impala::Thread::SuperviseThread (name=..., category=..., functor=..., thread_started=0x7fff2e6ebc10) at /data/jenkins/workspace/impala-umbrella-build-and-test-isilon/repos/Impala/be/src/util/thread.cc:325
> #20 0x000000000168ec42 in boost::_bi::list4<boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<boost::function<void()> >, boost::_bi::value<impala::Promise<long int>*> >::operator()<void (*)(const std::basic_string<char>&, const std::basic_string<char>&, boost::function<void()>, impala::Promise<long int>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(const std::basic_string<char, std::char_traits<char>, std::allocator<char> > &, const std::basic_string<char, std::char_traits<char>, std::allocator<char> > &, boost::function<void()>, impala::Promise<long> *), boost::_bi::list0 &, int) (this=0x8a7a3c0, f=@0x8a7a3b8, a=...) at /data/jenkins/workspace/impala-umbrella-build-and-test-isilon/Impala-Toolchain/boost-1.57.0-p1/include/boost/bind/bind.hpp:457
> #21 0x000000000168eb85 in boost::_bi::bind_t<void, void (*)(const std::basic_string<char, std::char_traits<char>, std::allocator<char> >&, const std::basic_string<char, std::char_traits<char>, std::allocator<char> >&, boost::function<void()>, impala::Promise<long int>*), boost::_bi::list4<boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<boost::function<void()> >, boost::_bi::value<impala::Promise<long int>*> > >::operator()(void) (this=0x8a7a3b8) at /data/jenkins/workspace/impala-umbrella-build-and-test-isilon/Impala-Toolchain/boost-1.57.0-p1/include/boost/bind/bind_template.hpp:20
> #22 0x000000000168eb48 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(const std::basic_string<char, std::char_traits<char>, std::allocator<char> >&, const std::basic_string<char, std::char_traits<char>, std::allocator<char> >&, boost::function<void()>, impala::Promise<long int>*), boost::_bi::list4<boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<boost::function<void()> >, boost::_bi::value<impala::Promise<long int>*> > > >::run(void) (this=0x8a7a200) at /data/jenkins/workspace/impala-umbrella-build-and-test-isilon/Impala-Toolchain/boost-1.57.0-p1/include/boost/thread/detail/thread.hpp:116
> #23 0x0000000001b6e38a in thread_proxy ()
> #24 0x000000306a207851 in start_thread () from /lib64/libpthread.so.0
> #25 0x0000003069ee894d in clone () from /lib64/libc.so.6
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)