You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@impala.apache.org by "Taras Bobrovytsky (JIRA)" <ji...@apache.org> on 2017/05/04 02:15:04 UTC
[jira] [Resolved] (IMPALA-5188) DCHECK in
UnionNode::GetNextPassThrough with GROUP BY, AVG
[ https://issues.apache.org/jira/browse/IMPALA-5188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Taras Bobrovytsky resolved IMPALA-5188.
---------------------------------------
Resolution: Fixed
Fix Version/s: Impala 2.9.0
{code}
commit 50e3abdc3daf89c76d0d40e7f8c926477c2a1db4
Author: Taras Bobrovytsky <tb...@cloudera.com>
Date: Tue Apr 11 14:07:19 2017 -0700
IMPALA-5188: Add slot sorting in TupleDescriptor::LayoutEquals()
The slot descriptor vectors are not guaranteed to be sorted on the slot
index within a tuple. As a result, TupleDescriptor::LayoutEquals()
sometimes returned a wrong result.
In this patch, we sort the vectors of slot descriptors on the slot index
within the tuple before comparing the vectors.
Testing:
- ran EE tests locally.
Change-Id: I426ad244678dbfe517262dfb7bbf4adc0247a35e
Reviewed-on: http://gerrit.cloudera.org:8080/6610
Reviewed-by: Dan Hecht <dh...@cloudera.com>
Reviewed-by: Alex Behm <al...@cloudera.com>
Tested-by: Impala Public Jenkins
{code}
> DCHECK in UnionNode::GetNextPassThrough with GROUP BY, AVG
> ----------------------------------------------------------
>
> Key: IMPALA-5188
> URL: https://issues.apache.org/jira/browse/IMPALA-5188
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 2.9.0
> Reporter: Michael Brown
> Assignee: Taras Bobrovytsky
> Priority: Blocker
> Labels: crash, query_generator, regression
> Fix For: Impala 2.9.0
>
>
> This query:
> {noformat}
> SELECT
> 1,
> 1
> UNION ALL
> SELECT
> AVG(id),
> id
> FROM functional.alltypes
> GROUP BY
> id
> {noformat}
> ... is sufficient enough to hit the DCHECK, which is:
> {noformat}
> union-node.cc:113] Check failed: child(child_idx_)->row_desc().LayoutEquals(row_batch->row_desc())
> {noformat}
> {noformat}
> #6 0x00000000028c502e in google::LogMessageFatal::~LogMessageFatal() ()
> #7 0x0000000001801da4 in impala::UnionNode::GetNextPassThrough (this=0xadff900, state=0xa704000, row_batch=0x7e03960) at /home/mikeb/Impala/be/src/exec/union-node.cc:113
> #8 0x00000000018043d7 in impala::UnionNode::GetNext (this=0xadff900, state=0xa704000, row_batch=0x7e03960, eos=0x7eff74b2cebf) at /home/mikeb/Impala/be/src/exec/union-node.cc:255
> #9 0x0000000001a73940 in impala::PlanFragmentExecutor::ExecInternal (this=0xa3c33d0) at /home/mikeb/Impala/be/src/runtime/plan-fragment-executor.cc:360
> #10 0x0000000001a735ec in impala::PlanFragmentExecutor::Exec (this=0xa3c33d0) at /home/mikeb/Impala/be/src/runtime/plan-fragment-executor.cc:337
> #11 0x0000000001a6cd8b in impala::FragmentInstanceState::Exec (this=0xa3c3100) at /home/mikeb/Impala/be/src/runtime/fragment-instance-state.cc:68
> #12 0x0000000001a783ef in impala::QueryExecMgr::ExecFInstance (this=0x9e01020, fis=0xa3c3100) at /home/mikeb/Impala/be/src/runtime/query-exec-mgr.cc:110
> #13 0x0000000001a7b218 in boost::_mfi::mf1<void, impala::QueryExecMgr, impala::FragmentInstanceState*>::operator() (this=0x8ee6920, p=0x9e01020, a1=0xa3c3100)
> at /home/mikeb/Impala/toolchain/boost-1.57.0-p1/include/boost/bind/mem_fn_template.hpp:165
> #14 0x0000000001a7b0a1 in boost::_bi::list2<boost::_bi::value<impala::QueryExecMgr*>, boost::_bi::value<impala::FragmentInstanceState*> >::operator()<boost::_mfi::mf1<void, impala::QueryExecMgr, impala::FragmentInstanceState*>, boost::_bi::list0> (this=0x8ee6930, f=..., a=...) at /home/mikeb/Impala/toolchain/boost-1.57.0-p1/include/boost/bind/bind.hpp:313
> #15 0x0000000001a7abd5 in boost::_bi::bind_t<void, boost::_mfi::mf1<void, impala::QueryExecMgr, impala::FragmentInstanceState*>, boost::_bi::list2<boost::_bi::value<impala::QueryExecMgr*>, boost::_bi::value<impala::FragmentInstanceState*> > >::operator() (this=0x8ee6920) at /home/mikeb/Impala/toolchain/boost-1.57.0-p1/include/boost/bind/bind_template.hpp:20
> #16 0x0000000001a7a7e0 in boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf1<void, impala::QueryExecMgr, impala::FragmentInstanceState*>, boost::_bi::list2<boost::_bi::value<impala::QueryExecMgr*>, boost::_bi::value<impala::FragmentInstanceState*> > >, void>::invoke (function_obj_ptr=...)
> at /home/mikeb/Impala/toolchain/boost-1.57.0-p1/include/boost/function/function_template.hpp:153
> #17 0x000000000137bd48 in boost::function0<void>::operator() (this=0x7eff74b2dd20) at /home/mikeb/Impala/toolchain/boost-1.57.0-p1/include/boost/function/function_template.hpp:767
> #18 0x000000000162e363 in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*) (name=..., category=..., functor=...,
> thread_started=0x7eff7632fa80) at /home/mikeb/Impala/be/src/util/thread.cc:325
> #19 0x0000000001636d3e in boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> >::operator()<void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list0&, int) (this=0x9eb77c0,
> f=@0x9eb77b8: 0x162e09e <impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*)>, a=...)
> at /home/mikeb/Impala/toolchain/boost-1.57.0-p1/include/boost/bind/bind.hpp:457
> #20 0x0000000001636c81 in boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> > >::operator()() (this=0x9eb77b8)
> at /home/mikeb/Impala/toolchain/boost-1.57.0-p1/include/boost/bind/bind_template.hpp:20
> #21 0x0000000001636c44 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> > > >::run() (this=0x9eb7600)
> at /home/mikeb/Impala/toolchain/boost-1.57.0-p1/include/boost/thread/detail/thread.hpp:116
> #22 0x0000000001af270a in thread_proxy ()
> #23 0x00007f0000172184 in start_thread (arg=0x7eff74b2e700) at pthread_create.c:312
> #24 0x00007effffe9f37d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
> {noformat}
> Plan:
> {noformat}
> +-----------------------------------------------------------+
> | Explain String |
> +-----------------------------------------------------------+
> | Estimated Per-Host Requirements: Memory=180.00MB VCores=2 |
> | |
> | PLAN-ROOT SINK |
> | | |
> | 05:EXCHANGE [UNPARTITIONED] |
> | | |
> | 00:UNION |
> | | constant-operands=1 |
> | | pass-through-operands: all |
> | | |
> | 04:AGGREGATE [FINALIZE] |
> | | output: avg:merge(id) |
> | | group by: id |
> | | |
> | 03:EXCHANGE [HASH(id)] |
> | | |
> | 02:AGGREGATE [STREAMING] |
> | | output: avg(id) |
> | | group by: id |
> | | |
> | 01:SCAN HDFS [functional.alltypes] |
> | partitions=24/24 files=24 size=478.45KB |
> +-----------------------------------------------------------+
> {noformat}
> That's as simple a query I could get. The {{AVG()}} and {{GROUP BY}} together seemed to be needed. For example, this query creates a similar plan but does not hit the dcheck:
> {noformat}
> SELECT
> 1
> UNION ALL
> SELECT
> id
> FROM functional.alltypes
> GROUP BY
> id
> {noformat}
> Original query generator query:
> {noformat}
> USE tpch_kudu;
> SELECT DISTINCT
> a1.n_nationkey,
> CAST(COALESCE(a1.n_nationkey, a1.n_regionkey, a1.n_nationkey) AS STRING) AS char_col,
> COALESCE(a1.n_regionkey, GREATEST(COALESCE(a1.n_nationkey, -70), COALESCE(a1.n_nationkey, -909)), a1.n_nationkey) AS int_col,
> COALESCE(a1.n_regionkey, a1.n_nationkey, COALESCE(a1.n_nationkey, a1.n_nationkey, LEAST(COALESCE(a1.n_nationkey, -273), COALESCE(a1.n_nationkey, 967)))) AS int_col_1
> FROM nation a1
> UNION ALL
> SELECT
> AVG(COALESCE(a2.s_suppkey, a2.s_nationkey, LEAST(COALESCE(a2.s_suppkey, 547), COALESCE(a2.s_suppkey, -534)))) AS float_col,
> CAST(COALESCE(a3.p_partkey, a3.p_partkey, a2.s_nationkey) AS STRING) AS char_col,
> AVG(a2.s_suppkey) AS float_col_1,
> GREATEST(COALESCE(LEAST(COALESCE(a2.s_suppkey, -764), 178.6745398063), 455.7368346861), COALESCE(a2.s_suppkey, 720)) AS decimal_col
> FROM supplier a2
> INNER JOIN part a3 ON (a2.s_nationkey) = (a3.p_size)
> GROUP BY
> CAST(COALESCE(a3.p_partkey, a3.p_partkey, a2.s_nationkey) AS STRING),
> GREATEST(COALESCE(LEAST(COALESCE(a2.s_suppkey, -764), 178.6745398063), 455.7368346861), COALESCE(a2.s_suppkey, 720))
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)