You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Wenzhe Zhou (Jira)" <ji...@apache.org> on 2021/11/19 01:12:00 UTC
[jira] [Comment Edited] (IMPALA-11029) DescriptorTable.copyTupleDescriptor throw exception for Kudu table
[ https://issues.apache.org/jira/browse/IMPALA-11029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17446224#comment-17446224 ]
Wenzhe Zhou edited comment on IMPALA-11029 at 11/19/21, 1:11 AM:
-----------------------------------------------------------------
In AnalyticEvalNode.constructEqExprs(), bufferedTupleDesc is created by calling DescriptorTable.copyTupleDescriptor().
In DescriptorTable.copyTupleDescriptor(), a TupleDescriptor object is created with slots copied from source TupleDescriptor, but its "path" member variable is set as null. This make the table associated with the bufferedTupleDesc detected as non Kudu table, and SlotDescriptor.isKuduStringSlot() always returns false for its copied slots. TupleDescriptor.getSlotSize() add 4 padding bytes for Kudu string slot. When the function is called to calculate slot size for string type slots for bufferedTupleDesc, the slot size will be 4 bytes less than the slot size calculated for the source TupleDescriptor if the table associated with source TupleDescriptor is Kudu table. This cause "Preconditions.checkState(d.getByteSize() == src.getByteSize())" in DescriptorTable.copyTupleDescriptor() fail to throw exception.
We should copy "path" from source TupleDescriptor for bufferedTupleDesc before coping slots from source TupleDescriptor.
was (Author: wzhou):
In AnalyticEvalNode.constructEqExprs(), bufferedTupleDesc is created by calling DescriptorTable.copyTupleDescriptor().
In DescriptorTable.copyTupleDescriptor(), a TupleDescriptor object is created with slots copied from source TupleDescriptor, but its "path" member variable is set as null. This make the table associated with the bufferedTupleDesc as non Kudu table, and SlotDescriptor.isKuduStringSlot() always returns false for its copied slots. TupleDescriptor.getSlotSize() add 4 padding bytes for Kudu string slot. When the function is called to calculate slot size for string type slots for bufferedTupleDesc, the slot size will be 4 bytes less than the slot size calculated for the source TupleDescriptor if the table associated with source TupleDescriptor is Kudu table. This cause "Preconditions.checkState(d.getByteSize() == src.getByteSize())" in DescriptorTable.copyTupleDescriptor() fail to throw exception.
We should copy "path" from source TupleDescriptor for bufferedTupleDesc before coping slots from source TupleDescriptor.
> DescriptorTable.copyTupleDescriptor throw exception for Kudu table
> ------------------------------------------------------------------
>
> Key: IMPALA-11029
> URL: https://issues.apache.org/jira/browse/IMPALA-11029
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
> Reporter: Wenzhe Zhou
> Assignee: Wenzhe Zhou
> Priority: Major
> Fix For: Impala 4.1.0
>
>
> This is a customer reported bug. In the case, planner throw exception with following stack trace:
> I1115 07:52:39.779211 14750 jni-util.cc:286] 4a467bd0ffba8c4b:0ce1e98a00000000] java.lang.IllegalStateException
> at com.google.common.base.Preconditions.checkState(Preconditions.java:492)
> at org.apache.impala.analysis.DescriptorTable.copyTupleDescriptor(DescriptorTable.java:85)
> at org.apache.impala.planner.AnalyticEvalNode.constructEqExprs(AnalyticEvalNode.java:184)
> at org.apache.impala.planner.AnalyticEvalNode.init(AnalyticEvalNode.java:158)
> at org.apache.impala.planner.AnalyticPlanner.createSortGroupPlan(AnalyticPlanner.java:495)
> at org.apache.impala.planner.AnalyticPlanner.createSingleNodePlan(AnalyticPlanner.java:139)
> at org.apache.impala.planner.SingleNodePlanner.createQueryPlan(SingleNodePlanner.java:295)
> at org.apache.impala.planner.SingleNodePlanner.createSingleNodePlan(SingleNodePlanner.java:170)
> at org.apache.impala.planner.Planner.createPlanFragments(Planner.java:120)
> at org.apache.impala.planner.Planner.createPlans(Planner.java:249)
> at org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1562)
> at org.apache.impala.service.Frontend.getPlannedExecRequest(Frontend.java:1939)
> at org.apache.impala.service.Frontend.doCreateExecRequest(Frontend.java:1779)
> at org.apache.impala.service.Frontend.getTExecRequest(Frontend.java:1644)
> at org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1614)
> at org.apache.impala.service.JniFrontend.createExecRequest(JniFrontend.java:164)
> I1115 07:52:39.779265 14750 status.cc:129] 4a467bd0ffba8c4b:0ce1e98a00000000] IllegalStateException: null
> @ 0x1f94ad9 impala::Status::Status()
> @ 0x297e8ed impala::JniUtil::GetJniExceptionMsg()
> @ 0x26b4180 impala::JniCall::Call<>()
> @ 0x26b0c07 impala::JniUtil::CallJniMethod<>()
> @ 0x26aec22 impala::Frontend::GetExecRequest()
> @ 0x30dda65 impala::QueryDriver::RunFrontendPlanner()
> @ 0x2713244 impala::ImpalaServer::ExecuteInternal()
> @ 0x2712791 impala::ImpalaServer::Execute()
> @ 0x285f9f4 impala::ImpalaServer::ExecuteStatementCommon()
> @ 0x2860895 impala::ImpalaServer::ExecuteStatement()
> @ 0x27727c7 apache::hive::service::cli::thrift::TCLIServiceProcessorT<>::process_ExecuteStatement()
> @ 0x27f4161 apache::hive::service::cli::thrift::TCLIServiceProcessorT<>::dispatchCall()
> @ 0x27f3d58 impala::ImpalaHiveServer2ServiceProcessorT<>::dispatchCall()
> @ 0x1e6ee6f apache::thrift::TDispatchProcessor::process()
> @ 0x2348784 apache::thrift::server::TAcceptQueueServer::Task::run()
> @ 0x233bfea impala::ThriftThread::RunRunnable()
> @ 0x233d606 boost::_mfi::mf2<>::operator()()
> @ 0x233d49a boost::_bi::list3<>::operator()<>()
> @ 0x233d1f0 boost::_bi::bind_t<>::operator()()
> @ 0x233d0b9 boost::detail::function::void_function_obj_invoker0<>::invoke()
> @ 0x22ab5cd boost::function0<>::operator()()
> @ 0x2a76d5e impala::Thread::SuperviseThread()
> @ 0x2a7f6ae boost::_bi::list5<>::operator()<>()
> @ 0x2a7f5d2 boost::_bi::bind_t<>::operator()()
> @ 0x2a7f593 boost::detail::thread_data<>::run()
> @ 0x43763b0 thread_proxy
> @ 0x7fd8ae06bea4 start_thread
> @ 0x7fd8aa9df9fc __clone
>
> The bug could be reproduced in Impala upstream with following queries:
> create database mydb;
> CREATE TABLE mydb.my_tab1 (
> id1 INT NOT NULL,
> agrmt INT NOT NULL,
> big_id BIGINT NOT NULL,
> outdated_flag STRING NOT NULL,
> mod_ts TIMESTAMP NOT NULL,
> PRIMARY KEY (id1, agrmt)
> )
> PARTITION BY HASH (id1) PARTITIONS 2
> STORED AS KUDU;
> CREATE TABLE mydb.my_tab2 (
> cl_id INT NOT NULL,
> cl_agrmt INT NOT NULL,
> outdat STRING NULL,
> mod_dat TIMESTAMP NULL,
> PRIMARY KEY (cl_id, cl_agrmt)
> )
> PARTITION BY HASH (cl_id) PARTITIONS 2
> STORED AS KUDU;
> insert into mydb.my_tab1 (
> id1,
> agrmt,
> big_id,
> outdated_flag,
> mod_ts)
> select
> cl_id,
> cast(row_number() over(order by null) as int),
> cl_agrmt,
> 'Y',
> case when outdat='Y' and mod_dat is not null then mod_dat else now() end
> from mydb.my_tab2 i
> left join mydb.my_tab1 u
> on u.big_id=i.cl_agrmt
> left join (
> select id1, big_id
> from mydb.my_tab1
> group by id1, big_id) uu
> on uu.big_id=i.cl_agrmt
> where u.big_id is null;
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org