You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Tim Armstrong (Jira)" <ji...@apache.org> on 2020/06/13 00:14:00 UTC
[jira] [Assigned] (IMPALA-6746) Reduce the number of comparison for
analytical functions with partitioning when incoming data is clustered
[ https://issues.apache.org/jira/browse/IMPALA-6746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tim Armstrong reassigned IMPALA-6746:
-------------------------------------
Assignee: (was: Adrian Ng)
> Reduce the number of comparison for analytical functions with partitioning when incoming data is clustered
> ----------------------------------------------------------------------------------------------------------
>
> Key: IMPALA-6746
> URL: https://issues.apache.org/jira/browse/IMPALA-6746
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
> Affects Versions: Impala 2.13.0
> Reporter: Mostafa Mokhtar
> Priority: Major
> Labels: performance
> Attachments: percentile query profile 2.txt
>
>
> Checking if the current row belongs to the same partition in ANALYTIC is very expensive, as it does N comparisons where N is number of rows, in cases when the cardinality of the partition column(s) is relatively small the values will be clustered.
> One optimization as proposed by [~alex.behm] is to check the first and last tuples in the batch and if they match go avoid calling AnalyticEvalNode::PrevRowCompare for the entire batch.
> For the query attached which is a common pattern the expected speedup is 20-30%.
> Query
> {code}
> select l_commitdate
> ,avg(l_extendedprice) as avg_perc
> ,percentile_cont (.25) within group (order by l_extendedprice asc) as perc_25
> ,percentile_cont (.5) within group (order by l_extendedprice asc) as perc_50
> ,percentile_cont (.75) within group (order by l_extendedprice asc) as perc_75
> ,percentile_cont (.90) within group (order by l_extendedprice asc) as perc_90
> from lineitem
> group by l_commitdate
> order by l_commitdate
> {code}
> Plan
> {code}
> F03:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
> | Per-Host Resources: mem-estimate=0B mem-reservation=0B
> PLAN-ROOT SINK
> | mem-estimate=0B mem-reservation=0B
> |
> 09:MERGING-EXCHANGE [UNPARTITIONED]
> | order by: l_commitdate ASC
> | mem-estimate=0B mem-reservation=0B
> | tuple-ids=5 row-size=66B cardinality=2559
> |
> F02:PLAN FRAGMENT [HASH(l_commitdate)] hosts=1 instances=1
> Per-Host Resources: mem-estimate=22.00MB mem-reservation=13.94MB
> 05:SORT
> | order by: l_commitdate ASC
> | mem-estimate=12.00MB mem-reservation=12.00MB spill-buffer=2.00MB
> | tuple-ids=5 row-size=66B cardinality=2559
> |
> 08:AGGREGATE [FINALIZE]
> | output: avg:merge(l_extendedprice), _percentile_cont_interpolation:merge(l_extendedprice, `_percentile_row_number_diff_0`), _percentile_cont_interpolation:merge(l_extendedprice, `_percentile_row_number_diff_1`), _percentile_cont_interpolation:merge(l_extendedprice, `_percentile_row_number_diff_2`), _percentile_cont_interpolation:merge(l_extendedprice, `_percentile_row_number_diff_3`)
> | group by: l_commitdate
> | mem-estimate=10.00MB mem-reservation=1.94MB spill-buffer=64.00KB
> | tuple-ids=4 row-size=66B cardinality=2559
> |
> 07:EXCHANGE [HASH(l_commitdate)]
> | mem-estimate=0B mem-reservation=0B
> | tuple-ids=3 row-size=66B cardinality=2559
> |
> F01:PLAN FRAGMENT [HASH(l_commitdate)] hosts=1 instances=1
> Per-Host Resources: mem-estimate=64.00MB mem-reservation=22.00MB
> 04:AGGREGATE [STREAMING]
> | output: avg(l_extendedprice), _percentile_cont_interpolation(l_extendedprice, row_number() - 1 - count(l_extendedprice) - 1 * 0.25), _percentile_cont_interpolation(l_extendedprice, row_number() - 1 - count(l_extendedprice) - 1 * 0.5), _percentile_cont_interpolation(l_extendedprice, row_number() - 1 - count(l_extendedprice) - 1 * 0.75), _percentile_cont_interpolation(l_extendedprice, row_number() - 1 - count(l_extendedprice) - 1 * 0.90)
> | group by: l_commitdate
> | mem-estimate=10.00MB mem-reservation=2.00MB spill-buffer=64.00KB
> | tuple-ids=3 row-size=66B cardinality=2559
> |
> 03:ANALYTIC
> | functions: count(l_extendedprice)
> | partition by: l_commitdate
> | mem-estimate=4.00MB mem-reservation=4.00MB spill-buffer=2.00MB
> | tuple-ids=9,7,8 row-size=50B cardinality=59986052
> |
> 02:ANALYTIC
> | functions: row_number()
> | partition by: l_commitdate
> | order by: l_extendedprice ASC
> | window: ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
> | mem-estimate=4.00MB mem-reservation=4.00MB spill-buffer=2.00MB
> | tuple-ids=9,7 row-size=42B cardinality=59986052
> |
> 01:SORT
> | order by: l_commitdate ASC NULLS FIRST, l_extendedprice ASC NULLS LAST
> | mem-estimate=46.00MB mem-reservation=12.00MB spill-buffer=2.00MB
> | tuple-ids=9 row-size=34B cardinality=59986052
> |
> 06:EXCHANGE [HASH(l_commitdate)]
> | mem-estimate=0B mem-reservation=0B
> | tuple-ids=0 row-size=34B cardinality=59986052
> |
> F00:PLAN FRAGMENT [RANDOM] hosts=1 instances=1
> Per-Host Resources: mem-estimate=88.00MB mem-reservation=0B
> 00:SCAN HDFS [tpch_10_parquet.lineitem, RANDOM]
> partitions=1/1 files=15 size=2.05GB
> stored statistics:
> table: rows=59986052 size=2.05GB
> columns: all
> extrapolated-rows=disabled
> mem-estimate=88.00MB mem-reservation=0B
> tuple-ids=0 row-size=34B cardinality=59986052
> {code}
> Call stack
> {code}
> libc.so.6!__memcmp_sse4_1 - memcmp-sse4.S
> impalad!StringCompare+0x14 - string-value.inline.h:40
> impalad!impala::StringValue::Eq+0x23 - string-value.inline.h:62
> impalad!impala::StringValue::operator==+0 - string-value.inline.h:66
> impalad!impala::Operators::Eq_StringVal_StringVal+0xd - operators-ir.cc:227
> impalad!impala::ScalarFnCall::InterpretEval<impala_udf::BooleanVal>+0x597 - scalar-fn-call.cc:485
> impalad!impala::ScalarFnCall::GetBooleanVal+0x24 - scalar-fn-call.cc:536
> impalad!impala::AndPredicate::GetBooleanVal+0x4d - compound-predicates.cc:36
> impalad!impala::OrPredicate::GetBooleanVal+0x4d - compound-predicates.cc:56
> impalad!impala::AndPredicate::GetBooleanVal+0x29 - compound-predicates.cc:33
> impalad!impala::ScalarExprEvaluator::GetBooleanVal+0x16 - scalar-expr-evaluator.cc:368
> impalad!impala::AnalyticEvalNode::PrevRowCompare+0xb - analytic-eval-node.cc:591
> impalad!impala::AnalyticEvalNode::ProcessChildBatch+0x227 - analytic-eval-node.cc:644
> impalad!impala::AnalyticEvalNode::ProcessChildBatches+0xf5 - analytic-eval-node.cc:604
> impalad!impala::AnalyticEvalNode::GetNext+0x269 - analytic-eval-node.cc:786
> impalad!impala::AnalyticEvalNode::ProcessChildBatches+0xbf - analytic-eval-node.cc:602
> impalad!impala::AnalyticEvalNode::GetNext+0x269 - analytic-eval-node.cc:786
> impalad!impala::PartitionedAggregationNode::GetRowsStreaming+0xa6 - partitioned-aggregation-node.cc:478
> impalad!impala::PartitionedAggregationNode::GetNext+0x221 - partitioned-aggregation-node.cc:369
> impalad!impala::FragmentInstanceState::ExecInternal+0x1b1 - fragment-instance-state.cc:277
> impalad!impala::FragmentInstanceState::Exec+0x29e - fragment-instance-state.cc:89
> impalad!impala::QueryState::ExecFInstance+0x249 - query-state.cc:394
> impalad!boost::function0<void>::operator()+0x1a - function_template.hpp:767
> impalad!impala::Thread::SuperviseThread+0x2e4 - thread.cc:356
> impalad!operator()<void (*)(const std::basic_string<char>&, const std::basic_string<char>&, boost::function<void()>, const impala::ThreadDebugInfo*, impala::Promise<long int>*), boost::_bi::list0>+0x5b - bind.hpp:525
> impalad!boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void (void)>, impala::ThreadDebugInfo const*, impala::Promise<long>*), boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void (void)>>, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long>*>>>::operator()+0 - bind_template.hpp:20
> impalad!boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void (void)>, impala::ThreadDebugInfo const*, impala::Promise<long>*), boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void (void)>>, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long>*>>>>::run+0x1e - thread.hpp:116
> impalad!thread_proxy+0xd9 - [Unknown]:[Unknown]
> libpthread.so.0!start_thread+0xc1 - pthread_create.c:312
> libc.so.6!__clone+0x6c - clone.S:111
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org