You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Xianqing He (Jira)" <ji...@apache.org> on 2021/07/16 07:22:00 UTC
[jira] [Comment Edited] (IMPALA-10799) Analysis slowdown with
inline views and thousands of column
[ https://issues.apache.org/jira/browse/IMPALA-10799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17381849#comment-17381849 ]
Xianqing He edited comment on IMPALA-10799 at 7/16/21, 7:21 AM:
----------------------------------------------------------------
{code:java}
`---[3082.909883ms] org.apache.impala.analysis.InlineViewRef:analyze()
+---[0.019775ms] org.apache.impala.analysis.Analyzer:<init>() #197
+---[0.005544ms] org.apache.impala.analysis.InlineViewRef:isCatalogView() #200
+---[0.097928ms] org.apache.impala.analysis.Analyzer:registerAuthAndAuditEvent() #201
+---[0.005474ms] org.apache.impala.analysis.Analyzer:isExplain() #202
+---[0.006005ms] org.apache.impala.analysis.Analyzer:getUser() #208
+---[0.008159ms] org.apache.impala.authorization.User:getName() #95
+---[0.005792ms] org.apache.impala.analysis.Analyzer:setMaskPrivChecks() #206
+---[0.003173ms] org.apache.impala.analysis.InlineViewRef:isCatalogView() #218
+---[0.005068ms] org.apache.impala.analysis.Analyzer:setUseHiveColLabels() #217
+---[61.537633ms] org.apache.impala.analysis.QueryStmt:analyze() #219
| `---[0.325471ms] org.apache.impala.analysis.BaseTableRef:analyze()
| +---[0.006527ms] org.apache.impala.analysis.Path:getRootTable() #63
| +---[0.0231ms] org.apache.impala.analysis.Analyzer:registerAuthAndAuditEvent() #95
| +---[0.094469ms] org.apache.impala.analysis.Analyzer:registerTableRef() #65
| +---[0.0031ms] org.apache.impala.analysis.BaseTableRef:getTable() #67
| +---[0.011061ms] org.apache.impala.analysis.Analyzer:checkTableCapability() #95
| +---[0.016223ms] org.apache.impala.analysis.BaseTableRef:analyzeTableSample() #68
| +---[0.01388ms] org.apache.impala.analysis.BaseTableRef:analyzeHints() #69
| +---[0.059547ms] org.apache.impala.analysis.BaseTableRef:analyzeJoin() #70
| `---[0.019664ms] org.apache.impala.analysis.BaseTableRef:analyzeSkipHeaderLineCount() #71
+---[0.027886ms] org.apache.impala.analysis.QueryStmt:getCorrelatedTupleIds() #220
+---[0.014883ms] org.apache.impala.analysis.QueryStmt:hasLimit() #227
+---[0.018362ms] org.apache.impala.analysis.QueryStmt:hasOffset() #95
+---[0.006523ms] org.apache.impala.analysis.Analyzer:setHasLimitOffsetClause() #226
+---[0.010455ms] org.apache.impala.analysis.QueryStmt:getMaterializedTupleIds() #228
+---[5.208874ms] org.apache.impala.analysis.Analyzer:registerTableRef() #229
+---[min=0.001294ms,max=0.018521ms,total=15.462617ms,count=10001] org.apache.impala.analysis.InlineViewRef:getColLabels() #251
+---[min=0.001285ms,max=0.022459ms,total=15.423138ms,count=10000] org.apache.impala.analysis.InlineViewRef:getColLabels() #252
+---[min=0.001288ms,max=0.020934ms,total=15.491207ms,count=10000] org.apache.impala.analysis.QueryStmt:getResultExprs() #253
+---[min=0.001381ms,max=0.030606ms,total=16.896863ms,count=10000] com.google.common.collect.Lists:newArrayList() #254
+---[min=0.001461ms,max=0.018934ms,total=18.045528ms,count=10000] org.apache.impala.analysis.Path:<init>() #95
+---[min=0.001197ms,max=0.020106ms,total=17.061368ms,count=10000] org.apache.impala.analysis.Path:resolve() #255
+---[min=0.001135ms,max=0.02319ms,total=13.949841ms,count=10001] com.google.common.base.Preconditions:checkState() #95
+---[min=0.002067ms,max=0.474963ms,total=28.860936ms,count=10000] org.apache.impala.analysis.Analyzer:registerSlotRef() #256
+---[min=0.001421ms,max=0.02092ms,total=17.052748ms,count=10000] org.apache.impala.analysis.SlotDescriptor:setSourceExpr() #257
+---[min=0.001613ms,max=0.024081ms,total=21.472925ms,count=10000] org.apache.impala.catalog.ColumnStats:fromExpr() #258
+---[min=0.001404ms,max=0.019149ms,total=17.096331ms,count=10000] org.apache.impala.analysis.SlotDescriptor:setStats() #95
+---[min=0.001417ms,max=0.043865ms,total=17.257881ms,count=10000] org.apache.impala.analysis.SlotRef:<init>() #259
+---[min=0.001536ms,max=32.649046ms,total=51.539365ms,count=10000] org.apache.impala.analysis.ExprSubstitutionMap:put() #260
+---[min=0.001329ms,max=0.686587ms,total=16.605861ms,count=10000] org.apache.impala.analysis.QueryStmt:getBaseTblResultExprs() #261
+---[min=0.001526ms,max=0.027195ms,total=18.329066ms,count=10000] org.apache.impala.analysis.ExprSubstitutionMap:put() #95
+---[min=0.001452ms,max=0.06173ms,total=17.527132ms,count=10000] org.apache.impala.analysis.InlineViewRef:createAuxPredicate() #262
+---[min=0.001391ms,max=0.019388ms,total=16.723757ms,count=10000] org.apache.impala.analysis.SlotRef:<init>() #263
+---[min=0.001277ms,max=0.042461ms,total=15.751651ms,count=10000] org.apache.impala.analysis.Expr:clone() #95
+---[min=0.00275ms,max=0.223975ms,total=34.853788ms,count=10000] org.apache.impala.analysis.Analyzer:createAuxEqPredicate() #95
+---[0.029475ms] org.slf4j.Logger:isTraceEnabled() #266
+---[2302.367335ms] org.apache.impala.analysis.ExprSubstitutionMap:checkComposedFrom() #271
+---[0.022327ms] org.apache.impala.analysis.InlineViewRef:analyzeTableSample() #273
+---[0.013037ms] org.apache.impala.analysis.InlineViewRef:analyzeHints() #274
`---[0.033295ms] org.apache.impala.analysis.InlineViewRef:analyzeJoin() #276
{code}
The other majority of the time is spent in *org.apache.impala.analysis.ExprSubstitutionMap:checkComposedFrom*
was (Author: xqhe):
`---[3082.909883ms] org.apache.impala.analysis.InlineViewRef:analyze()
+---[0.019775ms] org.apache.impala.analysis.Analyzer:<init>() #197
+---[0.005544ms] org.apache.impala.analysis.InlineViewRef:isCatalogView() #200
+---[0.097928ms] org.apache.impala.analysis.Analyzer:registerAuthAndAuditEvent() #201
+---[0.005474ms] org.apache.impala.analysis.Analyzer:isExplain() #202
+---[0.006005ms] org.apache.impala.analysis.Analyzer:getUser() #208
+---[0.008159ms] org.apache.impala.authorization.User:getName() #95
+---[0.005792ms] org.apache.impala.analysis.Analyzer:setMaskPrivChecks() #206
+---[0.003173ms] org.apache.impala.analysis.InlineViewRef:isCatalogView() #218
+---[0.005068ms] org.apache.impala.analysis.Analyzer:setUseHiveColLabels() #217
+---[61.537633ms] org.apache.impala.analysis.QueryStmt:analyze() #219
| `---[0.325471ms] org.apache.impala.analysis.BaseTableRef:analyze()
| +---[0.006527ms] org.apache.impala.analysis.Path:getRootTable() #63
| +---[0.0231ms] org.apache.impala.analysis.Analyzer:registerAuthAndAuditEvent() #95
| +---[0.094469ms] org.apache.impala.analysis.Analyzer:registerTableRef() #65
| +---[0.0031ms] org.apache.impala.analysis.BaseTableRef:getTable() #67
| +---[0.011061ms] org.apache.impala.analysis.Analyzer:checkTableCapability() #95
| +---[0.016223ms] org.apache.impala.analysis.BaseTableRef:analyzeTableSample() #68
| +---[0.01388ms] org.apache.impala.analysis.BaseTableRef:analyzeHints() #69
| +---[0.059547ms] org.apache.impala.analysis.BaseTableRef:analyzeJoin() #70
| `---[0.019664ms] org.apache.impala.analysis.BaseTableRef:analyzeSkipHeaderLineCount() #71
+---[0.027886ms] org.apache.impala.analysis.QueryStmt:getCorrelatedTupleIds() #220
+---[0.014883ms] org.apache.impala.analysis.QueryStmt:hasLimit() #227
+---[0.018362ms] org.apache.impala.analysis.QueryStmt:hasOffset() #95
+---[0.006523ms] org.apache.impala.analysis.Analyzer:setHasLimitOffsetClause() #226
+---[0.010455ms] org.apache.impala.analysis.QueryStmt:getMaterializedTupleIds() #228
+---[5.208874ms] org.apache.impala.analysis.Analyzer:registerTableRef() #229
+---[min=0.001294ms,max=0.018521ms,total=15.462617ms,count=10001] org.apache.impala.analysis.InlineViewRef:getColLabels() #251
+---[min=0.001285ms,max=0.022459ms,total=15.423138ms,count=10000] org.apache.impala.analysis.InlineViewRef:getColLabels() #252
+---[min=0.001288ms,max=0.020934ms,total=15.491207ms,count=10000] org.apache.impala.analysis.QueryStmt:getResultExprs() #253
+---[min=0.001381ms,max=0.030606ms,total=16.896863ms,count=10000] com.google.common.collect.Lists:newArrayList() #254
+---[min=0.001461ms,max=0.018934ms,total=18.045528ms,count=10000] org.apache.impala.analysis.Path:<init>() #95
+---[min=0.001197ms,max=0.020106ms,total=17.061368ms,count=10000] org.apache.impala.analysis.Path:resolve() #255
+---[min=0.001135ms,max=0.02319ms,total=13.949841ms,count=10001] com.google.common.base.Preconditions:checkState() #95
+---[min=0.002067ms,max=0.474963ms,total=28.860936ms,count=10000] org.apache.impala.analysis.Analyzer:registerSlotRef() #256
+---[min=0.001421ms,max=0.02092ms,total=17.052748ms,count=10000] org.apache.impala.analysis.SlotDescriptor:setSourceExpr() #257
+---[min=0.001613ms,max=0.024081ms,total=21.472925ms,count=10000] org.apache.impala.catalog.ColumnStats:fromExpr() #258
+---[min=0.001404ms,max=0.019149ms,total=17.096331ms,count=10000] org.apache.impala.analysis.SlotDescriptor:setStats() #95
+---[min=0.001417ms,max=0.043865ms,total=17.257881ms,count=10000] org.apache.impala.analysis.SlotRef:<init>() #259
+---[min=0.001536ms,max=32.649046ms,total=51.539365ms,count=10000] org.apache.impala.analysis.ExprSubstitutionMap:put() #260
+---[min=0.001329ms,max=0.686587ms,total=16.605861ms,count=10000] org.apache.impala.analysis.QueryStmt:getBaseTblResultExprs() #261
+---[min=0.001526ms,max=0.027195ms,total=18.329066ms,count=10000] org.apache.impala.analysis.ExprSubstitutionMap:put() #95
+---[min=0.001452ms,max=0.06173ms,total=17.527132ms,count=10000] org.apache.impala.analysis.InlineViewRef:createAuxPredicate() #262
+---[min=0.001391ms,max=0.019388ms,total=16.723757ms,count=10000] org.apache.impala.analysis.SlotRef:<init>() #263
+---[min=0.001277ms,max=0.042461ms,total=15.751651ms,count=10000] org.apache.impala.analysis.Expr:clone() #95
+---[min=0.00275ms,max=0.223975ms,total=34.853788ms,count=10000] org.apache.impala.analysis.Analyzer:createAuxEqPredicate() #95
+---[0.029475ms] org.slf4j.Logger:isTraceEnabled() #266
+---*{color:#FF0000}[2302.367335ms] org.apache.impala.analysis.ExprSubstitutionMap:checkComposedFrom(){color}* #271
+---[0.022327ms] org.apache.impala.analysis.InlineViewRef:analyzeTableSample() #273
+---[0.013037ms] org.apache.impala.analysis.InlineViewRef:analyzeHints() #274
`---[0.033295ms] org.apache.impala.analysis.InlineViewRef:analyzeJoin() #276
The other majority of the time is spent in *org.apache.impala.analysis.ExprSubstitutionMap:checkComposedFrom*
> Analysis slowdown with inline views and thousands of column
> -----------------------------------------------------------
>
> Key: IMPALA-10799
> URL: https://issues.apache.org/jira/browse/IMPALA-10799
> Project: IMPALA
> Issue Type: Improvement
> Components: Frontend
> Affects Versions: Impala 3.2.0
> Reporter: Xianqing He
> Assignee: Xianqing He
> Priority: Major
>
> If there are thousands of columns in the inlineview, it‘s very slow in analysis. For example, this sql will take almost 4s in analysis if the inline view has tens of thousands of column
> {code:java}
> select c1 from (select c1, c2... c10001 from T) T
> Query Compilation: 3s880ms
> - Translate start: 968.000ns (968.000ns)
> - Translate finished: 4.318ms (4.317ms)
> - Metadata of all 1 tables cached: 42.219ms (37.900ms)
> - Analysis finished: 3s776ms (3s734ms)
> - Value transfer graph computed: 3s806ms (30.163ms)
> - Single node plan created: 3s869ms (62.556ms)
> - Runtime filters computed: 3s874ms (5.603ms)
> - Distributed plan created: 3s874ms (128.086us)
> - Planning finished: 3s880ms (5.836ms)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org