You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Quanlong Huang (Jira)" <ji...@apache.org> on 2019/08/31 15:09:00 UTC
[jira] [Commented] (IMPALA-8912) Avoid calling computeStats twice
on HBaseScanNode
[ https://issues.apache.org/jira/browse/IMPALA-8912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16920148#comment-16920148 ]
Quanlong Huang commented on IMPALA-8912:
----------------------------------------
The second call is required since the limit_ may be updated, which may cut the cardinality. I think what we should avoid is sampling twice on hbase.
> Avoid calling computeStats twice on HBaseScanNode
> -------------------------------------------------
>
> Key: IMPALA-8912
> URL: https://issues.apache.org/jira/browse/IMPALA-8912
> Project: IMPALA
> Issue Type: Improvement
> Components: Frontend
> Affects Versions: Impala 2.8.0, Impala 2.9.0, Impala 2.10.0, Impala 2.11.0, Impala 3.0, Impala 2.12.0, Impala 3.1.0, Impala 3.2.0, Impala 3.3.0
> Reporter: Quanlong Huang
> Priority: Major
>
> For simple queries on HBase tables that has HBaseScanNode as the root of the SingleNodePlan, HBaseScanNode#computeStats will be called twice.
> Stacktrace for the first call:
> {code:java}
> at org.apache.impala.planner.HBaseScanNode.computeStats(HBaseScanNode.java:286)
> at org.apache.impala.planner.HBaseScanNode.init(HBaseScanNode.java:160)
> at org.apache.impala.planner.SingleNodePlanner.createScanNode(SingleNodePlanner.java:1405)
> at org.apache.impala.planner.SingleNodePlanner.createTableRefNode(SingleNodePlanner.java:1582)
> at org.apache.impala.planner.SingleNodePlanner.createTableRefsPlan(SingleNodePlanner.java:826)
> at org.apache.impala.planner.SingleNodePlanner.createSelectPlan(SingleNodePlanner.java:662)
> at org.apache.impala.planner.SingleNodePlanner.createQueryPlan(SingleNodePlanner.java:261)
> at org.apache.impala.planner.SingleNodePlanner.createSingleNodePlan(SingleNodePlanner.java:151)
> at org.apache.impala.planner.Planner.createPlan(Planner.java:117)
> at org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1169)
> at org.apache.impala.service.Frontend.getPlannedExecRequest(Frontend.java:1495)
> at org.apache.impala.service.Frontend.doCreateExecRequest(Frontend.java:1359)
> at org.apache.impala.service.Frontend.getTExecRequest(Frontend.java:1250)
> at org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1220)
> {code}
> Stacktrace for the second call:
> {code:java}
> at org.apache.impala.planner.HBaseScanNode.computeStats(HBaseScanNode.java:286)
> at org.apache.impala.planner.SingleNodePlanner.createQueryPlan(SingleNodePlanner.java:307)
> at org.apache.impala.planner.SingleNodePlanner.createSingleNodePlan(SingleNodePlanner.java:151)
> at org.apache.impala.planner.Planner.createPlan(Planner.java:117)
> at org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1169)
> at org.apache.impala.service.Frontend.getPlannedExecRequest(Frontend.java:1495)
> at org.apache.impala.service.Frontend.doCreateExecRequest(Frontend.java:1359)
> at org.apache.impala.service.Frontend.getTExecRequest(Frontend.java:1250)
> at org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1220)
> at org.apache.impala.service.JniFrontend.createExecRequest(JniFrontend.java:154)
> {code}
> Codes of the second call:
> {code:java}
> private PlanNode createQueryPlan(QueryStmt stmt, Analyzer analyzer, boolean disableTopN)
> throws ImpalaException {
> ......
> if (stmt.evaluateOrderBy() && sortHasMaterializedSlots) {
> root = createSortNode(analyzer, root, stmt.getSortInfo(), stmt.getLimit(),
> stmt.getOffset(), stmt.hasLimit(), disableTopN);
> } else {
> root.setLimit(stmt.getLimit());
> root.computeStats(analyzer); // <--- May call HBaseScanNode#computeStats here
> }
> return root;
> }
> {code}
> Logs for a simple query on an old version of Impala:
> {code:java}
> I0830 11:52:05.991547 41189 Analyzer.java:1578] new pred: stg.xxx_hbase.key >= 'key1' BinaryPredicate{op=>=, SlotRef{path=key, type=STRING, id=0} StringLiteral{value=key1}}
> I0830 11:52:05.991595 41189 Analyzer.java:1578] new pred: stg.xxx_hbase.key <= 'key2' BinaryPredicate{op=<=, SlotRef{path=key, type=STRING, id=0} StringLiteral{value=key2}}
> # <--------- 2 seconds here
> I0830 11:52:08.114225 41189 HBaseScanNode.java:217] computeStats HbaseScan: cardinality=1706076
> I0830 11:52:08.114341 41189 HBaseScanNode.java:223] computeStats HbaseScan: #nodes=100
> I0830 11:52:08.114452 41189 SingleNodePlanner.java:357] createCheapestJoinPlan
> # <--------- 2 seconds here
> I0830 11:52:10.260190 41189 HBaseScanNode.java:217] computeStats HbaseScan: cardinality=1706076
> I0830 11:52:10.260303 41189 HBaseScanNode.java:223] computeStats HbaseScan: #nodes=100
> I0830 11:52:10.260387 41189 SingleNodePlanner.java:357] createCheapestJoinPlan
> {code}
> Such kind of queries are usually point queries and are always expected to return fast. HBaseScanNode#computeStats is heavy since it requires RPCs to HBase. We should avoid calling it twice.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org