You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Qifan Chen (Jira)" <ji...@apache.org> on 2022/02/23 18:02:00 UTC
[jira] [Commented] (IMPALA-11132) Front-end test PlannerTest.testResourceRequirements can fail
[ https://issues.apache.org/jira/browse/IMPALA-11132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17496958#comment-17496958 ]
Qifan Chen commented on IMPALA-11132:
-------------------------------------
The estimation for the # of rows in HBase table scan is not capped by the # of rows from HMS when available.
> Front-end test PlannerTest.testResourceRequirements can fail
> ------------------------------------------------------------
>
> Key: IMPALA-11132
> URL: https://issues.apache.org/jira/browse/IMPALA-11132
> Project: IMPALA
> Issue Type: Test
> Reporter: Qifan Chen
> Assignee: Qifan Chen
> Priority: Major
>
> The test miscalculates per-host memory requirements, apparently due to an incorrect HBase cardinality estimate:
> {code:java}
> Section DISTRIBUTEDPLAN of query:
> select * from functional_hbase.alltypessmall
> Actual does not match expected result:
> Max Per-Host Resource Reservation: Memory=4.00MB Threads=2
> Per-Host Resource Estimates: Memory=10MB
> Codegen disabled by planner
> Analyzed query: SELECT * FROM functional_hbase.alltypessmall
> F01:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
> | Per-Host Resources: mem-estimate=5.08MB mem-reservation=4.00MB thread-reservation=1
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> PLAN-ROOT SINK
> | output exprs: functional_hbase.alltypessmall.id, functional_hbase.alltypessmall.bigint_col, functional_hbase.alltypessmall.bool_col, functional_hbase.alltypessmall.date_string_col, functional_hbase.alltypessmall.double_col, functional_hbase.alltypessmall.float_col, functional_hbase.alltypessmall.int_col, functional_hbase.alltypessmall.month, functional_hbase.alltypessmall.smallint_col, functional_hbase.alltypessmall.string_col, functional_hbase.alltypessmall.timestamp_col, functional_hbase.alltypessmall.tinyint_col, functional_hbase.alltypessmall.year
> | mem-estimate=4.00MB mem-reservation=4.00MB spill-buffer=2.00MB thread-reservation=0
> |
> 01:EXCHANGE [UNPARTITIONED]
> | mem-estimate=1.08MB mem-reservation=0B thread-reservation=0
> | tuple-ids=0 row-size=89B cardinality=28.57K
> | in pipelines: 00(GETNEXT)
> |
> F00:PLAN FRAGMENT [RANDOM] hosts=3 instances=3
> Per-Host Resources: mem-estimate=4.00KB mem-reservation=0B thread-reservation=1
> 00:SCAN HBASE [functional_hbase.alltypessmall]
> stored statistics:
> table: rows=100
> columns: all
> mem-estimate=4.00KB mem-reservation=0B thread-reservation=0
> tuple-ids=0 row-size=89B cardinality=28.57K
> in pipelines: 00(GETNEXT)
> Expected:
> Max Per-Host Resource Reservation: Memory=4.00MB Threads=2
> Per-Host Resource Estimates: Memory=10MB
> Codegen disabled by planner
> Analyzed query: SELECT * FROM functional_hbase.alltypessmall
> F01:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
> | Per-Host Resources: mem-estimate=4.02MB mem-reservation=4.00MB thread-reservation=1
> PLAN-ROOT SINK
> | output exprs: functional_hbase.alltypessmall.id, functional_hbase.alltypessmall.bigint_col, functional_hbase.alltypessmall.bool_col, functional_hbase.alltypessmall.date_string_col, functional_hbase.alltypessmall.double_col, functional_hbase.alltypessmall.float_col, functional_hbase.alltypessmall.int_col, functional_hbase.alltypessmall.month, functional_hbase.alltypessmall.smallint_col, functional_hbase.alltypessmall.string_col, functional_hbase.alltypessmall.timestamp_col, functional_hbase.alltypessmall.tinyint_col, functional_hbase.alltypessmall.year
> | mem-estimate=4.00MB mem-reservation=4.00MB spill-buffer=2.00MB thread-reservation=0
> |
> 01:EXCHANGE [UNPARTITIONED]
> | mem-estimate=16.00KB mem-reservation=0B thread-reservation=0
> | tuple-ids=0 row-size=89B cardinality=50
> | in pipelines: 00(GETNEXT)
> |
> F00:PLAN FRAGMENT [RANDOM] hosts=3 instances=3
> Per-Host Resources: mem-estimate=4.00KB mem-reservation=0B thread-reservation=1
> 00:SCAN HBASE [functional_hbase.alltypessmall]
> stored statistics:
> table: rows=100
> columns: all
> mem-estimate=4.00KB mem-reservation=0B thread-reservation=0
> tuple-ids=0 row-size=89B cardinality=50
> in pipelines: 00(GETNEXT)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org