You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by cs...@apache.org on 2022/02/23 22:58:03 UTC
[impala] 02/02: IMPALA-11150: Remove resource-requirements tests on functional_parquet.alltypes
This is an automated email from the ASF dual-hosted git repository.
csringhofer pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 7cfc73c2fd94335d7a19b6767b016f62ec4bde84
Author: Csaba Ringhofer <cs...@cloudera.com>
AuthorDate: Wed Feb 23 21:10:41 2022 +0100
IMPALA-11150: Remove resource-requirements tests on functional_parquet.alltypes
These test became flaky after IMPALA-10961 as it led to smaller and
varying size for the table. This is a short term solution to make
builds green as fixing the tests properly may take some time.
Change-Id: I5bf0f963d3e053345aec27e834974eeead4190ac
Reviewed-on: http://gerrit.cloudera.org:8080/18267
Reviewed-by: Fang-Yu Rao <fa...@cloudera.com>
Reviewed-by: Csaba Ringhofer <cs...@cloudera.com>
Tested-by: Csaba Ringhofer <cs...@cloudera.com>
---
.../queries/PlannerTest/resource-requirements.test | 263 ---------------------
1 file changed, 263 deletions(-)
diff --git a/testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test b/testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test
index 6702f84..ec5008d 100644
--- a/testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test
+++ b/testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test
@@ -153,269 +153,6 @@ Per-Instance Resources: mem-estimate=80.00MB mem-reservation=4.00MB thread-reser
tuple-ids=0 row-size=38B cardinality=6.00M
in pipelines: 00(GETNEXT)
====
-# Small parquet files - memory reservation is reduced because of small file size.
-select string_col from functional_parquet.alltypes;
----- PLAN
-Max Per-Host Resource Reservation: Memory=4.02MB Threads=2
-Per-Host Resource Estimates: Memory=26MB
-WARNING: The following tables are missing relevant table and/or column statistics.
-functional_parquet.alltypes
-Analyzed query: SELECT string_col FROM functional_parquet.alltypes
-
-F00:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
-| Per-Host Resources: mem-estimate=26.00MB mem-reservation=4.02MB thread-reservation=2
-PLAN-ROOT SINK
-| output exprs: string_col
-| mem-estimate=10.00MB mem-reservation=4.00MB spill-buffer=2.00MB thread-reservation=0
-|
-00:SCAN HDFS [functional_parquet.alltypes]
- HDFS partitions=24/24 files=24 size=201.80KB
- stored statistics:
- table: rows=unavailable size=unavailable
- partitions: 0/24 rows=unavailable
- columns: unavailable
- extrapolated-rows=disabled max-scan-range-rows=unavailable
- mem-estimate=16.00MB mem-reservation=16.00KB thread-reservation=1
- tuple-ids=0 row-size=12B cardinality=unavailable
- in pipelines: 00(GETNEXT)
----- DISTRIBUTEDPLAN
-Max Per-Host Resource Reservation: Memory=4.02MB Threads=3
-Per-Host Resource Estimates: Memory=26MB
-WARNING: The following tables are missing relevant table and/or column statistics.
-functional_parquet.alltypes
-Analyzed query: SELECT string_col FROM functional_parquet.alltypes
-
-F01:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
-| Per-Host Resources: mem-estimate=10.05MB mem-reservation=4.00MB thread-reservation=1
-PLAN-ROOT SINK
-| output exprs: string_col
-| mem-estimate=10.00MB mem-reservation=4.00MB spill-buffer=2.00MB thread-reservation=0
-|
-01:EXCHANGE [UNPARTITIONED]
-| mem-estimate=48.00KB mem-reservation=0B thread-reservation=0
-| tuple-ids=0 row-size=12B cardinality=unavailable
-| in pipelines: 00(GETNEXT)
-|
-F00:PLAN FRAGMENT [RANDOM] hosts=3 instances=3
-Per-Host Resources: mem-estimate=16.00MB mem-reservation=16.00KB thread-reservation=2
-00:SCAN HDFS [functional_parquet.alltypes, RANDOM]
- HDFS partitions=24/24 files=24 size=201.80KB
- stored statistics:
- table: rows=unavailable size=unavailable
- partitions: 0/24 rows=unavailable
- columns: unavailable
- extrapolated-rows=disabled max-scan-range-rows=unavailable
- mem-estimate=16.00MB mem-reservation=16.00KB thread-reservation=1
- tuple-ids=0 row-size=12B cardinality=unavailable
- in pipelines: 00(GETNEXT)
----- PARALLELPLANS
-Max Per-Host Resource Reservation: Memory=4.03MB Threads=3
-Per-Host Resource Estimates: Memory=42MB
-WARNING: The following tables are missing relevant table and/or column statistics.
-functional_parquet.alltypes
-Analyzed query: SELECT string_col FROM functional_parquet.alltypes
-
-F01:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
-| Per-Instance Resources: mem-estimate=10.09MB mem-reservation=4.00MB thread-reservation=1
-PLAN-ROOT SINK
-| output exprs: string_col
-| mem-estimate=10.00MB mem-reservation=4.00MB spill-buffer=2.00MB thread-reservation=0
-|
-01:EXCHANGE [UNPARTITIONED]
-| mem-estimate=96.00KB mem-reservation=0B thread-reservation=0
-| tuple-ids=0 row-size=12B cardinality=unavailable
-| in pipelines: 00(GETNEXT)
-|
-F00:PLAN FRAGMENT [RANDOM] hosts=3 instances=6
-Per-Instance Resources: mem-estimate=16.00MB mem-reservation=16.00KB thread-reservation=1
-00:SCAN HDFS [functional_parquet.alltypes, RANDOM]
- HDFS partitions=24/24 files=24 size=201.80KB
- stored statistics:
- table: rows=unavailable size=unavailable
- partitions: 0/24 rows=unavailable
- columns: unavailable
- extrapolated-rows=disabled max-scan-range-rows=unavailable
- mem-estimate=16.00MB mem-reservation=16.00KB thread-reservation=0
- tuple-ids=0 row-size=12B cardinality=unavailable
- in pipelines: 00(GETNEXT)
-====
-# Multi-column parquet scan with small files - memory reservation is reduced because of
-# small file size but a minimum amount is reserved per column.
-select int_col, float_col, string_col from functional_parquet.alltypes;
----- PLAN
-Max Per-Host Resource Reservation: Memory=4.02MB Threads=2
-Per-Host Resource Estimates: Memory=26MB
-WARNING: The following tables are missing relevant table and/or column statistics.
-functional_parquet.alltypes
-Analyzed query: SELECT int_col, float_col, string_col FROM
-functional_parquet.alltypes
-
-F00:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
-| Per-Host Resources: mem-estimate=26.00MB mem-reservation=4.02MB thread-reservation=2
-PLAN-ROOT SINK
-| output exprs: int_col, float_col, string_col
-| mem-estimate=10.00MB mem-reservation=4.00MB spill-buffer=2.00MB thread-reservation=0
-|
-00:SCAN HDFS [functional_parquet.alltypes]
- HDFS partitions=24/24 files=24 size=201.80KB
- stored statistics:
- table: rows=unavailable size=unavailable
- partitions: 0/24 rows=unavailable
- columns: unavailable
- extrapolated-rows=disabled max-scan-range-rows=unavailable
- mem-estimate=16.00MB mem-reservation=24.00KB thread-reservation=1
- tuple-ids=0 row-size=20B cardinality=unavailable
- in pipelines: 00(GETNEXT)
----- DISTRIBUTEDPLAN
-Max Per-Host Resource Reservation: Memory=4.02MB Threads=3
-Per-Host Resource Estimates: Memory=26MB
-WARNING: The following tables are missing relevant table and/or column statistics.
-functional_parquet.alltypes
-Analyzed query: SELECT int_col, float_col, string_col FROM
-functional_parquet.alltypes
-
-F01:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
-| Per-Host Resources: mem-estimate=10.07MB mem-reservation=4.00MB thread-reservation=1
-PLAN-ROOT SINK
-| output exprs: int_col, float_col, string_col
-| mem-estimate=10.00MB mem-reservation=4.00MB spill-buffer=2.00MB thread-reservation=0
-|
-01:EXCHANGE [UNPARTITIONED]
-| mem-estimate=71.99KB mem-reservation=0B thread-reservation=0
-| tuple-ids=0 row-size=20B cardinality=unavailable
-| in pipelines: 00(GETNEXT)
-|
-F00:PLAN FRAGMENT [RANDOM] hosts=3 instances=3
-Per-Host Resources: mem-estimate=16.00MB mem-reservation=24.00KB thread-reservation=2
-00:SCAN HDFS [functional_parquet.alltypes, RANDOM]
- HDFS partitions=24/24 files=24 size=201.80KB
- stored statistics:
- table: rows=unavailable size=unavailable
- partitions: 0/24 rows=unavailable
- columns: unavailable
- extrapolated-rows=disabled max-scan-range-rows=unavailable
- mem-estimate=16.00MB mem-reservation=24.00KB thread-reservation=1
- tuple-ids=0 row-size=20B cardinality=unavailable
- in pipelines: 00(GETNEXT)
----- PARALLELPLANS
-Max Per-Host Resource Reservation: Memory=4.05MB Threads=3
-Per-Host Resource Estimates: Memory=42MB
-WARNING: The following tables are missing relevant table and/or column statistics.
-functional_parquet.alltypes
-Analyzed query: SELECT int_col, float_col, string_col FROM
-functional_parquet.alltypes
-
-F01:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
-| Per-Instance Resources: mem-estimate=10.14MB mem-reservation=4.00MB thread-reservation=1
-PLAN-ROOT SINK
-| output exprs: int_col, float_col, string_col
-| mem-estimate=10.00MB mem-reservation=4.00MB spill-buffer=2.00MB thread-reservation=0
-|
-01:EXCHANGE [UNPARTITIONED]
-| mem-estimate=143.99KB mem-reservation=0B thread-reservation=0
-| tuple-ids=0 row-size=20B cardinality=unavailable
-| in pipelines: 00(GETNEXT)
-|
-F00:PLAN FRAGMENT [RANDOM] hosts=3 instances=6
-Per-Instance Resources: mem-estimate=16.00MB mem-reservation=24.00KB thread-reservation=1
-00:SCAN HDFS [functional_parquet.alltypes, RANDOM]
- HDFS partitions=24/24 files=24 size=201.80KB
- stored statistics:
- table: rows=unavailable size=unavailable
- partitions: 0/24 rows=unavailable
- columns: unavailable
- extrapolated-rows=disabled max-scan-range-rows=unavailable
- mem-estimate=16.00MB mem-reservation=24.00KB thread-reservation=0
- tuple-ids=0 row-size=20B cardinality=unavailable
- in pipelines: 00(GETNEXT)
-====
-# Parquet scan with no materialized columns. Need reservation to scan levels to determine
-# row count.
-select 'foo' from functional_parquet.alltypes
----- PLAN
-Max Per-Host Resource Reservation: Memory=4.02MB Threads=2
-Per-Host Resource Estimates: Memory=11MB
-WARNING: The following tables are missing relevant table and/or column statistics.
-functional_parquet.alltypes
-Analyzed query: SELECT 'foo' FROM functional_parquet.alltypes
-
-F00:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
-| Per-Host Resources: mem-estimate=11.00MB mem-reservation=4.02MB thread-reservation=2
-PLAN-ROOT SINK
-| output exprs: 'foo'
-| mem-estimate=10.00MB mem-reservation=4.00MB spill-buffer=2.00MB thread-reservation=0
-|
-00:SCAN HDFS [functional_parquet.alltypes]
- HDFS partitions=24/24 files=24 size=201.80KB
- stored statistics:
- table: rows=unavailable size=unavailable
- partitions: 0/24 rows=unavailable
- columns: all
- extrapolated-rows=disabled max-scan-range-rows=unavailable
- mem-estimate=1.00MB mem-reservation=16.00KB thread-reservation=1
- tuple-ids=0 row-size=0B cardinality=unavailable
- in pipelines: 00(GETNEXT)
----- DISTRIBUTEDPLAN
-Max Per-Host Resource Reservation: Memory=4.02MB Threads=3
-Per-Host Resource Estimates: Memory=11MB
-WARNING: The following tables are missing relevant table and/or column statistics.
-functional_parquet.alltypes
-Analyzed query: SELECT 'foo' FROM functional_parquet.alltypes
-
-F01:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
-| Per-Host Resources: mem-estimate=10.02MB mem-reservation=4.00MB thread-reservation=1
-PLAN-ROOT SINK
-| output exprs: 'foo'
-| mem-estimate=10.00MB mem-reservation=4.00MB spill-buffer=2.00MB thread-reservation=0
-|
-01:EXCHANGE [UNPARTITIONED]
-| mem-estimate=16.00KB mem-reservation=0B thread-reservation=0
-| tuple-ids=0 row-size=0B cardinality=unavailable
-| in pipelines: 00(GETNEXT)
-|
-F00:PLAN FRAGMENT [RANDOM] hosts=3 instances=3
-Per-Host Resources: mem-estimate=1.00MB mem-reservation=16.00KB thread-reservation=2
-00:SCAN HDFS [functional_parquet.alltypes, RANDOM]
- HDFS partitions=24/24 files=24 size=201.80KB
- stored statistics:
- table: rows=unavailable size=unavailable
- partitions: 0/24 rows=unavailable
- columns: all
- extrapolated-rows=disabled max-scan-range-rows=unavailable
- mem-estimate=1.00MB mem-reservation=16.00KB thread-reservation=1
- tuple-ids=0 row-size=0B cardinality=unavailable
- in pipelines: 00(GETNEXT)
----- PARALLELPLANS
-Max Per-Host Resource Reservation: Memory=4.03MB Threads=3
-Per-Host Resource Estimates: Memory=42MB
-WARNING: The following tables are missing relevant table and/or column statistics.
-functional_parquet.alltypes
-Analyzed query: SELECT 'foo' FROM functional_parquet.alltypes
-
-F01:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
-| Per-Instance Resources: mem-estimate=10.02MB mem-reservation=4.00MB thread-reservation=1
-PLAN-ROOT SINK
-| output exprs: 'foo'
-| mem-estimate=10.00MB mem-reservation=4.00MB spill-buffer=2.00MB thread-reservation=0
-|
-01:EXCHANGE [UNPARTITIONED]
-| mem-estimate=24.00KB mem-reservation=0B thread-reservation=0
-| tuple-ids=0 row-size=0B cardinality=unavailable
-| in pipelines: 00(GETNEXT)
-|
-F00:PLAN FRAGMENT [RANDOM] hosts=3 instances=6
-Per-Instance Resources: mem-estimate=16.00MB mem-reservation=16.00KB thread-reservation=1
-00:SCAN HDFS [functional_parquet.alltypes, RANDOM]
- HDFS partitions=24/24 files=24 size=201.80KB
- stored statistics:
- table: rows=unavailable size=unavailable
- partitions: 0/24 rows=unavailable
- columns: all
- extrapolated-rows=disabled max-scan-range-rows=unavailable
- mem-estimate=16.00MB mem-reservation=16.00KB thread-reservation=0
- tuple-ids=0 row-size=0B cardinality=unavailable
- in pipelines: 00(GETNEXT)
-====
# Parquet scan - reservation estimate based on dictionary-encoded size reduces
# reservation for column with low NDV.
select c_nationkey from tpch_parquet.customer