You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (JIRA)" <ji...@apache.org> on 2018/06/01 05:59:00 UTC
[jira] [Commented] (IMPALA-6338) Runtime profile for query with
limit may be missing pieces
[ https://issues.apache.org/jira/browse/IMPALA-6338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16497615#comment-16497615 ]
ASF subversion and git services commented on IMPALA-6338:
---------------------------------------------------------
Commit 1826ca3b99e1008a05b616ccd7daf31620a31d32 in impala's branch refs/heads/2.x from [~twmarshall]
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=1826ca3 ]
IMPALA-6338: Disable more flaky bloom filter tests
Until IMPALA-6338 is fixed, temporarily disable tests that are
affected by it - any test that has a 'limit' and relies on the
contents of the runtime profile. This patch disables the runtime
profile check for all such tests in bloom_filter.test
Change-Id: Ifc9da892efa3b27d63056ad8e3befac82808ffdb
Reviewed-on: http://gerrit.cloudera.org:8080/10530
Reviewed-by: Bikramjeet Vig <bi...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
> Runtime profile for query with limit may be missing pieces
> ----------------------------------------------------------
>
> Key: IMPALA-6338
> URL: https://issues.apache.org/jira/browse/IMPALA-6338
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 2.11.0, Impala 2.12.0
> Reporter: David Knupp
> Assignee: Thomas Tauber-Marshall
> Priority: Critical
> Labels: broken-build, flaky
> Attachments: profile-failure.txt, profile-success.txt
>
>
> Stack trace:
> {noformat}
> query_test/test_observability.py:123: in test_profile_fragment_instances
> assert results.runtime_profile.count("HDFS_SCAN_NODE") == 12
> E assert 11 == 12
> E + where 11 = <built-in method count of str object at 0x68bd2f0>('HDFS_SCAN_NODE')
> E + where <built-in method count of str object at 0x68bd2f0> = 'Query (id=ae4cee91aafc5c6c:11b545c600000000):\n DEBUG MODE WARNING: Query profile created while running a DEBUG buil...ontextSwitches: 0 (0)\n - TotalRawHdfsReadTime(*): 5s784ms\n - TotalReadThroughput: 17.33 MB/sec\n'.count
> E + where 'Query (id=ae4cee91aafc5c6c:11b545c600000000):\n DEBUG MODE WARNING: Query profile created while running a DEBUG buil...ontextSwitches: 0 (0)\n - TotalRawHdfsReadTime(*): 5s784ms\n - TotalReadThroughput: 17.33 MB/sec\n' = <tests.beeswax.impala_beeswax.ImpalaBeeswaxResult object at 0x6322e10>.runtime_profile
> {noformat}
> Query:
> {noformat}
> with l as (select * from tpch.lineitem UNION ALL select * from tpch.lineitem)
> select STRAIGHT_JOIN count(*) from (select * from tpch.lineitem a LIMIT 1) a
> join (select * from l LIMIT 2000000) b on a.l_orderkey = -b.l_orderkey;
> {noformat}
> Summary:
> {noformat}
> Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
> --------------------------------------------------------------------------------------------------------------------
> 05:AGGREGATE 1 0.000ns 0.000ns 1 1 28.00 KB 10.00 MB FINALIZE
> 04:HASH JOIN 1 15.000ms 15.000ms 0 1 141.06 MB 17.00 MB INNER JOIN, BROADCAST
> |--08:EXCHANGE 1 4s153ms 4s153ms 2.00M 2.00M 0 0 UNPARTITIONED
> | 07:EXCHANGE 1 3s783ms 3s783ms 2.00M 2.00M 0 0 UNPARTITIONED
> | 01:UNION 3 17.000ms 28.001ms 3.03M 2.00M 0 0
> | |--03:SCAN HDFS 3 0.000ns 0.000ns 0 6.00M 0 176.00 MB tpch.lineitem
> | 02:SCAN HDFS 3 6s133ms 6s948ms 3.03M 6.00M 24.02 MB 176.00 MB tpch.lineitem
> 06:EXCHANGE 1 5s655ms 5s655ms 1 1 0 0 UNPARTITIONED
> 00:SCAN HDFS 3 4s077ms 6s207ms 2 1 16.05 MB 176.00 MB tpch.lineitem a
> {noformat}
> Plan:
> {noformat}
> ----------------
> Max Per-Host Resource Reservation: Memory=17.00MB
> Per-Host Resource Estimates: Memory=379.00MB
> F01:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
> | Per-Host Resources: mem-estimate=27.00MB mem-reservation=17.00MB
> PLAN-ROOT SINK
> | mem-estimate=0B mem-reservation=0B
> |
> 05:AGGREGATE [FINALIZE]
> | output: count(*)
> | mem-estimate=10.00MB mem-reservation=0B spill-buffer=2.00MB
> | tuple-ids=7 row-size=8B cardinality=1
> |
> 04:HASH JOIN [INNER JOIN, BROADCAST]
> | hash predicates: a.l_orderkey = -1 * l_orderkey
> | fk/pk conjuncts: assumed fk/pk
> | runtime filters: RF000[bloom] <- -1 * l_orderkey
> | mem-estimate=17.00MB mem-reservation=17.00MB spill-buffer=1.00MB
> | tuple-ids=0,4 row-size=16B cardinality=1
> |
> |--08:EXCHANGE [UNPARTITIONED]
> | | mem-estimate=0B mem-reservation=0B
> | | tuple-ids=4 row-size=8B cardinality=2000000
> | |
> | F05:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
> | Per-Host Resources: mem-estimate=0B mem-reservation=0B
> | 07:EXCHANGE [UNPARTITIONED]
> | | limit: 2000000
> | | mem-estimate=0B mem-reservation=0B
> | | tuple-ids=4 row-size=8B cardinality=2000000
> | |
> | F04:PLAN FRAGMENT [RANDOM] hosts=3 instances=3
> | Per-Host Resources: mem-estimate=176.00MB mem-reservation=0B
> | 01:UNION
> | | pass-through-operands: all
> | | limit: 2000000
> | | mem-estimate=0B mem-reservation=0B
> | | tuple-ids=4 row-size=8B cardinality=2000000
> | |
> | |--03:SCAN HDFS [tpch.lineitem, RANDOM]
> | | partitions=1/1 files=1 size=718.94MB
> | | stored statistics:
> | | table: rows=6001215 size=718.94MB
> | | columns: all
> | | extrapolated-rows=disabled
> | | mem-estimate=176.00MB mem-reservation=0B
> | | tuple-ids=3 row-size=8B cardinality=6001215
> | |
> | 02:SCAN HDFS [tpch.lineitem, RANDOM]
> | partitions=1/1 files=1 size=718.94MB
> | stored statistics:
> | table: rows=6001215 size=718.94MB
> | columns: all
> | extrapolated-rows=disabled
> | mem-estimate=176.00MB mem-reservation=0B
> | tuple-ids=2 row-size=8B cardinality=6001215
> |
> 06:EXCHANGE [UNPARTITIONED]
> | limit: 1
> | mem-estimate=0B mem-reservation=0B
> | tuple-ids=0 row-size=8B cardinality=1
> |
> F00:PLAN FRAGMENT [RANDOM] hosts=3 instances=3
> Per-Host Resources: mem-estimate=176.00MB mem-reservation=0B
> 00:SCAN HDFS [tpch.lineitem a, RANDOM]
> partitions=1/1 files=1 size=718.94MB
> runtime filters: RF000[bloom] -> a.l_orderkey
> stored statistics:
> table: rows=6001215 size=718.94MB
> columns: all
> extrapolated-rows=disabled
> limit: 1
> mem-estimate=176.00MB mem-reservation=0B
> tuple-ids=0 row-size=8B cardinality=1
> ----------------
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org