You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Riza Suminto (Jira)" <ji...@apache.org> on 2023/06/02 01:59:00 UTC
[jira] [Commented] (IMPALA-11842) Improve memory estimation for streaming aggregate operator
[ https://issues.apache.org/jira/browse/IMPALA-11842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17728528#comment-17728528 ]
Riza Suminto commented on IMPALA-11842:
---------------------------------------
For specific example in this JIRA, this seems to be a case where backend code prevent preagg from growing its memory usage due to unmet reduction factor criteria.
> Improve memory estimation for streaming aggregate operator
> ----------------------------------------------------------
>
> Key: IMPALA-11842
> URL: https://issues.apache.org/jira/browse/IMPALA-11842
> Project: IMPALA
> Issue Type: Bug
> Reporter: Abhishek Rawat
> Priority: Critical
>
> Streaming aggregate operator can over estimate the peak memory and as a result Impala could request max memory allowed by admission controller. This impacts query concurrency and also causes unnecessary scaling.
> Looking at some cases in the following profile snippet, the estimated peak memory (8.15 GB) is *20X* from actual peak memory (416.07 MB).
> {code:java}
> Estimated Per-Host Mem: 247067273376
> Request Pool: root.default
> Per Host Min Memory Reservation: impala-executor-003-5:27010(1.34 GB) impala-executor-003-4:27010(1.35 GB) impala-executor-003-6:27010(1.34 GB) impala-executor-003-0:27010(1.35 GB) impala-executor-003-8:27010(1.35 GB) impala-executor-003-2:27010(1.34 GB) impala-executor-003-1:27010(1.35 GB) coordinator-0.coordinator-int.impala-dylan-impala.svc.cluster.local:27000(4.00 MB) impala-executor-003-3:27010(1.34 GB) impala-executor-003-9:27010(1.34 GB) impala-executor-003-7:27010(1.35 GB)
> Per Host Number of Fragment Instances: impala-executor-003-5:27010(37) impala-executor-003-4:27010(38) impala-executor-003-6:27010(37) impala-executor-003-0:27010(38) impala-executor-003-8:27010(38) impala-executor-003-2:27010(37) impala-executor-003-1:27010(38) coordinator-0.coordinator-int.impala-dylan-impala.svc.cluster.local:27000(1) impala-executor-003-3:27010(37) impala-executor-003-9:27010(37) impala-executor-003-7:27010(38)
> Latest admission queue reason: Not enough memory available on host impala-executor-003-5:27010. Needed 50.00 GB but only 33.06 GB out of 83.06 GB was available.
> Admission result: Admitted (queued)
> Initial admission queue reason: waited 83020 ms, reason: Not enough memory available on host impala-executor-003-5:27010. Needed 50.00 GB but only 33.06 GB out of 83.06 GB was available.
> Cluster Memory Admitted: 500.10 GB
> Executor Group: root.default-group-002
> ExecSummary:
> Operator #Hosts #Inst Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
> -----------------------------------------------------------------------------------------------------------------------------------------
> F04:ROOT 1 1 146.192us 146.192us 4.01 MB 4.00 MB
> 11:MERGING-EXCHANGE 1 1 3.297ms 3.297ms 100 100 1.88 MB 234.53 KB UNPARTITIONED
> F03:EXCHANGE SENDER 10 120 4.196ms 374.168ms 7.52 KB 0
> 05:TOP-N 10 120 22.184ms 1s028ms 12.00K 100 16.00 KB 1.56 KB
> 10:AGGREGATE 10 120 4m2s 12m14s 499.98K 487.66K 2.34 MB 10.00 MB FINALIZE
> 09:EXCHANGE 10 120 5s336ms 7s799ms 22.11B 487.66K 10.40 MB 3.09 MB HASH(vendor_id)
> F02:EXCHANGE SENDER 10 120 28s199ms 48s974ms 4.16 MB 0
> 04:AGGREGATE 10 120 1m17s 1m34s 22.11B 487.66K 21.02 MB 10.00 MB STREAMING
> 08:AGGREGATE 10 120 12m29s 22m36s 50.00B 50.00B 3.85 GB 10.87 GB
> 07:EXCHANGE 10 120 10s165ms 12s246ms 50.00B 50.00B 11.69 MB 12.34 MB HASH(vendor_id,purchase_id)
> F00:EXCHANGE SENDER 10 120 1m28s 1m49s 4.16 MB 0
> 03:AGGREGATE 10 120 2m34s 5m5s 50.00B 50.00B 416.07 MB 8.15 GB STREAMING
> 02:HASH JOIN 10 120 1m17s 1m40s 50.00B 50.00B 46.12 KB 0 INNER JOIN, BROADCAST
> |--F05:JOIN BUILD 10 10 557.800ms 613.307ms 408.02 MB 408.00 MB
> | 06:EXCHANGE 10 10 88.770ms 102.048ms 5.00M 5.00M 16.11 MB 10.10 MB BROADCAST
> | F01:EXCHANGE SENDER 5 5 190.673ms 212.904ms 75.23 KB 0
> | 00:SCAN HDFS 5 5 774.428ms 965.865ms 5.00M 5.00M 12.92 MB 64.00 MB tab.product
> 01:SCAN HDFS 10 120 5m48s 13m14s 50.00B 50.00B 32.92 MB 88.00 MB tab.pli {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org