You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Abhishek Rawat (Jira)" <ji...@apache.org> on 2023/01/12 19:03:00 UTC
[jira] [Updated] (IMPALA-11842) Improve memory estimation for streaming aggregate operator
[ https://issues.apache.org/jira/browse/IMPALA-11842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Abhishek Rawat updated IMPALA-11842:
------------------------------------
Description:
Streaming aggregate operator can over estimate the peak memory and as a result Impala could request max memory allowed by admission controller. This impacts query concurrency and also causes unnecessary scaling.
Looking at some cases in the following profile snippet, the estimated peak memory (8.15 GB) is *20X* from actual peak memory (416.07 MB).
{code:java}
Estimated Per-Host Mem: 247067273376
Request Pool: root.default
Per Host Min Memory Reservation: impala-executor-003-5:27010(1.34 GB) impala-executor-003-4:27010(1.35 GB) impala-executor-003-6:27010(1.34 GB) impala-executor-003-0:27010(1.35 GB) impala-executor-003-8:27010(1.35 GB) impala-executor-003-2:27010(1.34 GB) impala-executor-003-1:27010(1.35 GB) coordinator-0.coordinator-int.impala-dylan-impala.svc.cluster.local:27000(4.00 MB) impala-executor-003-3:27010(1.34 GB) impala-executor-003-9:27010(1.34 GB) impala-executor-003-7:27010(1.35 GB)
Per Host Number of Fragment Instances: impala-executor-003-5:27010(37) impala-executor-003-4:27010(38) impala-executor-003-6:27010(37) impala-executor-003-0:27010(38) impala-executor-003-8:27010(38) impala-executor-003-2:27010(37) impala-executor-003-1:27010(38) coordinator-0.coordinator-int.impala-dylan-impala.svc.cluster.local:27000(1) impala-executor-003-3:27010(37) impala-executor-003-9:27010(37) impala-executor-003-7:27010(38)
Latest admission queue reason: Not enough memory available on host impala-executor-003-5:27010. Needed 50.00 GB but only 33.06 GB out of 83.06 GB was available.
Admission result: Admitted (queued)
Initial admission queue reason: waited 83020 ms, reason: Not enough memory available on host impala-executor-003-5:27010. Needed 50.00 GB but only 33.06 GB out of 83.06 GB was available.
Cluster Memory Admitted: 500.10 GB
Executor Group: root.default-group-002
ExecSummary:
Operator #Hosts #Inst Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
-----------------------------------------------------------------------------------------------------------------------------------------
F04:ROOT 1 1 146.192us 146.192us 4.01 MB 4.00 MB
11:MERGING-EXCHANGE 1 1 3.297ms 3.297ms 100 100 1.88 MB 234.53 KB UNPARTITIONED
F03:EXCHANGE SENDER 10 120 4.196ms 374.168ms 7.52 KB 0
05:TOP-N 10 120 22.184ms 1s028ms 12.00K 100 16.00 KB 1.56 KB
10:AGGREGATE 10 120 4m2s 12m14s 499.98K 487.66K 2.34 MB 10.00 MB FINALIZE
09:EXCHANGE 10 120 5s336ms 7s799ms 22.11B 487.66K 10.40 MB 3.09 MB HASH(vendor_id)
F02:EXCHANGE SENDER 10 120 28s199ms 48s974ms 4.16 MB 0
04:AGGREGATE 10 120 1m17s 1m34s 22.11B 487.66K 21.02 MB 10.00 MB STREAMING
08:AGGREGATE 10 120 12m29s 22m36s 50.00B 50.00B 3.85 GB 10.87 GB
07:EXCHANGE 10 120 10s165ms 12s246ms 50.00B 50.00B 11.69 MB 12.34 MB HASH(vendor_id,purchase_id)
F00:EXCHANGE SENDER 10 120 1m28s 1m49s 4.16 MB 0
03:AGGREGATE 10 120 2m34s 5m5s 50.00B 50.00B 416.07 MB 8.15 GB STREAMING
02:HASH JOIN 10 120 1m17s 1m40s 50.00B 50.00B 46.12 KB 0 INNER JOIN, BROADCAST
|--F05:JOIN BUILD 10 10 557.800ms 613.307ms 408.02 MB 408.00 MB
| 06:EXCHANGE 10 10 88.770ms 102.048ms 5.00M 5.00M 16.11 MB 10.10 MB BROADCAST
| F01:EXCHANGE SENDER 5 5 190.673ms 212.904ms 75.23 KB 0
| 00:SCAN HDFS 5 5 774.428ms 965.865ms 5.00M 5.00M 12.92 MB 64.00 MB tab.product
01:SCAN HDFS 10 120 5m48s 13m14s 50.00B 50.00B 32.92 MB 88.00 MB tab.pli {code}
was:
Streaming aggregate operator can over estimate the peak memory and as a result Impala could request max memory allowed by admission controller. This impacts query concurrency and also causes unnecessary scaling.
Looking at some cases in the following profile snippet, the estimated peak memory (8.15 GB) is *20X* from actual peak memory (416.07 MB). I'll file a JIRA.
{code:java}
Estimated Per-Host Mem: 247067273376
Request Pool: root.default
Per Host Min Memory Reservation: impala-executor-003-5:27010(1.34 GB) impala-executor-003-4:27010(1.35 GB) impala-executor-003-6:27010(1.34 GB) impala-executor-003-0:27010(1.35 GB) impala-executor-003-8:27010(1.35 GB) impala-executor-003-2:27010(1.34 GB) impala-executor-003-1:27010(1.35 GB) coordinator-0.coordinator-int.impala-dylan-impala.svc.cluster.local:27000(4.00 MB) impala-executor-003-3:27010(1.34 GB) impala-executor-003-9:27010(1.34 GB) impala-executor-003-7:27010(1.35 GB)
Per Host Number of Fragment Instances: impala-executor-003-5:27010(37) impala-executor-003-4:27010(38) impala-executor-003-6:27010(37) impala-executor-003-0:27010(38) impala-executor-003-8:27010(38) impala-executor-003-2:27010(37) impala-executor-003-1:27010(38) coordinator-0.coordinator-int.impala-dylan-impala.svc.cluster.local:27000(1) impala-executor-003-3:27010(37) impala-executor-003-9:27010(37) impala-executor-003-7:27010(38)
Latest admission queue reason: Not enough memory available on host impala-executor-003-5:27010. Needed 50.00 GB but only 33.06 GB out of 83.06 GB was available.
Admission result: Admitted (queued)
Initial admission queue reason: waited 83020 ms, reason: Not enough memory available on host impala-executor-003-5:27010. Needed 50.00 GB but only 33.06 GB out of 83.06 GB was available.
Cluster Memory Admitted: 500.10 GB
Executor Group: root.default-group-002
ExecSummary:
Operator #Hosts #Inst Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
-----------------------------------------------------------------------------------------------------------------------------------------
F04:ROOT 1 1 146.192us 146.192us 4.01 MB 4.00 MB
11:MERGING-EXCHANGE 1 1 3.297ms 3.297ms 100 100 1.88 MB 234.53 KB UNPARTITIONED
F03:EXCHANGE SENDER 10 120 4.196ms 374.168ms 7.52 KB 0
05:TOP-N 10 120 22.184ms 1s028ms 12.00K 100 16.00 KB 1.56 KB
10:AGGREGATE 10 120 4m2s 12m14s 499.98K 487.66K 2.34 MB 10.00 MB FINALIZE
09:EXCHANGE 10 120 5s336ms 7s799ms 22.11B 487.66K 10.40 MB 3.09 MB HASH(vendor_id)
F02:EXCHANGE SENDER 10 120 28s199ms 48s974ms 4.16 MB 0
04:AGGREGATE 10 120 1m17s 1m34s 22.11B 487.66K 21.02 MB 10.00 MB STREAMING
08:AGGREGATE 10 120 12m29s 22m36s 50.00B 50.00B 3.85 GB 10.87 GB
07:EXCHANGE 10 120 10s165ms 12s246ms 50.00B 50.00B 11.69 MB 12.34 MB HASH(vendor_id,purchase_id)
F00:EXCHANGE SENDER 10 120 1m28s 1m49s 4.16 MB 0
03:AGGREGATE 10 120 2m34s 5m5s 50.00B 50.00B 416.07 MB 8.15 GB STREAMING
02:HASH JOIN 10 120 1m17s 1m40s 50.00B 50.00B 46.12 KB 0 INNER JOIN, BROADCAST
|--F05:JOIN BUILD 10 10 557.800ms 613.307ms 408.02 MB 408.00 MB
| 06:EXCHANGE 10 10 88.770ms 102.048ms 5.00M 5.00M 16.11 MB 10.10 MB BROADCAST
| F01:EXCHANGE SENDER 5 5 190.673ms 212.904ms 75.23 KB 0
| 00:SCAN HDFS 5 5 774.428ms 965.865ms 5.00M 5.00M 12.92 MB 64.00 MB tab.product
01:SCAN HDFS 10 120 5m48s 13m14s 50.00B 50.00B 32.92 MB 88.00 MB tab.pli {code}
> Improve memory estimation for streaming aggregate operator
> ----------------------------------------------------------
>
> Key: IMPALA-11842
> URL: https://issues.apache.org/jira/browse/IMPALA-11842
> Project: IMPALA
> Issue Type: Bug
> Reporter: Abhishek Rawat
> Priority: Major
>
> Streaming aggregate operator can over estimate the peak memory and as a result Impala could request max memory allowed by admission controller. This impacts query concurrency and also causes unnecessary scaling.
> Looking at some cases in the following profile snippet, the estimated peak memory (8.15 GB) is *20X* from actual peak memory (416.07 MB).
> {code:java}
> Estimated Per-Host Mem: 247067273376
> Request Pool: root.default
> Per Host Min Memory Reservation: impala-executor-003-5:27010(1.34 GB) impala-executor-003-4:27010(1.35 GB) impala-executor-003-6:27010(1.34 GB) impala-executor-003-0:27010(1.35 GB) impala-executor-003-8:27010(1.35 GB) impala-executor-003-2:27010(1.34 GB) impala-executor-003-1:27010(1.35 GB) coordinator-0.coordinator-int.impala-dylan-impala.svc.cluster.local:27000(4.00 MB) impala-executor-003-3:27010(1.34 GB) impala-executor-003-9:27010(1.34 GB) impala-executor-003-7:27010(1.35 GB)
> Per Host Number of Fragment Instances: impala-executor-003-5:27010(37) impala-executor-003-4:27010(38) impala-executor-003-6:27010(37) impala-executor-003-0:27010(38) impala-executor-003-8:27010(38) impala-executor-003-2:27010(37) impala-executor-003-1:27010(38) coordinator-0.coordinator-int.impala-dylan-impala.svc.cluster.local:27000(1) impala-executor-003-3:27010(37) impala-executor-003-9:27010(37) impala-executor-003-7:27010(38)
> Latest admission queue reason: Not enough memory available on host impala-executor-003-5:27010. Needed 50.00 GB but only 33.06 GB out of 83.06 GB was available.
> Admission result: Admitted (queued)
> Initial admission queue reason: waited 83020 ms, reason: Not enough memory available on host impala-executor-003-5:27010. Needed 50.00 GB but only 33.06 GB out of 83.06 GB was available.
> Cluster Memory Admitted: 500.10 GB
> Executor Group: root.default-group-002
> ExecSummary:
> Operator #Hosts #Inst Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
> -----------------------------------------------------------------------------------------------------------------------------------------
> F04:ROOT 1 1 146.192us 146.192us 4.01 MB 4.00 MB
> 11:MERGING-EXCHANGE 1 1 3.297ms 3.297ms 100 100 1.88 MB 234.53 KB UNPARTITIONED
> F03:EXCHANGE SENDER 10 120 4.196ms 374.168ms 7.52 KB 0
> 05:TOP-N 10 120 22.184ms 1s028ms 12.00K 100 16.00 KB 1.56 KB
> 10:AGGREGATE 10 120 4m2s 12m14s 499.98K 487.66K 2.34 MB 10.00 MB FINALIZE
> 09:EXCHANGE 10 120 5s336ms 7s799ms 22.11B 487.66K 10.40 MB 3.09 MB HASH(vendor_id)
> F02:EXCHANGE SENDER 10 120 28s199ms 48s974ms 4.16 MB 0
> 04:AGGREGATE 10 120 1m17s 1m34s 22.11B 487.66K 21.02 MB 10.00 MB STREAMING
> 08:AGGREGATE 10 120 12m29s 22m36s 50.00B 50.00B 3.85 GB 10.87 GB
> 07:EXCHANGE 10 120 10s165ms 12s246ms 50.00B 50.00B 11.69 MB 12.34 MB HASH(vendor_id,purchase_id)
> F00:EXCHANGE SENDER 10 120 1m28s 1m49s 4.16 MB 0
> 03:AGGREGATE 10 120 2m34s 5m5s 50.00B 50.00B 416.07 MB 8.15 GB STREAMING
> 02:HASH JOIN 10 120 1m17s 1m40s 50.00B 50.00B 46.12 KB 0 INNER JOIN, BROADCAST
> |--F05:JOIN BUILD 10 10 557.800ms 613.307ms 408.02 MB 408.00 MB
> | 06:EXCHANGE 10 10 88.770ms 102.048ms 5.00M 5.00M 16.11 MB 10.10 MB BROADCAST
> | F01:EXCHANGE SENDER 5 5 190.673ms 212.904ms 75.23 KB 0
> | 00:SCAN HDFS 5 5 774.428ms 965.865ms 5.00M 5.00M 12.92 MB 64.00 MB tab.product
> 01:SCAN HDFS 10 120 5m48s 13m14s 50.00B 50.00B 32.92 MB 88.00 MB tab.pli {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org