You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@impala.apache.org by "Mostafa Mokhtar (JIRA)" <ji...@apache.org> on 2017/08/01 00:09:00 UTC
[jira] [Created] (IMPALA-5746) Remote fragments continue to hold
onto memory after stopping the coordinator node Impala service
Mostafa Mokhtar created IMPALA-5746:
---------------------------------------
Summary: Remote fragments continue to hold onto memory after stopping the coordinator node Impala service
Key: IMPALA-5746
URL: https://issues.apache.org/jira/browse/IMPALA-5746
Project: IMPALA
Issue Type: Sub-task
Components: Distributed Exec
Affects Versions: Impala 2.10.0
Reporter: Mostafa Mokhtar
Assignee: Michael Ho
Priority: Blocker
Attachments: remote_fragments_holding_memory.txt
Repro
# Start running queries
# Kill the coordinator node
# On the running Impalad check the memz tab, remote fragments continue to run and hold on to resources
Remote fragments held on to memory +30 minutes after stopping the coordinator service.
Attached thread dump from an Impalad running remote fragments .
Snapshot of memz tab 30 minutes after killing the coordinator
{code}
Process: Limit=201.73 GB Total=5.32 GB Peak=179.36 GB
Free Disk IO Buffers: Total=1.87 GB Peak=1.87 GB
RequestPool=root.default: Total=1.35 GB Peak=178.51 GB
Query(f64169d4bb3c901c:3a21d8ae00000000): Total=2.64 MB Peak=104.73 MB
Fragment f64169d4bb3c901c:3a21d8ae00000051: Total=2.64 MB Peak=2.67 MB
AGGREGATION_NODE (id=15): Total=2.54 MB Peak=2.57 MB
Exprs: Total=30.12 KB Peak=30.12 KB
EXCHANGE_NODE (id=14): Total=0 Peak=0
DataStreamRecvr: Total=0 Peak=12.29 KB
DataStreamSender (dst_id=17): Total=85.31 KB Peak=85.31 KB
CodeGen: Total=1.53 KB Peak=374.50 KB
Block Manager: Limit=161.39 GB Total=512.00 KB Peak=1.54 MB
Query(2a4f12b3b4b1dc8c:db7e8cf200000000): Total=258.29 MB Peak=412.98 MB
Fragment 2a4f12b3b4b1dc8c:db7e8cf20000008c: Total=2.29 MB Peak=2.29 MB
SORT_NODE (id=11): Total=4.00 KB Peak=4.00 KB
AGGREGATION_NODE (id=20): Total=2.27 MB Peak=2.27 MB
Exprs: Total=25.12 KB Peak=25.12 KB
EXCHANGE_NODE (id=19): Total=0 Peak=0
DataStreamRecvr: Total=0 Peak=0
DataStreamSender (dst_id=21): Total=3.88 KB Peak=3.88 KB
CodeGen: Total=4.17 KB Peak=1.05 MB
Block Manager: Limit=161.39 GB Total=256.25 MB Peak=321.66 MB
Query(68421d2a5dea0775:83f5d97200000000): Total=282.77 MB Peak=443.53 MB
Fragment 68421d2a5dea0775:83f5d9720000004a: Total=26.77 MB Peak=26.92 MB
SORT_NODE (id=8): Total=8.00 KB Peak=8.00 KB
Exprs: Total=4.00 KB Peak=4.00 KB
ANALYTIC_EVAL_NODE (id=7): Total=4.00 KB Peak=4.00 KB
Exprs: Total=4.00 KB Peak=4.00 KB
SORT_NODE (id=6): Total=24.00 MB Peak=24.00 MB
AGGREGATION_NODE (id=12): Total=2.72 MB Peak=2.83 MB
Exprs: Total=85.12 KB Peak=85.12 KB
EXCHANGE_NODE (id=11): Total=0 Peak=0
DataStreamRecvr: Total=0 Peak=84.80 KB
DataStreamSender (dst_id=13): Total=1.27 KB Peak=1.27 KB
CodeGen: Total=24.80 KB Peak=4.13 MB
Block Manager: Limit=161.39 GB Total=280.50 MB Peak=286.52 MB
Query(e94c89fa89a74d27:82812bf900000000): Total=258.29 MB Peak=436.85 MB
Fragment e94c89fa89a74d27:82812bf90000008e: Total=2.29 MB Peak=2.29 MB
SORT_NODE (id=11): Total=4.00 KB Peak=4.00 KB
AGGREGATION_NODE (id=20): Total=2.27 MB Peak=2.27 MB
Exprs: Total=25.12 KB Peak=25.12 KB
EXCHANGE_NODE (id=19): Total=0 Peak=0
DataStreamRecvr: Total=0 Peak=0
DataStreamSender (dst_id=21): Total=3.88 KB Peak=3.88 KB
CodeGen: Total=4.17 KB Peak=1.05 MB
Block Manager: Limit=161.39 GB Total=256.25 MB Peak=321.62 MB
Query(4e43dad3bdc935d8:938b8b7e00000000): Total=2.65 MB Peak=105.60 MB
Fragment 4e43dad3bdc935d8:938b8b7e00000052: Total=2.65 MB Peak=2.68 MB
AGGREGATION_NODE (id=15): Total=2.55 MB Peak=2.57 MB
Exprs: Total=30.12 KB Peak=30.12 KB
EXCHANGE_NODE (id=14): Total=0 Peak=0
DataStreamRecvr: Total=0 Peak=13.68 KB
DataStreamSender (dst_id=17): Total=91.41 KB Peak=91.41 KB
CodeGen: Total=1.53 KB Peak=374.50 KB
Block Manager: Limit=161.39 GB Total=512.00 KB Peak=1.30 MB
Query(b34bdd65f1ed017e:5a0291bd00000000): Total=2.37 MB Peak=106.56 MB
Fragment b34bdd65f1ed017e:5a0291bd0000004b: Total=2.37 MB Peak=2.37 MB
SORT_NODE (id=6): Total=4.00 KB Peak=4.00 KB
AGGREGATION_NODE (id=10): Total=2.35 MB Peak=2.35 MB
Exprs: Total=34.12 KB Peak=34.12 KB
EXCHANGE_NODE (id=9): Total=0 Peak=0
DataStreamRecvr: Total=0 Peak=4.23 KB
DataStreamSender (dst_id=11): Total=3.45 KB Peak=3.45 KB
CodeGen: Total=4.51 KB Peak=1.11 MB
Block Manager: Limit=161.39 GB Total=256.00 KB Peak=912.81 KB
Query(b74ba58d53b6c45f:3e8228600000000): Total=190.41 MB Peak=425.09 MB
Fragment b74ba58d53b6c45f:3e822860000009f: Total=67.90 KB Peak=2.34 MB
SORT_NODE (id=14): Total=4.00 KB Peak=4.00 KB
HASH_JOIN_NODE (id=13): Total=42.25 KB Peak=42.25 KB
Exprs: Total=9.12 KB Peak=9.12 KB
Hash Join Builder (join_node_id=13): Total=9.12 KB Peak=9.12 KB
Hash Join Builder (join_node_id=13) Exprs: Total=9.12 KB Peak=9.12 KB
HDFS_SCAN_NODE (id=11): Total=0 Peak=0
EXCHANGE_NODE (id=24): Total=0 Peak=0
DataStreamRecvr: Total=0 Peak=0
DataStreamSender (dst_id=25): Total=1.05 KB Peak=1.05 KB
CodeGen: Total=12.59 KB Peak=2.29 MB
Block Manager: Limit=161.39 GB Total=160.75 MB Peak=160.83 MB
Fragment b74ba58d53b6c45f:3e8228600000085: Total=2.32 MB Peak=2.32 MB
AGGREGATION_NODE (id=21): Total=2.29 MB Peak=2.29 MB
Exprs: Total=44.12 KB Peak=44.12 KB
EXCHANGE_NODE (id=20): Total=0 Peak=0
DataStreamRecvr: Total=0 Peak=0
DataStreamSender (dst_id=23): Total=22.09 KB Peak=22.09 KB
CodeGen: Total=2.37 KB Peak=546.00 KB
Fragment b74ba58d53b6c45f:3e8228600000060: Total=188.02 MB Peak=188.34 MB
Runtime Filter Bank: Total=16.00 MB Peak=16.00 MB
AGGREGATION_NODE (id=9): Total=1.67 MB Peak=1.67 MB
Exprs: Total=44.12 KB Peak=44.12 KB
HASH_JOIN_NODE (id=8): Total=1.13 MB Peak=1.15 MB
Exprs: Total=9.12 KB Peak=9.12 KB
Hash Join Builder (join_node_id=8): Total=1.01 MB Peak=1.02 MB
Hash Join Builder (join_node_id=8) Exprs: Total=9.12 KB Peak=9.12 KB
HASH_JOIN_NODE (id=7): Total=169.14 MB Peak=169.14 MB
Exprs: Total=9.12 KB Peak=9.12 KB
Hash Join Builder (join_node_id=7): Total=169.01 MB Peak=169.02 MB
Hash Join Builder (join_node_id=7) Exprs: Total=9.12 KB Peak=9.12 KB
EXCHANGE_NODE (id=17): Total=0 Peak=0
DataStreamRecvr: Total=0 Peak=587.50 KB
EXCHANGE_NODE (id=18): Total=0 Peak=0
DataStreamRecvr: Total=0 Peak=316.11 KB
EXCHANGE_NODE (id=19): Total=0 Peak=0
DataStreamRecvr: Total=0 Peak=4.70 KB
DataStreamSender (dst_id=20): Total=58.39 KB Peak=58.39 KB
CodeGen: Total=16.80 KB Peak=2.83 MB
Query(cb4c14997ad6add2:c8f120100000000): Total=190.36 MB Peak=443.00 MB
Fragment cb4c14997ad6add2:c8f1201000000a4: Total=67.90 KB Peak=2.34 MB
SORT_NODE (id=14): Total=4.00 KB Peak=4.00 KB
HASH_JOIN_NODE (id=13): Total=42.25 KB Peak=42.25 KB
Exprs: Total=9.12 KB Peak=9.12 KB
Hash Join Builder (join_node_id=13): Total=9.12 KB Peak=9.12 KB
Hash Join Builder (join_node_id=13) Exprs: Total=9.12 KB Peak=9.12 KB
HDFS_SCAN_NODE (id=11): Total=0 Peak=0
EXCHANGE_NODE (id=24): Total=0 Peak=0
DataStreamRecvr: Total=0 Peak=0
DataStreamSender (dst_id=25): Total=1.05 KB Peak=1.05 KB
CodeGen: Total=12.59 KB Peak=2.29 MB
Block Manager: Limit=161.39 GB Total=160.75 MB Peak=160.83 MB
Fragment cb4c14997ad6add2:c8f120100000088: Total=2.33 MB Peak=2.33 MB
AGGREGATION_NODE (id=21): Total=2.29 MB Peak=2.29 MB
Exprs: Total=44.12 KB Peak=44.12 KB
EXCHANGE_NODE (id=20): Total=0 Peak=0
DataStreamRecvr: Total=0 Peak=0
DataStreamSender (dst_id=23): Total=26.83 KB Peak=26.83 KB
CodeGen: Total=2.37 KB Peak=546.00 KB
Fragment cb4c14997ad6add2:c8f120100000063: Total=187.97 MB Peak=188.08 MB
Runtime Filter Bank: Total=16.00 MB Peak=16.00 MB
AGGREGATION_NODE (id=9): Total=1.67 MB Peak=1.67 MB
Exprs: Total=44.12 KB Peak=44.12 KB
HASH_JOIN_NODE (id=8): Total=1.14 MB Peak=1.15 MB
Exprs: Total=9.12 KB Peak=9.12 KB
Hash Join Builder (join_node_id=8): Total=1.01 MB Peak=1.02 MB
Hash Join Builder (join_node_id=8) Exprs: Total=9.12 KB Peak=9.12 KB
HASH_JOIN_NODE (id=7): Total=169.07 MB Peak=169.14 MB
Exprs: Total=9.12 KB Peak=9.12 KB
Hash Join Builder (join_node_id=7): Total=169.01 MB Peak=169.02 MB
Hash Join Builder (join_node_id=7) Exprs: Total=9.12 KB Peak=9.12 KB
EXCHANGE_NODE (id=17): Total=0 Peak=0
DataStreamRecvr: Total=0 Peak=314.15 KB
EXCHANGE_NODE (id=18): Total=0 Peak=0
DataStreamRecvr: Total=0 Peak=861.18 KB
EXCHANGE_NODE (id=19): Total=0 Peak=0
DataStreamRecvr: Total=0 Peak=4.70 KB
DataStreamSender (dst_id=20): Total=58.39 KB Peak=58.39 KB
CodeGen: Total=16.80 KB Peak=2.83 MB
Query(f04a57ce97102dd7:c2a1081700000000): Total=190.31 MB Peak=419.11 MB
Fragment f04a57ce97102dd7:c2a1081700000085: Total=2.33 MB Peak=2.33 MB
AGGREGATION_NODE (id=21): Total=2.29 MB Peak=2.29 MB
Exprs: Total=44.12 KB Peak=44.12 KB
EXCHANGE_NODE (id=20): Total=0 Peak=0
DataStreamRecvr: Total=0 Peak=0
DataStreamSender (dst_id=23): Total=23.67 KB Peak=23.67 KB
CodeGen: Total=2.37 KB Peak=546.00 KB
Block Manager: Limit=161.39 GB Total=160.75 MB Peak=160.83 MB
Fragment f04a57ce97102dd7:c2a1081700000060: Total=187.99 MB Peak=188.07 MB
Runtime Filter Bank: Total=16.00 MB Peak=16.00 MB
AGGREGATION_NODE (id=9): Total=1.68 MB Peak=1.68 MB
Exprs: Total=44.12 KB Peak=44.12 KB
HASH_JOIN_NODE (id=8): Total=1.14 MB Peak=1.15 MB
Exprs: Total=9.12 KB Peak=9.12 KB
Hash Join Builder (join_node_id=8): Total=1.01 MB Peak=1.02 MB
Hash Join Builder (join_node_id=8) Exprs: Total=9.12 KB Peak=9.12 KB
HASH_JOIN_NODE (id=7): Total=169.09 MB Peak=169.14 MB
Exprs: Total=9.12 KB Peak=9.12 KB
Hash Join Builder (join_node_id=7): Total=169.01 MB Peak=169.02 MB
Hash Join Builder (join_node_id=7) Exprs: Total=9.12 KB Peak=9.12 KB
EXCHANGE_NODE (id=17): Total=0 Peak=0
DataStreamRecvr: Total=0 Peak=156.71 KB
EXCHANGE_NODE (id=18): Total=0 Peak=0
DataStreamRecvr: Total=0 Peak=1.32 MB
EXCHANGE_NODE (id=19): Total=0 Peak=0
DataStreamRecvr: Total=0 Peak=4.70 KB
DataStreamSender (dst_id=20): Total=58.39 KB Peak=58.39 KB
CodeGen: Total=16.80 KB Peak=2.83 MB
Untracked Memory: Total=2.10 GB
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)