You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@impala.apache.org by "Mostafa Mokhtar (JIRA)" <ji...@apache.org> on 2018/03/16 04:58:00 UTC
[jira] [Created] (IMPALA-6684) Large payload going through exchange
results in lots of untracked memory in the form of a string
Mostafa Mokhtar created IMPALA-6684:
---------------------------------------
Summary: Large payload going through exchange results in lots of untracked memory in the form of a string
Key: IMPALA-6684
URL: https://issues.apache.org/jira/browse/IMPALA-6684
Project: IMPALA
Issue Type: Bug
Components: Distributed Exec
Affects Versions: Impala 2.11.0
Reporter: Mostafa Mokhtar
Attachments: Untracked memory profile.txt, heap_profile_growth_40GB.pdf
While running a query that shuffles 1GB rows noticed that there is a large amount of untracked memory in the form of a string, this issue repros with/without KRPC.
[~tgar] identified that the memory is from *tuple_data_* and that the while the payload is transient much of the memory untracked.
[https://github.com/apache/impala/blob/master/be/src/runtime/row-batch.h#L88]
{code:java}
Process: Limit=200.00 GB Total=22.85 GB Peak=201.03 GB
Buffer Pool: Free Buffers: Total=4.13 GB
Buffer Pool: Clean Pages: Total=0
Buffer Pool: Unused Reservation: Total=-2.00 GB
Data Stream Service Queue: Limit=10.00 GB Total=0 Peak=1.00 GB
Data Stream Manager Deferred RPCs: Total=0 Peak=27.70 KB
TCMalloc Overhead: Total=206.90 MB
Free Disk IO Buffers: Total=1.15 GB Peak=1.89 GB
RequestPool=root.jenkins: Total=0 Peak=13.32 GB
RequestPool=root.mmokhtar: Total=13.67 GB Peak=117.15 GB
Query(ec4cab0940313fdd:d6196e4800000000): Reservation=72.00 MB ReservationLimit=160.00 GB OtherMemory=13.60 GB Total=13.67 GB Peak=13.67 GB
Unclaimed reservations: Reservation=2.00 MB OtherMemory=0 Total=2.00 MB Peak=176.00 MB
Fragment ec4cab0940313fdd:d6196e480000002b: Reservation=34.00 MB OtherMemory=4.00 GB Total=4.03 GB Peak=4.03 GB
AGGREGATION_NODE (id=11): Total=24.00 KB Peak=24.00 KB
Exprs: Total=4.00 KB Peak=4.00 KB
HASH_JOIN_NODE (id=10): Reservation=34.00 MB OtherMemory=34.25 KB Total=34.03 MB Peak=34.05 MB
Exprs: Total=13.12 KB Peak=13.12 KB
Hash Join Builder (join_node_id=10): Total=13.12 KB Peak=21.12 KB
Hash Join Builder (join_node_id=10) Exprs: Total=13.12 KB Peak=13.12 KB
EXCHANGE_NODE (id=20): Reservation=4.00 GB OtherMemory=0 Total=4.00 GB Peak=4.00 GB
KrpcDeferredRpcs: Total=0 Peak=0
EXCHANGE_NODE (id=21): Reservation=0 OtherMemory=0 Total=0 Peak=16.00 KB
KrpcDeferredRpcs: Total=0 Peak=0
KrpcDataStreamSender (dst_id=22): Total=5.12 KB Peak=5.12 KB
CodeGen: Total=7.79 KB Peak=1.46 MB
Fragment ec4cab0940313fdd:d6196e480000000f: Reservation=36.00 MB OtherMemory=9.60 GB Total=9.63 GB Peak=9.63 GB
UNION_NODE (id=0): Total=2.09 GB Peak=2.09 GB
Exprs: Total=2.07 GB Peak=2.07 GB
AGGREGATION_NODE (id=13): Reservation=36.00 MB OtherMemory=5.44 GB Total=5.48 GB Peak=5.48 GB
Exprs: Total=5.44 GB Peak=5.44 GB
EXCHANGE_NODE (id=12): Reservation=0 OtherMemory=0 Total=0 Peak=57.54 MB
KrpcDeferredRpcs: Total=0 Peak=8.30 MB
AGGREGATION_NODE (id=15): Total=17.12 KB Peak=17.12 KB
Exprs: Total=17.12 KB Peak=17.12 KB
EXCHANGE_NODE (id=14): Reservation=16.00 KB OtherMemory=0 Total=16.00 KB Peak=16.00 KB
KrpcDeferredRpcs: Total=0 Peak=0
KrpcDataStreamSender (dst_id=20): Total=1.03 GB Peak=1.03 GB
CodeGen: Total=8.04 KB Peak=1.02 MB
Fragment ec4cab0940313fdd:d6196e4800000007: Reservation=0 OtherMemory=0 Total=0 Peak=6.36 GB
AGGREGATION_NODE (id=2): Reservation=0 OtherMemory=0 Total=0 Peak=5.92 GB
HDFS_SCAN_NODE (id=1): Total=0 Peak=535.01 MB
KrpcDataStreamSender (dst_id=12): Total=0 Peak=84.57 MB
CodeGen: Total=0 Peak=592.00 KB
Fragment ec4cab0940313fdd:d6196e480000000e: Reservation=0 OtherMemory=0 Total=0 Peak=2.34 MB
AGGREGATION_NODE (id=4): Reservation=0 OtherMemory=0 Total=0 Peak=2.02 MB
HDFS_SCAN_NODE (id=3): Total=0 Peak=280.00 KB
KrpcDataStreamSender (dst_id=14): Total=0 Peak=35.82 KB
CodeGen: Total=0 Peak=622.00 KB
Fragment ec4cab0940313fdd:d6196e4800000024: Reservation=0 OtherMemory=0 Total=0 Peak=34.22 MB
UNION_NODE (id=5): Total=0 Peak=0
AGGREGATION_NODE (id=17): Reservation=0 OtherMemory=0 Total=0 Peak=34.02 MB
EXCHANGE_NODE (id=16): Reservation=0 OtherMemory=0 Total=0 Peak=64.00 KB
KrpcDeferredRpcs: Total=0 Peak=0
AGGREGATION_NODE (id=19): Reservation=0 OtherMemory=0 Total=0 Peak=1.96 MB
EXCHANGE_NODE (id=18): Reservation=0 OtherMemory=0 Total=0 Peak=16.00 KB
KrpcDeferredRpcs: Total=0 Peak=0
KrpcDataStreamSender (dst_id=21): Total=0 Peak=103.53 KB
CodeGen: Total=0 Peak=491.50 KB
Fragment ec4cab0940313fdd:d6196e4800000023: Reservation=0 OtherMemory=0 Total=0 Peak=2.67 MB
AGGREGATION_NODE (id=9): Reservation=0 OtherMemory=0 Total=0 Peak=2.02 MB
HDFS_SCAN_NODE (id=8): Total=0 Peak=552.00 KB
KrpcDataStreamSender (dst_id=18): Total=0 Peak=99.53 KB
CodeGen: Total=0 Peak=504.50 KB
Fragment ec4cab0940313fdd:d6196e480000001c: Reservation=0 OtherMemory=0 Total=0 Peak=280.68 MB
AGGREGATION_NODE (id=7): Reservation=0 OtherMemory=0 Total=0 Peak=34.05 MB
HDFS_SCAN_NODE (id=6): Total=0 Peak=246.54 MB
KrpcDataStreamSender (dst_id=16): Total=0 Peak=127.53 KB
CodeGen: Total=0 Peak=474.50 KB
Untracked Memory: Total=5.70 GB
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)