You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@impala.apache.org by "Mostafa Mokhtar (JIRA)" <ji...@apache.org> on 2018/02/28 03:24:00 UTC
[jira] [Created] (IMPALA-6596) Query failed with OOM on coordinator
but didn't while remote fragments on other nodes continue to run
Mostafa Mokhtar created IMPALA-6596:
---------------------------------------
Summary: Query failed with OOM on coordinator but didn't while remote fragments on other nodes continue to run
Key: IMPALA-6596
URL: https://issues.apache.org/jira/browse/IMPALA-6596
Project: IMPALA
Issue Type: Improvement
Affects Versions: Impala 2.11.0
Reporter: Mostafa Mokhtar
Assignee: Tim Armstrong
Attachments: RowSizeFailureProfile.txt
This could be related to IMPALA-2990.
Query
{code}
set NUM_SCANNER_THREADS=2;
set MAX_ROW_SIZE=4194304;
with cte as (select group_concat(cast(fnv_hash(concat(o_comment, 'd')) as string)) as c1,group_concat(cast(fnv_hash(concat(o_comment, 'e')) as string)) as c2 from orders where o_orderkey <1200000000 and o_orderdate <"1993-01-01" union all select group_concat(cast(fnv_hash(concat(o_comment, 'd')) as string)) as c1,group_concat(cast(fnv_hash(concat(o_comment, 'e')) as string)) as c2 from orders where o_orderkey <1200000000 and o_orderdate <"1993-01-01"), cte2 as (select c1,c2,s_suppkey from cte , supplier) select count(*) from cte2 t1, cte2 t2 where t1.s_suppkey = t2.s_suppkey group by t1.c1 , t1.c2 , t2.c1 , t2.c2 having count(*) = 1
{code}
Failed on coordinator node which is also an executor with
{code}
h4. _Status:_ Row of size 1.82 GB could not be materialized in plan node with id 14. Increase the max_row_size query option (currently 4.00 MB) to process larger rows.
{code}
Log on the coordinator has lots of entries with
{code}
I0227 19:20:58.057637 62974 impala-server.cc:1196] ReportExecStatus(): Received report for unknown query ID (probably closed or cancelled): 9b439fc1ee1addb7:82d4156900000000 I0227 19:20:58.152979 63129 impala-server.cc:1196] ReportExecStatus(): Received report for unknown query ID (probably closed or cancelled): 9b439fc1ee1addb7:82d4156900000000 I0227 19:20:58.714336 63930 impala-server.cc:1196] ReportExecStatus(): Received report for unknown query ID (probably closed or cancelled): 9b439fc1ee1addb7:82d4156900000000 I0227 19:20:58.718415 63095 impala-server.cc:1196] ReportExecStatus(): Received report for unknown query ID (probably closed or cancelled): 9b439fc1ee1addb7:82d4156900000000 I0227 19:20:58.757306 63339 impala-server.cc:1196] ReportExecStatus(): Received report for unknown query ID (probably closed or cancelled): 9b439fc1ee1addb7:82d4156900000000 I0227 19:20:58.762310 63406 impala-server.cc:1196] ReportExecStatus(): Received report for unknown query ID (probably closed or cancelled): 9b439fc1ee1addb7:82d4156900000000
{code}
From the memz tab on a different node
{code}
Query(9b439fc1ee1addb7:82d4156900000000): Reservation=122.00 MB ReservationLimit=80.00 GB OtherMemory=39.98 GB Total=40.10 GB Peak=40.10 GB Unclaimed reservations: Reservation=72.00 MB OtherMemory=0 Total=72.00 MB Peak=226.00 MB Fragment 9b439fc1ee1addb7:82d4156900000059: Reservation=0 OtherMemory=0 Total=0 Peak=632.88 KB AGGREGATION_NODE (id=29): Total=0 Peak=76.12 KB EXCHANGE_NODE (id=28): Reservation=0 OtherMemory=0 Total=0 Peak=0 DataStreamRecvr: Total=0 Peak=0 DataStreamSender (dst_id=30): Total=0 Peak=1.75 KB CodeGen: Total=0 Peak=547.00 KB Fragment 9b439fc1ee1addb7:82d4156900000050: Reservation=0 OtherMemory=0 Total=0 Peak=3.67 GB AGGREGATION_NODE (id=15): Total=0 Peak=76.12 KB HASH_JOIN_NODE (id=14): Reservation=0 OtherMemory=0 Total=0 Peak=1.85 GB Hash Join Builder (join_node_id=14): Total=0 Peak=13.12 KB EXCHANGE_NODE (id=26): Reservation=0 OtherMemory=0 Total=0 Peak=1.82 GB DataStreamRecvr: Total=0 Peak=1.82 GB EXCHANGE_NODE (id=27): Reservation=0 OtherMemory=0 Total=0 Peak=1.82 GB DataStreamRecvr: Total=0 Peak=1.82 GB DataStreamSender (dst_id=28): Total=0 Peak=15.75 KB CodeGen: Total=0 Peak=1.81 MB Fragment 9b439fc1ee1addb7:82d4156900000023: Reservation=26.00 MB OtherMemory=19.99 GB Total=20.02 GB Peak=20.02 GB Runtime Filter Bank: Reservation=2.00 MB ReservationLimit=2.00 MB OtherMemory=0 Total=2.00 MB Peak=2.00 MB NESTED_LOOP_JOIN_NODE (id=6): Total=3.63 GB Peak=3.63 GB Nested Loop Join Builder: Total=3.63 GB Peak=3.63 GB HDFS_SCAN_NODE (id=5): Reservation=24.00 MB OtherMemory=2.88 MB Total=26.88 MB Peak=26.88 MB EXCHANGE_NODE (id=20): Reservation=0 OtherMemory=0 Total=0 Peak=1.82 GB DataStreamRecvr: Total=0 Peak=1.82 GB DataStreamSender (dst_id=26): Total=16.36 GB Peak=16.36 GB CodeGen: Total=628.00 B Peak=222.50 KB Fragment 9b439fc1ee1addb7:82d4156900000007: Reservation=0 OtherMemory=0 Total=0 Peak=734.73 MB AGGREGATION_NODE (id=2): Total=0 Peak=516.29 MB HDFS_SCAN_NODE (id=1): Reservation=0 OtherMemory=0 Total=0 Peak=350.27 MB DataStreamSender (dst_id=16): Total=0 Peak=3.88 KB CodeGen: Total=0 Peak=839.50 KB Fragment 9b439fc1ee1addb7:82d4156900000014: Reservation=0 OtherMemory=0 Total=0 Peak=734.18 MB AGGREGATION_NODE (id=4): Total=0 Peak=516.33 MB HDFS_SCAN_NODE (id=3): Reservation=0 OtherMemory=0 Total=0 Peak=349.92 MB DataStreamSender (dst_id=18): Total=0 Peak=3.88 KB CodeGen: Total=0 Peak=839.50 KB Fragment 9b439fc1ee1addb7:82d4156900000047: Reservation=24.00 MB OtherMemory=19.99 GB Total=20.02 GB Peak=20.02 GB NESTED_LOOP_JOIN_NODE (id=13): Total=3.63 GB Peak=3.63 GB Nested Loop Join Builder: Total=3.63 GB Peak=3.63 GB HDFS_SCAN_NODE (id=12): Reservation=24.00 MB OtherMemory=2.88 MB Total=26.88 MB Peak=26.88 MB EXCHANGE_NODE (id=25): Reservation=0 OtherMemory=0 Total=0 Peak=1.82 GB DataStreamRecvr: Total=0 Peak=1.82 GB DataStreamSender (dst_id=27): Total=16.36 GB Peak=16.36 GB CodeGen: Total=234.00 B Peak=52.50 KB Fragment 9b439fc1ee1addb7:82d415690000002b: Reservation=0 OtherMemory=0 Total=0 Peak=734.73 MB AGGREGATION_NODE (id=9): Total=0 Peak=516.29 MB HDFS_SCAN_NODE (id=8): Reservation=0 OtherMemory=0 Total=0 Peak=350.24 MB DataStreamSender (dst_id=21): Total=0 Peak=3.88 KB CodeGen: Total=0 Peak=839.50 KB Fragment 9b439fc1ee1addb7:82d4156900000038: Reservation=0 OtherMemory=0 Total=0 Peak=734.73 MB AGGREGATION_NODE (id=11): Total=0 Peak=516.29 MB HDFS_SCAN_NODE (id=10): Reservation=0 OtherMemory=0 Total=0 Peak=350.52 MB DataStreamSender (dst_id=23): Total=0 Peak=3.88 KB CodeGen: Total=0 Peak=839.50 KB
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)