You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Michael Smith (Jira)" <ji...@apache.org> on 2023/06/01 00:13:00 UTC
[jira] [Work stopped] (IMPALA-12135) TestDataStreamSenderTpch.test_krpc_datastream_sender_shuffle fails with memory limit exceeded
[ https://issues.apache.org/jira/browse/IMPALA-12135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Work on IMPALA-12135 stopped by Michael Smith.
----------------------------------------------
> TestDataStreamSenderTpch.test_krpc_datastream_sender_shuffle fails with memory limit exceeded
> ----------------------------------------------------------------------------------------------
>
> Key: IMPALA-12135
> URL: https://issues.apache.org/jira/browse/IMPALA-12135
> Project: IMPALA
> Issue Type: Bug
> Reporter: Tamas Mate
> Assignee: Riza Suminto
> Priority: Major
>
> impala-asf-master-core-erasure-coding build is failing with memory limit exceeded
> {code:none}
> query_test/test_datastream_sender.py:36: in test_krpc_datastream_sender_shuffle self.run_test_case('datastream-sender', vector) common/impala_test_suite.py:718: in run_test_case
> result = exec_fn(query, user=test_section.get('USER', '').strip() or None) common/impala_test_suite.py:656: in __exec_in_impala
> result = self.__execute_query(target_impalad_client, query, user=user) common/impala_test_suite.py:992: in __execute_query
> return impalad_client.execute(query, user=user) common/impala_connection.py:214: in execute
> return self.__beeswax_client.execute(sql_stmt, user=user) beeswax/impala_beeswax.py:191: in execute
> handle = self.__execute_query(query_string.strip(), user=user) beeswax/impala_beeswax.py:369: in __execute_query
> self.wait_for_finished(handle) beeswax/impala_beeswax.py:390: in wait_for_finished
> raise ImpalaBeeswaxException("Query aborted:" + error_log, None) E ImpalaBeeswaxException: ImpalaBeeswaxException:
> E Query aborted:Memory limit exceeded: Error occurred on backend impala-ec2-centos79-m6i-4xlarge-ondemand-0ae4.vpc.cloudera.com:27002 by fragment 274c1835003503b5:b6e769a900000009 E Memory left in process limit: 8.29 GB E Memory left in query limit: -46.05 MB E Query(274c1835003503b5:b6e769a900000000): memory limit exceeded. Limit=1.00 GB Reservation=19.50 MB ReservationLimit=819.20 MB OtherMemory=1.03 GB Total=1.04 GB Peak=1.04 GB
> E Unclaimed reservations: Reservation=2.00 MB OtherMemory=0 Total=2.00 MB Peak=75.69 MB
> E Fragment 274c1835003503b5:b6e769a900000007: Reservation=0 OtherMemory=0 Total=0 Peak=6.68 MB
> E AGGREGATION_NODE (id=11): Reservation=0 OtherMemory=0 Total=0 Peak=2.03 MB
> E GroupingAggregator 0: Reservation=0 OtherMemory=0 Total=0 Peak=2.02 MB
> E HDFS_SCAN_NODE (id=10): Reservation=0 OtherMemory=0 Total=0 Peak=4.63 MB
> E KrpcDataStreamSender (dst_id=24): Total=0 Peak=14.25 KB
> E RowBatchSerialization: Total=0 Peak=31.00 B
> E Fragment 274c1835003503b5:b6e769a900000006: Reservation=0 OtherMemory=0 Total=0 Peak=12.78 MB
> E AGGREGATION_NODE (id=9): Reservation=0 OtherMemory=0 Total=0 Peak=3.03 MB
> E GroupingAggregator 0: Reservation=0 OtherMemory=0 Total=0 Peak=3.02 MB
> E HDFS_SCAN_NODE (id=8): Reservation=0 OtherMemory=0 Total=0 Peak=9.72 MB
> E KrpcDataStreamSender (dst_id=22): Total=0 Peak=17.45 KB
> E RowBatchSerialization: Total=0 Peak=3.24 KB
> E Fragment 274c1835003503b5:b6e769a900000008: Reservation=0 OtherMemory=0 Total=0 Peak=4.95 MB
> E UNION_NODE (id=7): Total=0 Peak=0
> E AGGREGATION_NODE (id=23): Reservation=0 OtherMemory=0 Total=0 Peak=4.77 MB
> E GroupingAggregator 0: Reservation=0 OtherMemory=0 Total=0 Peak=4.77 MB
> E EXCHANGE_NODE (id=22): Reservation=0 OtherMemory=0 Total=0 Peak=136.00 KB
> E KrpcDeferredRpcs: Total=0 Peak=0
> E AGGREGATION_NODE (id=25): Reservation=0 OtherMemory=0 Total=0 Peak=2.08 MB
> E GroupingAggregator 0: Reservation=0 OtherMemory=0 Total=0 Peak=2.08 MB
> E EXCHANGE_NODE (id=24): Reservation=0 OtherMemory=0 Total=0 Peak=16.00 KB
> E KrpcDeferredRpcs: Total=0 Peak=0
> E KrpcDataStreamSender (dst_id=27): Total=0 Peak=17.45 KB
> E RowBatchSerialization: Total=0 Peak=3.24 KB
> E CodeGen: Total=0 Peak=1.19 MB
> E CodeGen: Total=0 Peak=1.15 MB
> E CodeGen: Total=0 Peak=1.21 MB
> E Fragment 274c1835003503b5:b6e769a900000003: Reservation=0 OtherMemory=0 Total=0 Peak=10.68 MB
> E AGGREGATION_NODE (id=5): Reservation=0 OtherMemory=0 Total=0 Peak=2.14 MB
> E GroupingAggregator 0: Reservation=0 OtherMemory=0 Total=0 Peak=2.03 MB
> E HDFS_SCAN_NODE (id=4): Reservation=0 OtherMemory=0 Total=0 Peak=8.63 MB
> E KrpcDataStreamSender (dst_id=18): Total=0 Peak=6.18 KB
> E RowBatchSerialization: Total=0 Peak=90.00 B
> E CodeGen: Total=0 Peak=1.31 MB
> E Fragment 274c1835003503b5:b6e769a900000004: Reservation=0 OtherMemory=0 Total=0 Peak=4.13 MB
> E AGGREGATION_NODE (id=6): Reservation=0 OtherMemory=0 Total=0 Peak=2.03 MB
> E GroupingAggregator 0: Reservation=0 OtherMemory=0 Total=0 Peak=2.02 MB
> E AGGREGATION_NODE (id=19): Reservation=0 OtherMemory=0 Total=0 Peak=2.10 MB
> E GroupingAggregator 0: Reservation=0 OtherMemory=0 Total=0 Peak=2.09 MB
> E EXCHANGE_NODE (id=18): Reservation=0 OtherMemory=0 Total=0 Peak=16.00 KB
> E KrpcDeferredRpcs: Total=0 Peak=0
> E KrpcDataStreamSender (dst_id=20): Total=0 Peak=6.19 KB
> E RowBatchSerialization: Total=0 Peak=100.00 B
> E CodeGen: Total=0 Peak=2.10 MB
> E Fragment 274c1835003503b5:b6e769a900000001: Reservation=0 OtherMemory=0 Total=0 Peak=42.16 MB
> E AGGREGATION_NODE (id=2): Reservation=0 OtherMemory=0 Total=0 Peak=32.12 MB
> E GroupingAggregator 0: Reservation=0 OtherMemory=0 Total=0 Peak=32.03 MB
> E HDFS_SCAN_NODE (id=1): Reservation=0 OtherMemory=0 Total=0 Peak=12.90 MB
> E KrpcDataStreamSender (dst_id=14): Total=0 Peak=262.58 KB
> E RowBatchSerialization: Total=0 Peak=256.49 KB
> E CodeGen: Total=0 Peak=1.27 MB
> E Fragment 274c1835003503b5:b6e769a900000002: Reservation=0 OtherMemory=0 Total=0 Peak=53.93 MB
> E AGGREGATION_NODE (id=3): Reservation=0 OtherMemory=0 Total=0 Peak=32.96 MB
> E GroupingAggregator 0: Reservation=0 OtherMemory=0 Total=0 Peak=32.95 MB
> E AGGREGATION_NODE (id=15): Reservation=0 OtherMemory=0 Total=0 Peak=37.53 MB
> E GroupingAggregator 0: Reservation=0 OtherMemory=0 Total=0 Peak=37.53 MB
> E EXCHANGE_NODE (id=14): Reservation=0 OtherMemory=0 Total=0 Peak=16.37 MB
> E KrpcDeferredRpcs: Total=0 Peak=0
> E KrpcDataStreamSender (dst_id=16): Total=0 Peak=4.88 MB
> E RowBatchSerialization: Total=0 Peak=4.87 MB
> E CodeGen: Total=0 Peak=2.10 MB
> E Fragment 274c1835003503b5:b6e769a900000005: Reservation=9.00 MB OtherMemory=794.49 MB Total=803.49 MB Peak=887.31 MB
> E UNION_NODE (id=0): Total=227.00 MB Peak=227.00 MB
> E Exprs: Total=226.14 MB Peak=226.14 MB
> E AGGREGATION_NODE (id=17): Reservation=9.00 MB OtherMemory=14.85 MB Total=23.85 MB Peak=24.84 MB
> E GroupingAggregator 0: Reservation=9.00 MB OtherMemory=14.85 MB Total=23.85 MB Peak=24.84 MB
> E Exprs: Total=14.84 MB Peak=15.83 MB
> E EXCHANGE_NODE (id=16): Reservation=0 OtherMemory=0 Total=0 Peak=11.02 MB
> E KrpcDeferredRpcs: Total=0 Peak=0
> E AGGREGATION_NODE (id=21): Reservation=0 OtherMemory=17.12 KB Total=17.12 KB Peak=17.12 KB
> E GroupingAggregator 0: Total=17.12 KB Peak=17.12 KB
> E Exprs: Total=17.12 KB Peak=17.12 KB
> E EXCHANGE_NODE (id=20): Reservation=16.00 KB OtherMemory=0 Total=16.00 KB Peak=16.00 KB
> E KrpcDeferredRpcs: Total=0 Peak=0
> E KrpcDataStreamSender (dst_id=26): Total=431.20 MB Peak=431.20 MB
> E RowBatchSerialization: Total=431.20 MB Peak=431.20 MB
> E CodeGen: Total=0 Peak=2.73 MB
> E Fragment 274c1835003503b5:b6e769a900000009: Reservation=8.50 MB OtherMemory=256.06 MB Total=264.56 MB Peak=264.56 MB
> E AGGREGATION_NODE (id=13): Total=4.00 KB Peak=4.00 KB
> E NonGroupingAggregator 0: Total=4.00 KB Peak=4.00 KB
> E Exprs: Total=4.00 KB Peak=4.00 KB
> E HASH_JOIN_NODE (id=12): Reservation=8.50 MB OtherMemory=34.25 KB Total=8.53 MB Peak=8.55 MB
> E Exprs: Total=13.12 KB Peak=13.12 KB
> E Hash Join Builder (join_node_id=12): Total=13.12 KB Peak=21.12 KB
> E Hash Join Builder (join_node_id=12) Exprs: Total=13.12 KB Peak=13.12 KB
> E EXCHANGE_NODE (id=26): Reservation=256.01 MB OtherMemory=0 Total=256.01 MB Peak=256.01 MB
> E KrpcDeferredRpcs: Total=0 Peak=0
> E EXCHANGE_NODE (id=27): Reservation=0 OtherMemory=0 Total=0 Peak=272.00 KB
> E KrpcDeferredRpcs: Total=0 Peak=0
> E KrpcDataStreamSender (dst_id=28): Total=5.12 KB Peak=5.12 KB
> E RowBatchSerialization: Total=0 Peak=0
> E CodeGen: Total=0 Peak=1.87 MB
> E E Memory limit exceeded: Error occurred on backend impala-ec2-centos79-m6i-4xlarge-ondemand-0ae4.vpc.cloudera.com:27002 by fragment 274c1835003503b5:b6e769a900000009 E Memory left in process limit: 8.29 GB E Memory left in query limit: -46.05 MB E Query(274c1835003503b5:b6e769a900000000): memory limit exceeded. Limit=1.00 GB Reservation=19.50 MB ReservationLimit=819.20 MB OtherMemory=1.03 GB Total=1.04 GB Peak=1.04 GB
> E Unclaimed reservations: Reservation=2.00 MB OtherMemory=0 Total=2.00 MB Peak=75.69 MB
> E Fragment 274c1835003503b5:b6e769a900000007: Reservation=0 OtherMemory=0 Total=0 Peak=6.68 MB
> E AGGREGATION_NODE (id=11): Reservation=0 OtherMemory=0 Total=0 Peak=2.03 MB
> E GroupingAggregator 0: Reservation=0 OtherMemory=0 Total=0 Peak=2.02 MB
> E HDFS_SCAN_NODE (id=10): Reservation=0 OtherMemory=0 Total=0 Peak=4.63 MB
> E KrpcDataStreamSender (dst_id=24): Total=0 Peak=14.25 KB
> E RowBatchSerialization: Total=0 Peak=31.00 B
> E Fragment 274c1835003503b5:b6e769a900000006: Reservation=0 OtherMemory=0 Total=0 Peak=12.78 MB
> E AGGREGATION_NODE (id=9): Reservation=0 OtherMemory=0 Total=0 Peak=3.03 MB
> E GroupingAggregator 0: Reservation=0 OtherMemory=0 Total=0 Peak=3.02 MB
> E HDFS_SCAN_NODE (id=8): Reservation=0 OtherMemory=0 Total=0 Peak=9.72 MB
> E KrpcDataStreamSender (dst_id=22): Total=0 Peak=17.45 KB
> E RowBatchSerialization: Total=0 Peak=3.24 KB
> E Fragment 274c1835003503b5:b6e769a900000008: Reservation=0 OtherMemory=0 Total=0 Peak=4.95 MB
> E UNION_NODE (id=7): Total=0 Peak=0
> E AGGREGATION_NODE (id=23): Reservation=0 OtherMemory=0 Total=0 Peak=4.77 MB
> E GroupingAggregator 0: Reservation=0 OtherMemory=0 Total=0 Peak=4.77 MB
> E EXCHANGE_NODE (id=22): Reservation=0 OtherMemory=0 Total=0 Peak=136.00 KB
> E KrpcDeferredRpcs: Total=0 Peak=0
> E AGGREGATION_NODE (id=25): Reservation=0 OtherMemory=0 Total=0 Peak=2.08 MB
> E GroupingAggregator 0: Reservation=0 OtherMemory=0 Total=0 Peak=2.08 MB
> E EXCHANGE_NODE (id=24): Reservation=0 OtherMemory=0 Total=0 Peak=16.00 KB
> E KrpcDeferredRpcs: Total=0 Peak=0
> E KrpcDataStreamSender (dst_id=27): Total=0 Peak=17.45 KB
> E RowBatchSerialization: Total=0 Peak=3.24 KB
> E CodeGen: Total=0 Peak=1.19 MB
> E CodeGen: Total=0 Peak=1.15 MB
> E CodeGen: Total=0 Peak=1.21 MB
> E Fragment 274c1835003503b5:b6e769a900000003: Reservation=0 OtherMemory=0 Total=0 Peak=10.68 MB
> E AGGREGATION_NODE (id=5): Reservation=0 OtherMemory=0 Total=0 Peak=2.14 MB
> E GroupingAggregator 0: Reservation=0 OtherMemory=0 Total=0 Peak=2.03 MB
> E HDFS_SCAN_NODE (id=4): Reservation=0 OtherMemory=0 Total=0 Peak=8.63 MB
> E KrpcDataStreamSender (dst_id=18): Total=0 Peak=6.18 KB
> E RowBatchSerialization: Total=0 Peak=90.00 B
> E CodeGen: Total=0 Peak=1.31 MB
> E Fragment 274c1835003503b5:b6e769a900000004: Reservation=0 OtherMemory=0 Total=0 Peak=4.13 MB
> E AGGREGATION_NODE (id=6): Reservation=0 OtherMemory=0 Total=0 Peak=2.03 MB
> E GroupingAggregator 0: Reservation=0 OtherMemory=0 Total=0 Peak=2.02 MB
> E AGGREGATION_NODE (id=19): Reservation=0 OtherMemory=0 Total=0 Peak=2.10 MB
> E GroupingAggregator 0: Reservation=0 OtherMemory=0 Total=0 Peak=2.09 MB
> E EXCHANGE_NODE (id=18): Reservation=0 OtherMemory=0 Total=0 Peak=16.00 KB
> E KrpcDeferredRpcs: Total=0 Peak=0
> E KrpcDataStreamSender (dst_id=20): Total=0 Peak=6.19 KB
> E RowBatchSerialization: Total=0 Peak=100.00 B
> E CodeGen: Total=0 Peak=2.10 MB
> E Fragment 274c1835003503b5:b6e769a900000001: Reservation=0 OtherMemory=0 Total=0 Peak=42.16 MB
> E AGGREGATION_NODE (id=2): Reservation=0 OtherMemory=0 Total=0 Peak=32.12 MB
> E GroupingAggregator 0: Reservation=0 OtherMemory=0 Total=0 Peak=32.03 MB
> E HDFS_SCAN_NODE (id=1): Reservation=0 OtherMemory=0 Total=0 Peak=12.90 MB
> E KrpcDataStreamSender (dst_id=14): Total=0 Peak=262.58 KB
> E RowBatchSerialization: Total=0 Peak=256.49 KB
> E CodeGen: Total=0 Peak=1.27 MB
> E Fragment 274c1835003503b5:b6e769a900000002: Reservation=0 OtherMemory=0 Total=0 Peak=53.93 MB
> E AGGREGATION_NODE (id=3): Reservation=0 OtherMemory=0 Total=0 Peak=32.96 MB
> E GroupingAggregator 0: Reservation=0 OtherMemory=0 Total=0 Peak=32.95 MB
> E AGGREGATION_NODE (id=15): Reservation=0 OtherMemory=0 Total=0 Peak=37.53 MB
> E GroupingAggregator 0: Reservation=0 OtherMemory=0 Total=0 Peak=37.53 MB
> E EXCHANGE_NODE (id=14): Reservation=0 OtherMemory=0 Total=0 Peak=16.37 MB
> E KrpcDeferredRpcs: Total=0 Peak=0
> E KrpcDataStreamSender (dst_id=16): Total=0 Peak=4.88 MB
> E RowBatchSerialization: Total=0 Peak=4.87 MB
> E CodeGen: Total=0 Peak=2.10 MB
> E Fragment 274c1835003503b5:b6e769a900000005: Reservation=9.00 MB OtherMemory=794.49 MB Total=803.49 MB Peak=887.31 MB
> E UNION_NODE (id=0): Total=227.00 MB Peak=227.00 MB
> E Exprs: Total=226.14 MB Peak=226.14 MB
> E AGGREGATION_NODE (id=17): Reservation=9.00 MB OtherMemory=14.85 MB Total=23.85 MB Peak=24.84 MB
> E GroupingAggregator 0: Reservation=9.00 MB OtherMemory=14.85 MB Total=23.85 MB Peak=24.84 MB
> E Exprs: Total=14.84 MB Peak=15.83 MB
> E EXCHANGE_NODE (id=16): Reservation=0 OtherMemory=0 Total=0 Peak=11.02 MB
> E KrpcDeferredRpcs: Total=0 Peak=0
> E AGGREGATION_NODE (id=21): Reservation=0 OtherMemory=17.12 KB Total=17.12 KB Peak=17.12 KB
> E GroupingAggregator 0: Total=17.12 KB Peak=17.12 KB
> E Exprs: Total=17.12 KB Peak=17.12 KB
> E EXCHANGE_NODE (id=20): Reservation=16.00 KB OtherMemory=0 Total=16.00 KB Peak=16.00 KB
> E KrpcDeferredRpcs: Total=0 Peak=0
> E KrpcDataStreamSender (dst_id=26): Total=431.20 MB Peak=431.20 MB
> E RowBatchSerialization: Total=431.20 MB Peak=431.20 MB
> E CodeGen: Total=0 Peak=2.73 MB
> E Fragment 274c1835003503b5:b6e769a900000009: Reservation=8.50 MB OtherMemory=256.06 MB Total=264.56 MB Peak=264.56 MB
> E AGGREGATION_NODE (id=13): Total=4.00 KB Peak=4.00 KB
> E NonGroupingAggregator 0: Total=4.00 KB Peak=4.00 KB
> E Exprs: Total=4.00 KB Peak=4.00 KB
> E HASH_JOIN_NODE (id=12): Reservation=8.50 MB OtherMemory=34.25 KB Total=8.53 MB Peak=8.55 MB
> E Exprs: Total=13.12 KB Peak=13.12 KB
> E Hash Join Builder (join_node_id=12): Total=13.12 KB Peak=21.12 KB
> E Hash Join Builder (join_node_id=12) Exprs: Total=13.12 KB Peak=13.12 KB
> E EXCHANGE_NODE (id=26): Reservation=256.01 MB OtherMemory=0 Total=256.01 MB Peak=256.01 MB
> E KrpcDeferredRpcs: Total=0 Peak=0
> E EXCHANGE_NODE (id=27): Reservation=0 OtherMemory=0 Total=0 Peak=272.00 KB
> E KrpcDeferredRpcs: Total=0 Peak=0
> E KrpcDataStreamSender (dst_id=28): Total=5.12 KB Peak=5.12 KB
> E RowBatchSerialization: Total=0 Peak=0
> E CodeGen: Total=0 Peak=1.87 MB (1 of 4 similar)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org