You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Riza Suminto (Jira)" <ji...@apache.org> on 2021/04/17 03:45:00 UTC
[jira] [Commented] (IMPALA-10611) test_wide_row fails with 'Failed
to allocate row batch'
[ https://issues.apache.org/jira/browse/IMPALA-10611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17324160#comment-17324160 ]
Riza Suminto commented on IMPALA-10611:
---------------------------------------
Hi [~arawat], it looks like test_wide_row becomes flaky after IMPALA-9856 (enable result spooling by default).
I remember changing this test, setting query option max_row_size, but not raising its mem_limit. We should probably add 10MB to the mem_limit to match the max_row_size query option.
I'd like to assign this Jira to myself.
> test_wide_row fails with 'Failed to allocate row batch'
> -------------------------------------------------------
>
> Key: IMPALA-10611
> URL: https://issues.apache.org/jira/browse/IMPALA-10611
> Project: IMPALA
> Issue Type: Bug
> Reporter: Abhishek Rawat
> Priority: Major
>
> It appears that we may not be accounting for memory properly somewhere and as a result we end up with allocations much more than reservations resulting in OOM error. It's also possible we may not be freeing memory properly (could be a leak or memory accumulation due to smart pointers etc).
> It is also possible that something (table data?) changed in the test case and as a result the memory limit is not sufficient.
> *Error:*
> {code:java}
> Messagequery_test/test_scanners.py:276: in test_wide_row self.run_test_case('QueryTest/wide-row', new_vector) common/impala_test_suite.py:665: in run_test_case result = exec_fn(query, user=test_section.get('USER', '').strip() or None) common/impala_test_suite.py:603: in __exec_in_impala result = self.__execute_query(target_impalad_client, query, user=user) common/impala_test_suite.py:923: in __execute_query return impalad_client.execute(query, user=user) common/impala_connection.py:205: in execute return self.__beeswax_client.execute(sql_stmt, user=user) beeswax/impala_beeswax.py:187: in execute handle = self.__execute_query(query_string.strip(), user=user) beeswax/impala_beeswax.py:365: in __execute_query self.wait_for_finished(handle) beeswax/impala_beeswax.py:386: in wait_for_finished raise ImpalaBeeswaxException("Query aborted:" + error_log, None) E ImpalaBeeswaxException: ImpalaBeeswaxException: E Query aborted:Memory limit exceeded: Failed to allocate row batch E EXCHANGE_NODE (id=1) could not allocate 16.00 MB without exceeding limit. E Error occurred on backend ip-172-31-14-211:27000 E Memory left in process limit: 9.09 GB E Memory left in query limit: 9.95 MB E Query(7647dab034192a59:e4f2972900000000): Limit=100.00 MB Reservation=40.00 MB ReservationLimit=68.00 MB OtherMemory=50.05 MB Total=90.05 MB Peak=90.05 MB E Unclaimed reservations: Reservation=32.00 MB OtherMemory=0 Total=32.00 MB Peak=40.00 MB E Fragment 7647dab034192a59:e4f2972900000001: Reservation=8.00 MB OtherMemory=50.04 MB Total=58.04 MB Peak=58.05 MB E HDFS_SCAN_NODE (id=0): Reservation=8.00 MB OtherMemory=30.01 MB Total=38.01 MB Peak=58.03 MB E KrpcDataStreamSender (dst_id=1): Total=9.84 KB Peak=9.84 KB E CodeGen: Total=1004.00 B Peak=118.00 KB E Fragment 7647dab034192a59:e4f2972900000000: Reservation=0 OtherMemory=16.00 KB Total=16.00 KB Peak=16.00 KB E EXCHANGE_NODE (id=1): Reservation=8.00 KB OtherMemory=0 Total=8.00 KB Peak=8.00 KB E KrpcDeferredRpcs: Total=0 Peak=0 E PLAN_ROOT_SINK: Total=0 Peak=0 E CodeGen: Total=90.00 B Peak=65.00 KB{code}
> *StackTrace:*
> {code:java}
> query_test/test_scanners.py:276: in test_wide_row
> self.run_test_case('QueryTest/wide-row', new_vector)
> common/impala_test_suite.py:665: in run_test_case
> result = exec_fn(query, user=test_section.get('USER', '').strip() or None)
> common/impala_test_suite.py:603: in __exec_in_impala
> result = self.__execute_query(target_impalad_client, query, user=user)
> common/impala_test_suite.py:923: in __execute_query
> return impalad_client.execute(query, user=user)
> common/impala_connection.py:205: in execute
> return self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:187: in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:365: in __execute_query
> self.wait_for_finished(handle)
> beeswax/impala_beeswax.py:386: in wait_for_finished
> raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
> E ImpalaBeeswaxException: ImpalaBeeswaxException:
> E Query aborted:Memory limit exceeded: Failed to allocate row batch
> E EXCHANGE_NODE (id=1) could not allocate 16.00 MB without exceeding limit.
> E Error occurred on backend ip-172-31-14-211:27000
> E Memory left in process limit: 9.09 GB
> E Memory left in query limit: 9.95 MB
> E Query(7647dab034192a59:e4f2972900000000): Limit=100.00 MB Reservation=40.00 MB ReservationLimit=68.00 MB OtherMemory=50.05 MB Total=90.05 MB Peak=90.05 MB
> E Unclaimed reservations: Reservation=32.00 MB OtherMemory=0 Total=32.00 MB Peak=40.00 MB
> E Fragment 7647dab034192a59:e4f2972900000001: Reservation=8.00 MB OtherMemory=50.04 MB Total=58.04 MB Peak=58.05 MB
> E HDFS_SCAN_NODE (id=0): Reservation=8.00 MB OtherMemory=30.01 MB Total=38.01 MB Peak=58.03 MB
> E KrpcDataStreamSender (dst_id=1): Total=9.84 KB Peak=9.84 KB
> E CodeGen: Total=1004.00 B Peak=118.00 KB
> E Fragment 7647dab034192a59:e4f2972900000000: Reservation=0 OtherMemory=16.00 KB Total=16.00 KB Peak=16.00 KB
> E EXCHANGE_NODE (id=1): Reservation=8.00 KB OtherMemory=0 Total=8.00 KB Peak=8.00 KB
> E KrpcDeferredRpcs: Total=0 Peak=0
> E PLAN_ROOT_SINK: Total=0 Peak=0
> E CodeGen: Total=90.00 B Peak=65.00 KB
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org