You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2024/08/10 14:11:00 UTC
[jira] [Commented] (IMPALA-13280) Cluster of memory oversubscription errors
[ https://issues.apache.org/jira/browse/IMPALA-13280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17872592#comment-17872592 ]
ASF subversion and git services commented on IMPALA-13280:
----------------------------------------------------------
Commit c8253cee9175219ab04ceff1400b701ec4c368e8 in impala's branch refs/heads/master from Riza Suminto
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=c8253cee9 ]
IMPALA-13280: Speed up test_under_statement_expression_limit
This test spent over 7 minutes doing constant folding in Frontend
Planner and over 20 minutes doing CodeGen in Coordinator Backend. It was
holding resources from other concurrent test queries during exhaustive
test run.
This test was added by IMPALA-4551, which mainly test expression limit
during query planning. This patch disable codegen and tweak the
expression a bit to cut codegen and constant folding time.
Testing:
- Manually run test_under_statement_expression_limit in my local
machine. After patch, the test can complete in around 4 seconds.
Change-Id: If91b973fef95a89cc24a9172fdf24e2f11a19e6d
Reviewed-on: http://gerrit.cloudera.org:8080/21652
Reviewed-by: Michael Smith <mi...@cloudera.com>
Reviewed-by: Yida Wu <wy...@gmail.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
> Cluster of memory oversubscription errors
> -----------------------------------------
>
> Key: IMPALA-13280
> URL: https://issues.apache.org/jira/browse/IMPALA-13280
> Project: IMPALA
> Issue Type: Bug
> Reporter: Andrew Sherman
> Assignee: Riza Suminto
> Priority: Critical
>
> Recent changes seem to have resulted in memory oversubscription errors in TestQuery tests.
> Error Message
> {code}
> query_test/test_queries.py:146: in test_except self.run_test_case('QueryTest/except', vector) common/impala_test_suite.py:789: in run_test_case user=test_section.get('USER', '').strip() or None) common/impala_test_suite.py:708: in __exec_in_impala result = self.__execute_query(target_impalad_client, query, user=user) common/impala_test_suite.py:1079: in __execute_query return impalad_client.execute(query, user=user) common/impala_connection.py:378: in execute self.__cursor.execute(sql_stmt, configuration=self.__query_options) /data/jenkins/workspace/impala-cdw-master-exhaustive/repos/Impala/infra/python/env-gcc10.4.0/lib/python2.7/site-packages/impala/hiveserver2.py:343: in execute self._wait_to_finish() # make execute synchronous /data/jenkins/workspace/impala-cdw-master-exhaustive/repos/Impala/infra/python/env-gcc10.4.0/lib/python2.7/site-packages/impala/hiveserver2.py:428: in _wait_to_finish raise OperationalError(resp.errorMessage) E OperationalError: Failed to get minimum memory reservation of 4.09 MB on daemon impala-ec2-centos79-m6i-4xlarge-xldisk-191f.vpc.cloudera.com:27000 for query 8e498fe323d4f770:0364fa4900000000 due to following error: Memory limit exceeded: Could not allocate memory while trying to increase reservation. E Query(8e498fe323d4f770:0364fa4900000000) could not allocate 4.09 MB without exceeding limit. E Error occurred on backend impala-ec2-centos79-m6i-4xlarge-xldisk-191f.vpc.cloudera.com:27000 E Memory left in process limit: 1.19 GB E Query(8e498fe323d4f770:0364fa4900000000): Reservation=0 ReservationLimit=9.60 GB OtherMemory=0 Total=0 Peak=0 E Memory is likely oversubscribed. Reducing query concurrency or configuring admission control may help avoid this error.
> {code}
> Stacktrace
> {code}
> query_test/test_queries.py:146: in test_except
> self.run_test_case('QueryTest/except', vector)
> common/impala_test_suite.py:789: in run_test_case
> user=test_section.get('USER', '').strip() or None)
> common/impala_test_suite.py:708: in __exec_in_impala
> result = self.__execute_query(target_impalad_client, query, user=user)
> common/impala_test_suite.py:1079: in __execute_query
> return impalad_client.execute(query, user=user)
> common/impala_connection.py:378: in execute
> self.__cursor.execute(sql_stmt, configuration=self.__query_options)
> /data/jenkins/workspace/impala-cdw-master-exhaustive/repos/Impala/infra/python/env-gcc10.4.0/lib/python2.7/site-packages/impala/hiveserver2.py:343: in execute
> self._wait_to_finish() # make execute synchronous
> /data/jenkins/workspace/impala-cdw-master-exhaustive/repos/Impala/infra/python/env-gcc10.4.0/lib/python2.7/site-packages/impala/hiveserver2.py:428: in _wait_to_finish
> raise OperationalError(resp.errorMessage)
> E OperationalError: Failed to get minimum memory reservation of 4.09 MB on daemon impala-ec2-centos79-m6i-4xlarge-xldisk-191f.vpc.cloudera.com:27000 for query 8e498fe323d4f770:0364fa4900000000 due to following error: Memory limit exceeded: Could not allocate memory while trying to increase reservation.
> E Query(8e498fe323d4f770:0364fa4900000000) could not allocate 4.09 MB without exceeding limit.
> E Error occurred on backend impala-ec2-centos79-m6i-4xlarge-xldisk-191f.vpc.cloudera.com:27000
> E Memory left in process limit: 1.19 GB
> E Query(8e498fe323d4f770:0364fa4900000000): Reservation=0 ReservationLimit=9.60 GB OtherMemory=0 Total=0 Peak=0
> E Memory is likely oversubscribed. Reducing query concurrency or configuring admission control may help avoid this error.
> {code}
> Standard Error
> {code}
> -- Executing use functional_parquet
> -- getting log for operation: <tests.common.impala_connection.OperationHandle object at 0x7f10199e1f90>
> -- getting runtime profile operation: <tests.common.impala_connection.OperationHandle object at 0x7f10199e1f90>
> -- closing query for operation handle: <tests.common.impala_connection.OperationHandle object at 0x7f10199e1f90>
> -- 2024-08-05 07:13:57,673 INFO MainThread: Closing active operation
> -- 2024-08-05 07:13:57,674 INFO MainThread: Loading query test file: /data/jenkins/workspace/impala-cdw-master-exhaustive/repos/Impala/testdata/workloads/functional-query/queries/QueryTest/except.test
> -- Executing select id, bool_col, tinyint_col, smallint_col, int_col, bigint_col, float_col, double_col, date_string_col, string_col, timestamp_col, year, month from alltypestiny
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org