You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@impala.apache.org by "Quanlong Huang (Jira)" <ji...@apache.org> on 2024/10/06 12:25:00 UTC
[jira] [Created] (IMPALA-13422) TestIcebergTable.test_load failed when running concurrently
Quanlong Huang created IMPALA-13422:
---------------------------------------
Summary: TestIcebergTable.test_load failed when running concurrently
Key: IMPALA-13422
URL: https://issues.apache.org/jira/browse/IMPALA-13422
Project: IMPALA
Issue Type: Bug
Reporter: Quanlong Huang
Assignee: Quanlong Huang
TestIcebergTable.test_load failed when two tests running concurrently in an exhaustive build:
{code:java}
19:12:26 [gw2] FAILED query_test/test_iceberg.py::TestIcebergTable::test_load[protocol: beeswax | exec_option: {'test_replan': 1, 'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: parquet/none]
19:12:26 [gw3] FAILED query_test/test_iceberg.py::TestIcebergTable::test_load[protocol: beeswax | exec_option: {'test_replan': 1, 'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': True, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: parquet/none]{code}
Stacktrace of both tests are the same:
{code:python}
query_test/test_iceberg.py:1300: in test_load
"iceberg_mixed_file_format_test", "parquet")
common/file_utils.py:62: in create_iceberg_table_from_directory
check_call(['hdfs', 'dfs', '-put', '-d', local_dir, hdfs_dir])
/data/jenkins/workspace/impala-asf-master-exhaustive-release-arm/Impala-Toolchain/toolchain-packages-gcc10.4.0/python-2.7.16/lib/python2.7/subprocess.py:190: in check_call
raise CalledProcessError(retcode, cmd)
E CalledProcessError: Command '['hdfs', 'dfs', '-put', '-d', '/data/jenkins/workspace/impala-asf-master-exhaustive-release-arm/repos/Impala/testdata/data/iceberg_test/iceberg_mixed_file_format_test', '/test-warehouse/iceberg_mixed_file_format_test']' returned non-zero exit status 1{code}
Stderr of the first test:
{code}
CREATE DATABASE `test_load_a61184e9`;
-- 2024-10-03 19:12:05,971 INFO MainThread: Started query 0e431e374cd58716:0013fb8100000000
-- 2024-10-03 19:12:06,022 INFO MainThread: Created database "test_load_a61184e9" for test ID "query_test/test_iceberg.py::TestIcebergTable::()::test_load[protocol: beeswax | exec_option: {'test_replan': 1, 'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: parquet/none]"
Picked up JAVA_TOOL_OPTIONS: -javaagent:/data/jenkins/workspace/impala-asf-master-exhaustive-release-arm/repos/Impala/fe/target/dependency/jamm-0.4.0.jar
Picked up JAVA_TOOL_OPTIONS: -javaagent:/data/jenkins/workspace/impala-asf-master-exhaustive-release-arm/repos/Impala/fe/target/dependency/jamm-0.4.0.jar
put: `/test-warehouse/iceberg_mixed_file_format_test/data/00000-0-data-gfurnstahl_20220906113255_8d49367d-e338-4996-ade5-ee500a19c1d1-job_16619542960420_0003-1-00001.orc': File exists
put: `/test-warehouse/iceberg_mixed_file_format_test/metadata/d43cc1ea-096f-4594-9583-b1b27f8f0230-m0.avro': File exists
put: `/test-warehouse/iceberg_mixed_file_format_test/metadata/snap-5574591442446832859-1-055baf62-de6d-4583-bf21-f187f9482343.avro': File exists
put: `/test-warehouse/iceberg_mixed_file_format_test/metadata/snap-6603964444137547572-1-871d1473-8566-46c0-a530-a2256b3f396f.avro': File exists
put: `/test-warehouse/iceberg_mixed_file_format_test/metadata/v2.metadata.json': File exists
put: `/test-warehouse/iceberg_mixed_file_format_test/metadata/v4.metadata.json': File exists
put: `/test-warehouse/iceberg_mixed_file_format_test/metadata/v6.metadata.json': File exists
put: `/test-warehouse/iceberg_mixed_file_format_test/metadata/v8.metadata.json': File exists{code}
stderr of the second test:
{code}
CREATE DATABASE `test_load_53f920bb`;
-- 2024-10-03 19:12:05,971 INFO MainThread: Started query 68460e4230189328:9f6261bc00000000
-- 2024-10-03 19:12:06,022 INFO MainThread: Created database "test_load_53f920bb" for test ID "query_test/test_iceberg.py::TestIcebergTable::()::test_load[protocol: beeswax | exec_option: {'test_replan': 1, 'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': True, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: parquet/none]"
Picked up JAVA_TOOL_OPTIONS: -javaagent:/data/jenkins/workspace/impala-asf-master-exhaustive-release-arm/repos/Impala/fe/target/dependency/jamm-0.4.0.jar
Picked up JAVA_TOOL_OPTIONS: -javaagent:/data/jenkins/workspace/impala-asf-master-exhaustive-release-arm/repos/Impala/fe/target/dependency/jamm-0.4.0.jar
put: `/test-warehouse/iceberg_mixed_file_format_test/data/00000-0-data-gfurnstahl_20220906113044_157fc172-f5d3-4c70-8653-fff150b6136a-job_16619542960420_0002-1-00001.parquet': File exists
put: `/test-warehouse/iceberg_mixed_file_format_test/data/00000-0-data-gfurnstahl_20220906114830_907f72c7-36ac-4135-8315-27ff880faff0-job_16619542960420_0004-1-00001.parquet': File exists
put: `/test-warehouse/iceberg_mixed_file_format_test/data/00000-0-data-gfurnstahl_20220906114900_9c1b7b46-5643-428f-a007-519c5500ed04-job_16619542960420_0004-1-00001.orc': File exists
put: `/test-warehouse/iceberg_mixed_file_format_test/metadata/055baf62-de6d-4583-bf21-f187f9482343-m0.avro': File exists
put: `/test-warehouse/iceberg_mixed_file_format_test/metadata/871d1473-8566-46c0-a530-a2256b3f396f-m0.avro': File exists
put: `/test-warehouse/iceberg_mixed_file_format_test/metadata/a0332d05-4c9c-4640-8656-69ba61b7859a-m0.avro': File exists
put: `/test-warehouse/iceberg_mixed_file_format_test/metadata/snap-1847986799759674135-1-a0332d05-4c9c-4640-8656-69ba61b7859a.avro': File exists
put: `/test-warehouse/iceberg_mixed_file_format_test/metadata/snap-5589333083859779254-1-d43cc1ea-096f-4594-9583-b1b27f8f0230.avro': File exists
put: `/test-warehouse/iceberg_mixed_file_format_test/metadata/v1.metadata.json': File exists
put: `/test-warehouse/iceberg_mixed_file_format_test/metadata/v3.metadata.json': File exists
put: `/test-warehouse/iceberg_mixed_file_format_test/metadata/v5.metadata.json': File exists
put: `/test-warehouse/iceberg_mixed_file_format_test/metadata/v7.metadata.json': File exists
put: `/test-warehouse/iceberg_mixed_file_format_test/metadata/version-hint.txt': File exists{code}
It seems the failure is due to two runs of the test adding files to the same path.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)