You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Wenzhe Zhou (Jira)" <ji...@apache.org> on 2021/02/23 22:44:00 UTC
[jira] [Commented] (IMPALA-10539) s3 build failures caused by the
conflicts of two Jenkins jobs running on the same bucket
[ https://issues.apache.org/jira/browse/IMPALA-10539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289405#comment-17289405 ]
Wenzhe Zhou commented on IMPALA-10539:
--------------------------------------
There are more build failures for s3 related builds since last Thursday. Some of failures were caused by the new bugs introduced by new patches.
But other failures seems caused by s3 testing environment like s3a-file-system. This issue happened when creating tpcds testcase data or running bin/run-all-tests.sh.
There is another failure which complain "S3 copy failed" as below:
metadata/test_load.py:78: in setup_method
"\{0}/\{1}/100101.txt".format(STAGING_PATH, i)) util/hdfs_util.py:241: in copy
'\{0} copy failed: '.format(self.filesystem_type) + stderr + "; " + stdout
E AssertionError: S3 copy failed: 21/02/23 01:07:11 WARN impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-s3a-file-system.properties,hadoop-metrics2.properties
E 21/02/23 01:07:11 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
E 21/02/23 01:07:11 INFO impl.MetricsSystemImpl: s3a-file-system metrics system started
E 21/02/23 01:07:12 INFO Configuration.deprecation: No unit for fs.s3a.connection.request.timeout(0) assuming SECONDS
E cp: `/test-warehouse/test_load_staging/1/100101.txt': No such file or directory: `s3a://impala-test-uswest2-1/test-warehouse/test_load_staging/1/100101.txt'
E 21/02/23 01:07:13 INFO impl.MetricsSystemImpl: Stopping s3a-file-system metrics system...
E 21/02/23 01:07:13 INFO impl.MetricsSystemImpl: s3a-file-system metrics system stopped.
E 21/02/23 01:07:13 INFO impl.MetricsSystemImpl: s3a-file-system metrics system shutdown complete.
All test cases for front end testing org.apache.impala.planner.S3PlannerTest failed in another build, and saw errors like
12:49:38 E ImpalaBeeswaxException: ImpalaBeeswaxException:
12:49:38 E INNER EXCEPTION: <class 'beeswaxd.ttypes.BeeswaxException'>
12:49:38 E MESSAGE: InternalException: Unable to cache data source library at location 's3a://impala-test-uswest2-2/test-warehouse/data-sources/test-data-source.jar'. Check that the file exists and is readable. Message: Failed to get file info s3a://impala-test-uswest2-2/test-warehouse/data-sources/test-data-source.jar
12:49:38 E Error(2): No such file or directory
All these failures happened when accessing files under directory "test-warehouse" in s3a file system.
> s3 build failures caused by the conflicts of two Jenkins jobs running on the same bucket
> ----------------------------------------------------------------------------------------
>
> Key: IMPALA-10539
> URL: https://issues.apache.org/jira/browse/IMPALA-10539
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
> Affects Versions: Impala 4.0
> Environment: s3 build
> Reporter: Wenzhe Zhou
> Assignee: Laszlo Gaal
> Priority: Major
> Labels: broken-build
>
> This issue happened a handful of times in the last few days for different s3 builds.
> Log messages:
> 22:28:09 Creating tpcds testcase data (logging to /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/logs/data_loading/create-tpcds-testcase-data.log)...
> 22:28:09 FAILED (Took: 0 min 3 sec)
> 22:28:12 '/data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/testdata/bin/create-tpcds-testcase-files.sh' failed. Tail of log:
> 22:28:12 at org.apache.hadoop.fs.s3a.S3AFileSystem$OperationCallbacksImpl.copyFile(S3AFileSystem.java:1653)
> 22:28:12 at org.apache.hadoop.fs.s3a.impl.RenameOperation.copySourceAndUpdateTracker(RenameOperation.java:636)
> 22:28:12 at org.apache.hadoop.fs.s3a.impl.RenameOperation.lambda$initiateCopy$1(RenameOperation.java:600)
> 22:28:12 at org.apache.hadoop.fs.s3a.impl.CallableSupplier.get(CallableSupplier.java:62)
> 22:28:12 at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
> 22:28:12 at org.apache.hadoop.util.SemaphoredDelegatingExecutor$RunnableWithPermitRelease.run(SemaphoredDelegatingExecutor.java:197)
> 22:28:12 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 22:28:12 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 22:28:12 at java.lang.Thread.run(Thread.java:748)
> 22:28:12 Caused by: java.io.FileNotFoundException: copyFile(test-warehouse/tpcds-testcase-data/impala-testcase-data-7cf4cea0-932c-47fb-a4b0-00985d26528d, user/jenkins/.Trash/Current/test-warehouse/tpcds-testcase-data1614061691533/impala-testcase-data-7cf4cea0-932c-47fb-a4b0-00985d26528d) on test-warehouse/tpcds-testcase-data/impala-testcase-data-7cf4cea0-932c-47fb-a4b0-00985d26528d: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: DA4BB61317C3469E; S3 Extended Request ID: oSFXrBiZ7pDu1pQISlVzxsKL59aTgIA5KR6iTT77rEglPp9TFudtSkMlfWPZ+JX7zXQRPk1xwec=; Proxy: null), S3 Extended Request ID: oSFXrBiZ7pDu1pQISlVzxsKL59aTgIA5KR6iTT77rEglPp9TFudtSkMlfWPZ+JX7zXQRPk1xwec=:404 Not Found
> 22:28:12 at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:260)
> 22:28:12 at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:117)
> 22:28:12 at org.apache.hadoop.fs.s3a.S3AFileSystem.copyFile(S3AFileSystem.java:3698)
> 22:28:12 ... 10 more
> 22:28:12 Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: DA4BB61317C3469E; S3 Extended Request ID: oSFXrBiZ7pDu1pQISlVzxsKL59aTgIA5KR6iTT77rEglPp9TFudtSkMlfWPZ+JX7zXQRPk1xwec=; Proxy: null), S3 Extended Request ID: oSFXrBiZ7pDu1pQISlVzxsKL59aTgIA5KR6iTT77rEglPp9TFudtSkMlfWPZ+JX7zXQRPk1xwec=
> 22:28:12 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1828)
> 22:28:12 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1412)
> 22:28:12 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1374)
> 22:28:12 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145)
> 22:28:12 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802)
> 22:28:12 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)
> 22:28:12 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)
> 22:28:12 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)
> 22:28:12 at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)
> 22:28:12 at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550)
> 22:28:12 at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530)
> 22:28:12 at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5227)
> 22:28:12 at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5173)
> 22:28:12 at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1360)
> 22:28:12 at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getObjectMetadata$6(S3AFileSystem.java:2027)
> 22:28:12 at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:412)
> 22:28:12 at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:375)
> 22:28:12 at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:2017)
> 22:28:12 at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$copyFile$18(S3AFileSystem.java:3700)
> 22:28:12 at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:115)
> 22:28:12 ... 11 more
> 22:28:12 rm: Failed to move to trash: s3a://impala-test-uswest2-1/test-warehouse/tpcds-testcase-data: No such file or directory: s3a://impala-test-uswest2-1/test-warehouse/tpcds-testcase-data. Consider using -skipTrash option
> 22:28:12 21/02/22 22:28:12 INFO impl.MetricsSystemImpl: Stopping s3a-file-system metrics system...
> 22:28:12 21/02/22 22:28:12 INFO impl.MetricsSystemImpl: s3a-file-system metrics system stopped.
> 22:28:12 21/02/22 22:28:12 INFO impl.MetricsSystemImpl: s3a-file-system metrics system shutdown complete.
> 22:28:12 ++ report_build_error 39
> 22:28:12 +++ cd /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala
> 22:28:12 +++ awk 'NR == 39' /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/testdata/bin/create-tpcds-testcase-files.sh
> 22:28:12 ++ ERROR_MSG='hadoop fs -rm -r -f $TESTCASE_DATA_DIR'
> 22:28:12 +++ basename – /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/testdata/bin/create-tpcds-testcase-files.sh
> 22:28:12 ++ FILENAME=create-tpcds-testcase-files.sh
> 22:28:12 ++ echo ERROR in /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/testdata/bin/create-tpcds-testcase-files.sh at line 39: hadoop fs -rm -r -f '$TESTCASE_DATA_DIR'
> 22:28:12 ERROR in /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/testdata/bin/create-tpcds-testcase-files.sh at line 39: hadoop fs -rm -r -f $TESTCASE_DATA_DIR
> 22:28:12 ++ /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/bin/generate_junitxml.py --step create-tpcds-testcase-files --error 'Error in /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/testdata/bin/create-tpcds-testcase-files.sh at line 39: hadoop fs -rm -r -f $TESTCASE_DATA_DIR'
> 22:28:12 Generated: /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/logs/extra_junit_xml_logs/generate_junitxml.buildall.create-tpcds-testcase-files.20210223_06_28_12.xml
> 22:28:12 ERROR in /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/testdata/bin/create-load-data.sh at line 48:
> 22:28:12 Generated: /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/logs/extra_junit_xml_logs/generate_junitxml.buildall.create-load-data.20210223_06_28_12.xml
> 22:28:12 + echo 'buildall.sh ' -format '-snapshot_file /data/jenkins/workspace/impala-cdpd-master-core-s3/testdata/test-warehouse-SNAPSHOT/test-warehouse-cdh7-651-SNAPSHOT.tar.gz' '-metastore_snapshot_file /data/jenkins/workspace/impala-cdpd-master-core-s3/testdata/hive_metastore_snapshot/hive_impala_dump_cdh7-651.txt failed.'
> 22:28:12 buildall.sh -format -snapshot_file /data/jenkins/workspace/impala-cdpd-master-core-s3/testdata/test-warehouse-SNAPSHOT/test-warehouse-cdh7-651-SNAPSHOT.tar.gz -metastore_snapshot_file /data/jenkins/workspace/impala-cdpd-master-core-s3/testdata/hive_metastore_snapshot/hive_impala_dump_cdh7-651.txt failed.
> 22:28:12 + RET_VAL=1
> 22:28:12 + [[ DEBUG = \A\S\A\N ]]
> 22:28:12 + [[ DEBUG = \S\T\R\E\S\S ]]
> 22:28:12 + [[ DEBUG = \T\S\A\N ]]
> 22:28:12 + '[' 1 -ne 0 ']'
> 22:28:12 + echo
> 22:28:12
> 22:28:12 + echo
> 22:28:12
> 22:28:12 + echo '**** BUILD FAILED! ****'
> 22:28:12 **** BUILD FAILED! ****
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org