You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@impala.apache.org by "Wenzhe Zhou (Jira)" <ji...@apache.org> on 2021/02/26 08:33:00 UTC
[jira] [Closed] (IMPALA-10539) s3 build failures caused by the
conflicts of two Jenkins jobs running on the same bucket
[ https://issues.apache.org/jira/browse/IMPALA-10539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wenzhe Zhou closed IMPALA-10539.
--------------------------------
Resolution: Not A Bug
> s3 build failures caused by the conflicts of two Jenkins jobs running on the same bucket
> ----------------------------------------------------------------------------------------
>
> Key: IMPALA-10539
> URL: https://issues.apache.org/jira/browse/IMPALA-10539
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
> Affects Versions: Impala 4.0
> Environment: s3 build
> Reporter: Wenzhe Zhou
> Assignee: Laszlo Gaal
> Priority: Major
> Labels: broken-build
>
> This issue happened a handful of times in the last few days for different s3 builds.
> Log messages:
> 22:28:09 Creating tpcds testcase data (logging to /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/logs/data_loading/create-tpcds-testcase-data.log)...
> 22:28:09 FAILED (Took: 0 min 3 sec)
> 22:28:12 '/data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/testdata/bin/create-tpcds-testcase-files.sh' failed. Tail of log:
> 22:28:12 at org.apache.hadoop.fs.s3a.S3AFileSystem$OperationCallbacksImpl.copyFile(S3AFileSystem.java:1653)
> 22:28:12 at org.apache.hadoop.fs.s3a.impl.RenameOperation.copySourceAndUpdateTracker(RenameOperation.java:636)
> 22:28:12 at org.apache.hadoop.fs.s3a.impl.RenameOperation.lambda$initiateCopy$1(RenameOperation.java:600)
> 22:28:12 at org.apache.hadoop.fs.s3a.impl.CallableSupplier.get(CallableSupplier.java:62)
> 22:28:12 at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
> 22:28:12 at org.apache.hadoop.util.SemaphoredDelegatingExecutor$RunnableWithPermitRelease.run(SemaphoredDelegatingExecutor.java:197)
> 22:28:12 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 22:28:12 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 22:28:12 at java.lang.Thread.run(Thread.java:748)
> 22:28:12 Caused by: java.io.FileNotFoundException: copyFile(test-warehouse/tpcds-testcase-data/impala-testcase-data-7cf4cea0-932c-47fb-a4b0-00985d26528d, user/jenkins/.Trash/Current/test-warehouse/tpcds-testcase-data1614061691533/impala-testcase-data-7cf4cea0-932c-47fb-a4b0-00985d26528d) on test-warehouse/tpcds-testcase-data/impala-testcase-data-7cf4cea0-932c-47fb-a4b0-00985d26528d: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: DA4BB61317C3469E; S3 Extended Request ID: oSFXrBiZ7pDu1pQISlVzxsKL59aTgIA5KR6iTT77rEglPp9TFudtSkMlfWPZ+JX7zXQRPk1xwec=; Proxy: null), S3 Extended Request ID: oSFXrBiZ7pDu1pQISlVzxsKL59aTgIA5KR6iTT77rEglPp9TFudtSkMlfWPZ+JX7zXQRPk1xwec=:404 Not Found
> 22:28:12 at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:260)
> 22:28:12 at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:117)
> 22:28:12 at org.apache.hadoop.fs.s3a.S3AFileSystem.copyFile(S3AFileSystem.java:3698)
> 22:28:12 ... 10 more
> 22:28:12 Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: DA4BB61317C3469E; S3 Extended Request ID: oSFXrBiZ7pDu1pQISlVzxsKL59aTgIA5KR6iTT77rEglPp9TFudtSkMlfWPZ+JX7zXQRPk1xwec=; Proxy: null), S3 Extended Request ID: oSFXrBiZ7pDu1pQISlVzxsKL59aTgIA5KR6iTT77rEglPp9TFudtSkMlfWPZ+JX7zXQRPk1xwec=
> 22:28:12 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1828)
> 22:28:12 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1412)
> 22:28:12 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1374)
> 22:28:12 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145)
> 22:28:12 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802)
> 22:28:12 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)
> 22:28:12 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)
> 22:28:12 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)
> 22:28:12 at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)
> 22:28:12 at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550)
> 22:28:12 at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530)
> 22:28:12 at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5227)
> 22:28:12 at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5173)
> 22:28:12 at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1360)
> 22:28:12 at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getObjectMetadata$6(S3AFileSystem.java:2027)
> 22:28:12 at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:412)
> 22:28:12 at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:375)
> 22:28:12 at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:2017)
> 22:28:12 at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$copyFile$18(S3AFileSystem.java:3700)
> 22:28:12 at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:115)
> 22:28:12 ... 11 more
> 22:28:12 rm: Failed to move to trash: s3a://impala-test-uswest2-1/test-warehouse/tpcds-testcase-data: No such file or directory: s3a://impala-test-uswest2-1/test-warehouse/tpcds-testcase-data. Consider using -skipTrash option
> 22:28:12 21/02/22 22:28:12 INFO impl.MetricsSystemImpl: Stopping s3a-file-system metrics system...
> 22:28:12 21/02/22 22:28:12 INFO impl.MetricsSystemImpl: s3a-file-system metrics system stopped.
> 22:28:12 21/02/22 22:28:12 INFO impl.MetricsSystemImpl: s3a-file-system metrics system shutdown complete.
> 22:28:12 ++ report_build_error 39
> 22:28:12 +++ cd /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala
> 22:28:12 +++ awk 'NR == 39' /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/testdata/bin/create-tpcds-testcase-files.sh
> 22:28:12 ++ ERROR_MSG='hadoop fs -rm -r -f $TESTCASE_DATA_DIR'
> 22:28:12 +++ basename – /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/testdata/bin/create-tpcds-testcase-files.sh
> 22:28:12 ++ FILENAME=create-tpcds-testcase-files.sh
> 22:28:12 ++ echo ERROR in /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/testdata/bin/create-tpcds-testcase-files.sh at line 39: hadoop fs -rm -r -f '$TESTCASE_DATA_DIR'
> 22:28:12 ERROR in /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/testdata/bin/create-tpcds-testcase-files.sh at line 39: hadoop fs -rm -r -f $TESTCASE_DATA_DIR
> 22:28:12 ++ /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/bin/generate_junitxml.py --step create-tpcds-testcase-files --error 'Error in /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/testdata/bin/create-tpcds-testcase-files.sh at line 39: hadoop fs -rm -r -f $TESTCASE_DATA_DIR'
> 22:28:12 Generated: /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/logs/extra_junit_xml_logs/generate_junitxml.buildall.create-tpcds-testcase-files.20210223_06_28_12.xml
> 22:28:12 ERROR in /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/testdata/bin/create-load-data.sh at line 48:
> 22:28:12 Generated: /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/logs/extra_junit_xml_logs/generate_junitxml.buildall.create-load-data.20210223_06_28_12.xml
> 22:28:12 + echo 'buildall.sh ' -format '-snapshot_file /data/jenkins/workspace/impala-cdpd-master-core-s3/testdata/test-warehouse-SNAPSHOT/test-warehouse-cdh7-651-SNAPSHOT.tar.gz' '-metastore_snapshot_file /data/jenkins/workspace/impala-cdpd-master-core-s3/testdata/hive_metastore_snapshot/hive_impala_dump_cdh7-651.txt failed.'
> 22:28:12 buildall.sh -format -snapshot_file /data/jenkins/workspace/impala-cdpd-master-core-s3/testdata/test-warehouse-SNAPSHOT/test-warehouse-cdh7-651-SNAPSHOT.tar.gz -metastore_snapshot_file /data/jenkins/workspace/impala-cdpd-master-core-s3/testdata/hive_metastore_snapshot/hive_impala_dump_cdh7-651.txt failed.
> 22:28:12 + RET_VAL=1
> 22:28:12 + [[ DEBUG = \A\S\A\N ]]
> 22:28:12 + [[ DEBUG = \S\T\R\E\S\S ]]
> 22:28:12 + [[ DEBUG = \T\S\A\N ]]
> 22:28:12 + '[' 1 -ne 0 ']'
> 22:28:12 + echo
> 22:28:12
> 22:28:12 + echo
> 22:28:12
> 22:28:12 + echo '**** BUILD FAILED! ****'
> 22:28:12 **** BUILD FAILED! ****
--
This message was sent by Atlassian Jira
(v8.3.4#803005)