[jira] [Created] (IMPALA-10539) Creating tpcds testcase data

["Wenzhe Zhou (Jira)" <ji...@apache.org>]

Wenzhe Zhou created IMPALA-10539:
------------------------------------

             Summary: Creating tpcds testcase data
                 Key: IMPALA-10539
                 URL: https://issues.apache.org/jira/browse/IMPALA-10539
             Project: IMPALA
          Issue Type: Bug
          Components: Infrastructure
    Affects Versions: Impala 4.0
         Environment: s3 build
            Reporter: Wenzhe Zhou


This issue happened a handful of times in the last few days on s3 build. 

Log messages:

22:28:09 Creating tpcds testcase data (logging to /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/logs/data_loading/create-tpcds-testcase-data.log)...
22:28:09 FAILED (Took: 0 min 3 sec)
22:28:12 '/data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/testdata/bin/create-tpcds-testcase-files.sh' failed. Tail of log:
22:28:12 at org.apache.hadoop.fs.s3a.S3AFileSystem$OperationCallbacksImpl.copyFile(S3AFileSystem.java:1653)
22:28:12 at org.apache.hadoop.fs.s3a.impl.RenameOperation.copySourceAndUpdateTracker(RenameOperation.java:636)
22:28:12 at org.apache.hadoop.fs.s3a.impl.RenameOperation.lambda$initiateCopy$1(RenameOperation.java:600)
22:28:12 at org.apache.hadoop.fs.s3a.impl.CallableSupplier.get(CallableSupplier.java:62)
22:28:12 at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
22:28:12 at org.apache.hadoop.util.SemaphoredDelegatingExecutor$RunnableWithPermitRelease.run(SemaphoredDelegatingExecutor.java:197)
22:28:12 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
22:28:12 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
22:28:12 at java.lang.Thread.run(Thread.java:748)
22:28:12 Caused by: java.io.FileNotFoundException: copyFile(test-warehouse/tpcds-testcase-data/impala-testcase-data-7cf4cea0-932c-47fb-a4b0-00985d26528d, user/jenkins/.Trash/Current/test-warehouse/tpcds-testcase-data1614061691533/impala-testcase-data-7cf4cea0-932c-47fb-a4b0-00985d26528d) on test-warehouse/tpcds-testcase-data/impala-testcase-data-7cf4cea0-932c-47fb-a4b0-00985d26528d: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: DA4BB61317C3469E; S3 Extended Request ID: oSFXrBiZ7pDu1pQISlVzxsKL59aTgIA5KR6iTT77rEglPp9TFudtSkMlfWPZ+JX7zXQRPk1xwec=; Proxy: null), S3 Extended Request ID: oSFXrBiZ7pDu1pQISlVzxsKL59aTgIA5KR6iTT77rEglPp9TFudtSkMlfWPZ+JX7zXQRPk1xwec=:404 Not Found
22:28:12 at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:260)
22:28:12 at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:117)
22:28:12 at org.apache.hadoop.fs.s3a.S3AFileSystem.copyFile(S3AFileSystem.java:3698)
22:28:12 ... 10 more
22:28:12 Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404; Error Code: 404 Not Found; Request ID: DA4BB61317C3469E; S3 Extended Request ID: oSFXrBiZ7pDu1pQISlVzxsKL59aTgIA5KR6iTT77rEglPp9TFudtSkMlfWPZ+JX7zXQRPk1xwec=; Proxy: null), S3 Extended Request ID: oSFXrBiZ7pDu1pQISlVzxsKL59aTgIA5KR6iTT77rEglPp9TFudtSkMlfWPZ+JX7zXQRPk1xwec=
22:28:12 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1828)
22:28:12 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1412)
22:28:12 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1374)
22:28:12 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145)
22:28:12 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802)
22:28:12 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)
22:28:12 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)
22:28:12 at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)
22:28:12 at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)
22:28:12 at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550)
22:28:12 at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530)
22:28:12 at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5227)
22:28:12 at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5173)
22:28:12 at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1360)
22:28:12 at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getObjectMetadata$6(S3AFileSystem.java:2027)
22:28:12 at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:412)
22:28:12 at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:375)
22:28:12 at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:2017)
22:28:12 at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$copyFile$18(S3AFileSystem.java:3700)
22:28:12 at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:115)
22:28:12 ... 11 more
22:28:12 rm: Failed to move to trash: s3a://impala-test-uswest2-1/test-warehouse/tpcds-testcase-data: No such file or directory: s3a://impala-test-uswest2-1/test-warehouse/tpcds-testcase-data. Consider using -skipTrash option
22:28:12 21/02/22 22:28:12 INFO impl.MetricsSystemImpl: Stopping s3a-file-system metrics system...
22:28:12 21/02/22 22:28:12 INFO impl.MetricsSystemImpl: s3a-file-system metrics system stopped.
22:28:12 21/02/22 22:28:12 INFO impl.MetricsSystemImpl: s3a-file-system metrics system shutdown complete.
22:28:12 ++ report_build_error 39
22:28:12 +++ cd /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala
22:28:12 +++ awk 'NR == 39' /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/testdata/bin/create-tpcds-testcase-files.sh
22:28:12 ++ ERROR_MSG='hadoop fs -rm -r -f $TESTCASE_DATA_DIR'
22:28:12 +++ basename -- /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/testdata/bin/create-tpcds-testcase-files.sh
22:28:12 ++ FILENAME=create-tpcds-testcase-files.sh
22:28:12 ++ echo ERROR in /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/testdata/bin/create-tpcds-testcase-files.sh at line 39: hadoop fs -rm -r -f '$TESTCASE_DATA_DIR'
22:28:12 ERROR in /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/testdata/bin/create-tpcds-testcase-files.sh at line 39: hadoop fs -rm -r -f $TESTCASE_DATA_DIR
22:28:12 ++ /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/bin/generate_junitxml.py --step create-tpcds-testcase-files --error 'Error in /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/testdata/bin/create-tpcds-testcase-files.sh at line 39: hadoop fs -rm -r -f $TESTCASE_DATA_DIR'
22:28:12 Generated: /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/logs/extra_junit_xml_logs/generate_junitxml.buildall.create-tpcds-testcase-files.20210223_06_28_12.xml
22:28:12 ERROR in /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/testdata/bin/create-load-data.sh at line 48:
22:28:12 Generated: /data/jenkins/workspace/impala-cdpd-master-core-s3/repos/Impala/logs/extra_junit_xml_logs/generate_junitxml.buildall.create-load-data.20210223_06_28_12.xml
22:28:12 + echo 'buildall.sh ' -format '-snapshot_file /data/jenkins/workspace/impala-cdpd-master-core-s3/testdata/test-warehouse-SNAPSHOT/test-warehouse-cdh7-651-SNAPSHOT.tar.gz' '-metastore_snapshot_file /data/jenkins/workspace/impala-cdpd-master-core-s3/testdata/hive_metastore_snapshot/hive_impala_dump_cdh7-651.txt failed.'
22:28:12 buildall.sh -format -snapshot_file /data/jenkins/workspace/impala-cdpd-master-core-s3/testdata/test-warehouse-SNAPSHOT/test-warehouse-cdh7-651-SNAPSHOT.tar.gz -metastore_snapshot_file /data/jenkins/workspace/impala-cdpd-master-core-s3/testdata/hive_metastore_snapshot/hive_impala_dump_cdh7-651.txt failed.
22:28:12 + RET_VAL=1
22:28:12 + [[ DEBUG = \A\S\A\N ]]
22:28:12 + [[ DEBUG = \S\T\R\E\S\S ]]
22:28:12 + [[ DEBUG = \T\S\A\N ]]
22:28:12 + '[' 1 -ne 0 ']'
22:28:12 + echo
22:28:12
22:28:12 + echo
22:28:12
22:28:12 + echo '**** BUILD FAILED! ****'
22:28:12 **** BUILD FAILED! ****



--
This message was sent by Atlassian Jira
(v8.3.4#803005)