You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Steve Loughran (JIRA)" <ji...@apache.org> on 2016/06/02 09:46:59 UTC
[jira] [Comment Edited] (HADOOP-13231) Isolate test path used by a
few S3A tests for more reliable parallel execution.
[ https://issues.apache.org/jira/browse/HADOOP-13231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15312036#comment-15312036 ]
Steve Loughran edited comment on HADOOP-13231 at 6/2/16 9:46 AM:
-----------------------------------------------------------------
Chris, here are the errors I saw on a parallel HADOOP-13171 run
{code}
Running org.apache.hadoop.fs.s3a.TestS3ABlockingThreadPool
Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 25.167 sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.TestS3ABlockingThreadPool
testRegularMultiPartUpload(org.apache.hadoop.fs.s3a.TestS3ABlockingThreadPool) Time elapsed: 13.218 sec <<< ERROR!
java.io.FileNotFoundException: saving output on tests3a/97e97bdb-5ca4-46b8-a3d2-7dbb2100101b: com.amazonaws.services.s3.model.AmazonS3Exception: The specified upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchUpload; Request ID: 0A0C0CA457261DB8), S3 Extended Request ID: kLavwGlMzkH7dm1uwbCQ1N6fK8HfUOi/PkNf4lSE7rXRQVmLkkqE6Nd86n2wu8UOZy8jdXK7+TY=
at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:106)
at org.apache.hadoop.fs.s3a.S3AOutputStream.close(S3AOutputStream.java:123)
at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101)
at org.apache.hadoop.fs.contract.ContractTestUtils.generateTestFile(ContractTestUtils.java:864)
at org.apache.hadoop.fs.contract.ContractTestUtils.createAndVerifyFile(ContractTestUtils.java:892)
at org.apache.hadoop.fs.s3a.TestS3ABlockingThreadPool.testRegularMultiPartUpload(TestS3ABlockingThreadPool.java:68)
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: The specified upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchUpload; Request ID: 0A0C0CA457261DB8)
at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1182)
at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:770)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3785)
at com.amazonaws.services.s3.AmazonS3Client.completeMultipartUpload(AmazonS3Client.java:2705)
at com.amazonaws.services.s3.transfer.internal.CompleteMultipartUpload.call(CompleteMultipartUpload.java:91)
at com.amazonaws.services.s3.transfer.internal.CompleteMultipartUpload.call(CompleteMultipartUpload.java:38)
at org.apache.hadoop.fs.s3a.BlockingThreadPoolExecutorService$CallableWithPermitRelease.call(BlockingThreadPoolExecutorService.java:239)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
testFastMultiPartUpload(org.apache.hadoop.fs.s3a.TestS3ABlockingThreadPool) Time elapsed: 11.866 sec <<< ERROR!
java.io.FileNotFoundException: Completing multi-part upload on tests3a/77dd6617-f0ac-45d4-bd3f-3475c07bb002: com.amazonaws.services.s3.model.AmazonS3Exception: The specified upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchUpload; Request ID: C8FD35777E51FF7C), S3 Extended Request ID: MZ8VyewWz4/qUu5h/MTDv+F7q9co2fhUkOCLxSt9Z5mXkO8vimUGQW9Fizvm0hkG82IPuvaiWaI=
at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1182)
at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:770)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3785)
at com.amazonaws.services.s3.AmazonS3Client.completeMultipartUpload(AmazonS3Client.java:2705)
at org.apache.hadoop.fs.s3a.S3AFastOutputStream$MultiPartUpload.complete(S3AFastOutputStream.java:374)
at org.apache.hadoop.fs.s3a.S3AFastOutputStream$MultiPartUpload.access$200(S3AFastOutputStream.java:319)
at org.apache.hadoop.fs.s3a.S3AFastOutputStream.close(S3AFastOutputStream.java:255)
at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101)
at org.apache.hadoop.fs.contract.ContractTestUtils.generateTestFile(ContractTestUtils.java:864)
at org.apache.hadoop.fs.contract.ContractTestUtils.createAndVerifyFile(ContractTestUtils.java:892)
at org.apache.hadoop.fs.s3a.TestS3ABlockingThreadPool.testFastMultiPartUpload(TestS3ABlockingThreadPool.java:76)
Running org.apache.hadoop.fs.s3a.TestS3ABlocksize
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.275 sec - in org.apache.hadoop.fs.s3a
{code}
was (Author: stevel@apache.org):
Chris, here are the errors I saw on a parallel HADOOP-13171 run
{code}
Running org.apache.hadoop.fs.s3a.TestS3ABlockingThreadPool
Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 25.167 sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.TestS3ABlockingThreadPool
testRegularMultiPartUpload(org.apache.hadoop.fs.s3a.TestS3ABlockingThreadPool) Time elapsed: 13.218 sec <<< ERROR!
java.io.FileNotFoundException: saving output on tests3a/97e97bdb-5ca4-46b8-a3d2-7dbb2100101b: com.amazonaws.services.s3.model.AmazonS3Exception: The specified upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchUpload; Request ID: 0A0C0CA457261DB8), S3 Extended Request ID: kLavwGlMzkH7dm1uwbCQ1N6fK8HfUOi/PkNf4lSE7rXRQVmLkkqE6Nd86n2wu8UOZy8jdXK7+TY=
at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:106)
at org.apache.hadoop.fs.s3a.S3AOutputStream.close(S3AOutputStream.java:123)
at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101)
at org.apache.hadoop.fs.contract.ContractTestUtils.generateTestFile(ContractTestUtils.java:864)
at org.apache.hadoop.fs.contract.ContractTestUtils.createAndVerifyFile(ContractTestUtils.java:892)
at org.apache.hadoop.fs.s3a.TestS3ABlockingThreadPool.testRegularMultiPartUpload(TestS3ABlockingThreadPool.java:68)
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: The specified upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchUpload; Request ID: 0A0C0CA457261DB8)
at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1182)
at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:770)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3785)
at com.amazonaws.services.s3.AmazonS3Client.completeMultipartUpload(AmazonS3Client.java:2705)
at com.amazonaws.services.s3.transfer.internal.CompleteMultipartUpload.call(CompleteMultipartUpload.java:91)
at com.amazonaws.services.s3.transfer.internal.CompleteMultipartUpload.call(CompleteMultipartUpload.java:38)
at org.apache.hadoop.fs.s3a.BlockingThreadPoolExecutorService$CallableWithPermitRelease.call(BlockingThreadPoolExecutorService.java:239)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
testFastMultiPartUpload(org.apache.hadoop.fs.s3a.TestS3ABlockingThreadPool) Time elapsed: 11.866 sec <<< ERROR!
java.io.FileNotFoundException: Completing multi-part upload on tests3a/77dd6617-f0ac-45d4-bd3f-3475c07bb002: com.amazonaws.services.s3.model.AmazonS3Exception: The specified upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchUpload; Request ID: C8FD35777E51FF7C), S3 Extended Request ID: MZ8VyewWz4/qUu5h/MTDv+F7q9co2fhUkOCLxSt9Z5mXkO8vimUGQW9Fizvm0hkG82IPuvaiWaI=
at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1182)
at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:770)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3785)
at com.amazonaws.services.s3.AmazonS3Client.completeMultipartUpload(AmazonS3Client.java:2705)
at org.apache.hadoop.fs.s3a.S3AFastOutputStream$MultiPartUpload.complete(S3AFastOutputStream.java:374)
at org.apache.hadoop.fs.s3a.S3AFastOutputStream$MultiPartUpload.access$200(S3AFastOutputStream.java:319)
at org.apache.hadoop.fs.s3a.S3AFastOutputStream.close(S3AFastOutputStream.java:255)
at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101)
at org.apache.hadoop.fs.contract.ContractTestUtils.generateTestFile(ContractTestUtils.java:864)
at org.apache.hadoop.fs.contract.ContractTestUtils.createAndVerifyFile(ContractTestUtils.java:892)
at org.apache.hadoop.fs.s3a.TestS3ABlockingThreadPool.testFastMultiPartUpload(TestS3ABlockingThreadPool.java:76)
Running org.apache.hadoop.fs.s3a.TestS3ABlocksize
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.275 sec - in org.apache.hadoop.fs.s3a
{cpde}
> Isolate test path used by a few S3A tests for more reliable parallel execution.
> -------------------------------------------------------------------------------
>
> Key: HADOOP-13231
> URL: https://issues.apache.org/jira/browse/HADOOP-13231
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
> Reporter: Chris Nauroth
> Assignee: Chris Nauroth
> Priority: Minor
>
> I have noticed a few more spots in S3A tests that do not make use of the isolated test directory path when running in parallel mode. While I don't have any evidence that this is really causing problems for parallel test runs right now, it would still be good practice to clean these up.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org