You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Chris Nauroth (JIRA)" <ji...@apache.org> on 2016/04/26 00:16:13 UTC
[jira] [Commented] (HADOOP-13060) While trying to perform a Distcp
command, we see the error Exception in thread "main"
java.lang.NoSuchMethodError:
com.amazonaws.services.s3.transfer.TransferManager.(Lcom/amazonaws/services/s3/AmazonS3;Ljava/util/concurrent/ThreadPoolExecutor;)V
[ https://issues.apache.org/jira/browse/HADOOP-13060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15257154#comment-15257154 ]
Chris Nauroth commented on HADOOP-13060:
----------------------------------------
bq. We are using aws-java-sdk.jar version 1.10.69...
HADOOP-12269 changed the dependency from aws-java-sdk:1.7.4 to aws-java-sdk-s3:1.10.6. Can you try checking again that you were using "aws-java-sdk-s3" and not "aws-java-sdk"? We are successfully testing with aws-java-sdk-s3:1.10.6. This is potentially confusing because the names are so similar.
> While trying to perform a Distcp command, we see the error Exception in thread "main" java.lang.NoSuchMethodError: com.amazonaws.services.s3.transfer.TransferManager.<init>(Lcom/amazonaws/services/s3/AmazonS3;Ljava/util/concurrent/ThreadPoolExecutor;)V
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-13060
> URL: https://issues.apache.org/jira/browse/HADOOP-13060
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 2.7.1
> Reporter: Jenny Dong
>
> While trying to do a distcp from our native HDFS cluster to S3, we get the following error/stacktrace :
> We are using hadoop-aws.jar version 2.7.1. We are using aws-java-sdk.jar version 1.10.69 (we bumped this up from 2.7.4 because we were getting errors seen in HADOOP-12420 + other authentication errors).
> Exception in thread "main" java.lang.NoSuchMethodError: com.amazonaws.services.s3.transfer.TransferManager.<init>(Lcom/amazonaws/services/s3/AmazonS3;Ljava/util/concurrent/ThreadPoolExecutor;)V
> at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:287)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92)
> at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
> at org.apache.hadoop.tools.DistCp.configureOutputFormat(DistCp.java:333)
> at org.apache.hadoop.tools.DistCp.createJob(DistCp.java:237)
> at org.apache.hadoop.tools.DistCp.createAndSubmitJob(DistCp.java:174)
> at org.apache.hadoop.tools.DistCp.execute(DistCp.java:153)
> I dug into both classes com.amazonaws.services.s3.transfer.TransferManager & org.apache.hadoop.fs.s3a.S3AFileSystem. The only difference is S3AFileSystem created a ThreadPoolExecutor (which implements AbstractExecutorService which implements ExecutorService). I also checked on the classpath to make sure the version of the jars being picked up is what I expected.
> Help would be much appreciated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)