[jira] [Commented] (HADOOP-15553) DOCO - not clear how to integrate hadoop-aws and aws-sdk jars with S3A

["Steve Loughran (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-15553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16519634#comment-16519634 ] 

Steve Loughran commented on HADOOP-15553:
-----------------------------------------

I agree, we can improve this.

The latest versions call out the key issue: don't try and mix AWS SDK and hadoop-aws versions, or try to drop in a newhadoop-aws JAR. .This will only [lead to stack traces}https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/troubleshooting_s3a.md].

What we haven't done is ever do that big table of versioning, [Mvnrepositoryhttp://mvnrepository.com/artifact/org.apache.hadoop/hadoop-aws] is the best place to look there: it is guaranteed to be correct for all versions. 

Happy to take a patch to the hadoop-aws docs to point to it, and maybe pull out the warning about version mixing to being on the core index page

> DOCO - not clear how to integrate hadoop-aws and aws-sdk jars with S3A
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-15553
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15553
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: documentation, fs/s3
>    Affects Versions: 2.8.3
>            Reporter: t oo
>            Priority: Major
>
> Documentation update, a compatibility matrix would be nice, especially since upgrading to new aws java sdk does that work :
>  
>  On a hadoop common 2.7.5 cluster,  I take AWS Java SDK [1.7.5|https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk/1.7.5]  and Hadoop-aws [2.7.5|https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-aws/2.7.5] and use them in my Spark app writing to S3a, this is fine.
> On another cluster (this time hadoop common 2.8.3 cluster) I take AWS Java SDK 1.10.6  and Hadoop-aws 2.8.3 and try to use them in my Spark app writing to S3a, but get errors about missing classes. Turns out this version of aws java sdk is tiny with basically nothing inside.
> Couldn't find any documentation helping users on this incompatibility.
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org