You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "t oo (JIRA)" <ji...@apache.org> on 2018/06/21 12:12:00 UTC
[jira] [Created] (HADOOP-15553) DOCO - not clear how to integrate
hadoop-aws and aws-sdk jars with S3A
t oo created HADOOP-15553:
-----------------------------
Summary: DOCO - not clear how to integrate hadoop-aws and aws-sdk jars with S3A
Key: HADOOP-15553
URL: https://issues.apache.org/jira/browse/HADOOP-15553
Project: Hadoop Common
Issue Type: New Feature
Components: tools
Affects Versions: 2.8.3
Reporter: t oo
Documentation update, a compatibility matrix would be nice, especially since upgrading to new aws java sdk does that work :
On a hadoop common 2.7.5 cluster, I take AWS Java SDK [1.7.5|https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk/1.7.5] and Hadoop-aws [2.7.5|https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-aws/2.7.5] and use them in my Spark app writing to S3a, this is fine.
On another cluster (this time hadoop common 2.8.3 cluster) I take AWS Java SDK 1.10.6 and Hadoop-aws 2.8.3 and try to use them in my Spark app writing to S3a, but get errors about missing classes. Turns out this version of aws java sdk is tiny with basically nothing inside.
Couldn't find any documentation helping users on this incompatibility.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org