You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Rafal Kwasny (JIRA)" <ji...@apache.org> on 2015/01/21 16:02:35 UTC
[jira] [Created] (SPARK-5348) s3a:// protocol and hadoop-aws
dependency
Rafal Kwasny created SPARK-5348:
-----------------------------------
Summary: s3a:// protocol and hadoop-aws dependency
Key: SPARK-5348
URL: https://issues.apache.org/jira/browse/SPARK-5348
Project: Spark
Issue Type: Improvement
Affects Versions: 1.2.0
Reporter: Rafal Kwasny
Priority: Minor
Hadoop 2.6.0+ includes new s3a library which is a lot better than the old one.
https://issues.apache.org/jira/browse/HADOOP-10400
Unfortunately this library is linking with amazon sdk which I think is not compatible with apache licensing. For this reason it is not a dependency for hadoop-client.
What I did is just add a profile for maven build, not sure if there is a better option to integrate this, we could also add a new hadoop-2.6 profile.
{noformat}
diff --git a/pom.xml b/pom.xml
index a9e7c9d..71ee776 100644
--- a/pom.xml
+++ b/pom.xml
@@ -1214,6 +1214,27 @@
</profile>
<profile>
+ <id>hadoop-aws</id>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-aws</artifactId>
+ <version>${hadoop.version}</version>
+ <exclusions>
+ <exclusion>
+ <groupId>com.fasterxml.jackson.core</groupId>
+ <artifactId>jackson-annotations</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.fasterxml.jackson.core</groupId>
+ <artifactId>jackson-databind</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ </dependencies>
+ </profile>
+
+ <profile>
<id>java8-tests</id>
<build>
<plugins>
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org