You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Marcelo Vanzin (JIRA)" <ji...@apache.org> on 2016/02/01 00:27:39 UTC
[jira] [Commented] (SPARK-7481) Add Hadoop 2.6+ profile to pull in
object store FS accessors
[ https://issues.apache.org/jira/browse/SPARK-7481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15125566#comment-15125566 ]
Marcelo Vanzin commented on SPARK-7481:
---------------------------------------
It doesn't necessarily affect this proposal. It would make it easier to have people add these separately - just drop the jars in Spark's "lib" directory and suddenly they're part of Spark.
But if you don't add the dependency explicitly in Spark's build, they'll not be included in Spark's packaging, so there would still be a manual step to add support for those backends.
> Add Hadoop 2.6+ profile to pull in object store FS accessors
> ------------------------------------------------------------
>
> Key: SPARK-7481
> URL: https://issues.apache.org/jira/browse/SPARK-7481
> Project: Spark
> Issue Type: Improvement
> Components: Build
> Affects Versions: 1.3.1
> Reporter: Steve Loughran
>
> To keep the s3n classpath right, to add s3a, swift & azure, the dependencies of spark in a 2.6+ profile need to add the relevant object store packages (hadoop-aws, hadoop-openstack, hadoop-azure)
> this adds more stuff to the client bundle, but will mean a single spark package can talk to all of the stores.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org