[jira] [Resolved] (SPARK-7870) Publish jars against hadoop 1 (default) too

["Sean Owen (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-7870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sean Owen resolved SPARK-7870.
------------------------------
    Resolution: Not A Problem

The published JARs in Maven can only show a dependency on one particular Hadoop version no matter what. However, you aren't intended to rely on the Hadoop bindings of Spark, as the Spark dependency is always supposed to be {{provided}}. So, if you build and deploy Spark for Hadoop 1 on your cluster, and build your app against the Spark APIs (which have no Hadoop dependency per se), the app should run correctly already.

> Publish jars against hadoop 1 (default) too
> -------------------------------------------
>
>                 Key: SPARK-7870
>                 URL: https://issues.apache.org/jira/browse/SPARK-7870
>             Project: Spark
>          Issue Type: Bug
>          Components: Build
>    Affects Versions: 1.3.2, 1.4.0
>            Reporter: Andy Petrella
>              Labels: hadoop-1.0, hadoop-2.0, jars, publish
>
> The published jars were always (or at least since 1.1.0, see SPARK-3764) published using hadoop 2 binaries.
> However, the default are hadoop 1. Which means that an application that doesn't use hdfs will fail at simple functions like `saveAsObjectFile` (since it uses hadoop interfaces).
> There are a couple of points that can be considered wrong here:
> * it's not well stated that the published binaries aren't what you can expect base on spark rules w.r.t hadoop 1.0.4
> * if an application doesn't need hdfs, the dev couldn't rely on public libraries, but he has to build it's own set without asking for a specific hadoop version (hence will get the 1.0.4) and deploy it on his own :-/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org