You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by David McWhorter <mc...@ccri.com> on 2014/12/19 18:09:45 UTC
spark-yarn_2.10 1.2.0 artifacts
Hi all,
Thanks for your work on spark! I am trying to locate spark-yarn jars
for the new 1.2.0 release. The jars for spark-core, etc, are on maven
central, but the spark-yarn jars are missing.
Confusingly and perhaps relatedly, I also can't seem to get the
spark-yarn artifact to install on my local computer when I run 'mvn
-Pyarn -Phadoop-2.2 -Dhadoop.version=2.2.0 -DskipTests clean install'.
At the install plugin stage, maven reports:
[INFO] --- maven-install-plugin:2.5.1:install (default-install) @
spark-yarn_2.10 ---
[INFO] Skipping artifact installation
Any help or insights into how to use spark-yarn_2.10 1.2.0 in a maven
build would be appreciated.
David
--
David McWhorter
Software Engineer
Commonwealth Computer Research, Inc.
1422 Sachem Place, Unit #1
Charlottesville, VA 22901
mcwhorter@ccri.com | 434.299.0090x204
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: spark-yarn_2.10 1.2.0 artifacts
Posted by Jong Wook Kim <jo...@nyu.edu>.
I don't think that's the case.
spark-yarn contains `org.apache.spark.deploy.yarn` package, whereas
spark-network-yarn contains `org.apache.spark.network.yarn`, and they do
different things.
The former contains codes for deploying Spark applications to YARN cluster,
and called when running `spark-submit --master yarn...` and the latter
contains a shuffle service that runs in NM process.
While spark-yarn is not a programming interface and normally used only by
Spark toolchain, I need it as a library dependency, for programatically
launching spark application on YARN.
spark-yarn.jar does appear in '/yarn/stable/target` when building Spark
1.2.0 from source code and I'm manually adding that to my maven repository.
But I'd like to know why it is excluded from the maven repo.
--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/spark-yarn-2-10-1-2-0-artifacts-tp9853p10023.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: spark-yarn_2.10 1.2.0 artifacts
Posted by David McWhorter <mc...@ccri.com>.
Thank you, Sean, using spark-network-yarn seems to do the trick.
On 12/19/2014 12:13 PM, Sean Owen wrote:
> I believe spark-yarn does not exist from 1.2 onwards. Have a look at
> spark-network-yarn for where some of that went, I believe.
>
> On Fri, Dec 19, 2014 at 5:09 PM, David McWhorter <mc...@ccri.com> wrote:
>> Hi all,
>>
>> Thanks for your work on spark! I am trying to locate spark-yarn jars for
>> the new 1.2.0 release. The jars for spark-core, etc, are on maven central,
>> but the spark-yarn jars are missing.
>>
>> Confusingly and perhaps relatedly, I also can't seem to get the spark-yarn
>> artifact to install on my local computer when I run 'mvn -Pyarn -Phadoop-2.2
>> -Dhadoop.version=2.2.0 -DskipTests clean install'. At the install plugin
>> stage, maven reports:
>>
>> [INFO] --- maven-install-plugin:2.5.1:install (default-install) @
>> spark-yarn_2.10 ---
>> [INFO] Skipping artifact installation
>>
>> Any help or insights into how to use spark-yarn_2.10 1.2.0 in a maven build
>> would be appreciated.
>>
>> David
>>
>> --
>>
>> David McWhorter
>> Software Engineer
>> Commonwealth Computer Research, Inc.
>> 1422 Sachem Place, Unit #1
>> Charlottesville, VA 22901
>> mcwhorter@ccri.com | 434.299.0090x204
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>> For additional commands, e-mail: dev-help@spark.apache.org
>>
--
David McWhorter
Software Engineer
Commonwealth Computer Research, Inc.
1422 Sachem Place, Unit #1
Charlottesville, VA 22901
mcwhorter@ccri.com | 434.299.0090x204
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: spark-yarn_2.10 1.2.0 artifacts
Posted by Sean Owen <so...@cloudera.com>.
I believe spark-yarn does not exist from 1.2 onwards. Have a look at
spark-network-yarn for where some of that went, I believe.
On Fri, Dec 19, 2014 at 5:09 PM, David McWhorter <mc...@ccri.com> wrote:
> Hi all,
>
> Thanks for your work on spark! I am trying to locate spark-yarn jars for
> the new 1.2.0 release. The jars for spark-core, etc, are on maven central,
> but the spark-yarn jars are missing.
>
> Confusingly and perhaps relatedly, I also can't seem to get the spark-yarn
> artifact to install on my local computer when I run 'mvn -Pyarn -Phadoop-2.2
> -Dhadoop.version=2.2.0 -DskipTests clean install'. At the install plugin
> stage, maven reports:
>
> [INFO] --- maven-install-plugin:2.5.1:install (default-install) @
> spark-yarn_2.10 ---
> [INFO] Skipping artifact installation
>
> Any help or insights into how to use spark-yarn_2.10 1.2.0 in a maven build
> would be appreciated.
>
> David
>
> --
>
> David McWhorter
> Software Engineer
> Commonwealth Computer Research, Inc.
> 1422 Sachem Place, Unit #1
> Charlottesville, VA 22901
> mcwhorter@ccri.com | 434.299.0090x204
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org