You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@spark.apache.org by Zoltán Zvara <zo...@gmail.com> on 2015/03/25 09:45:25 UTC

Can't assembly YARN project with SBT

Hi!

I'm using the latest IntelliJ and I can't compile the yarn project into the
Spark assembly fat JAR. That is why I'm getting a SparkException with
message "Unable to load YARN support". The yarn project is also missing
from SBT tasks and I can't add it. How can I force sbt to include?

Thanks!

Zvara Zoltán



mail, hangout, skype: zoltan.zvara@gmail.com

mobile, viber: +36203129543

bank: 10918001-00000021-50480008

address: Hungary, 2475 Kápolnásnyék, Kossuth 6/a

elte: HSKSJZ (ZVZOAAI.ELTE)

Re: Can't assembly YARN project with SBT

Posted by Zoltán Zvara <zo...@gmail.com>.

Hi!

It seems that the problem of "unable to load YARN support" present only
when I run my job from code and by not using the spark-submit script. IMO
this is related to SPARK-5144
<https://issues.apache.org/jira/browse/SPARK-5144>. I'm running QueueStream
example with a single change: sparkConf.setMaster("yarn-client").

I'm building Spark with: sbt/sbt -Pyarn -Dhadoop.version=2.6.0 assembly

Zvara Zoltán



mail, hangout, skype: zoltan.zvara@gmail.com

mobile, viber: +36203129543

bank: 10918001-00000021-50480008

address: Hungary, 2475 Kápolnásnyék, Kossuth 6/a

elte: HSKSJZ (ZVZOAAI.ELTE)

2015-03-25 9:45 GMT+01:00 Zoltán Zvara <zo...@gmail.com>:

> Hi!
>
> I'm using the latest IntelliJ and I can't compile the yarn project into
> the Spark assembly fat JAR. That is why I'm getting a SparkException with
> message "Unable to load YARN support". The yarn project is also missing
> from SBT tasks and I can't add it. How can I force sbt to include?
>
> Thanks!
>
> Zvara Zoltán
>
>
>
> mail, hangout, skype: zoltan.zvara@gmail.com
>
> mobile, viber: +36203129543
>
> bank: 10918001-00000021-50480008
>
> address: Hungary, 2475 Kápolnásnyék, Kossuth 6/a
>
> elte: HSKSJZ (ZVZOAAI.ELTE)
>