You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Alessandro Baretta <al...@gmail.com> on 2014/12/27 03:15:45 UTC
Assembly jar file name does not match profile selection
I am building spark with sbt off of branch 1.2. I'm using the following
command:
sbt/sbt -Pyarn -Phadoop-2.3 assembly
(http://spark.apache.org/docs/latest/building-spark.html#building-with-sbt)
Although the jar file I obtain does contain the proper version of the
hadoop libraries (v. 2.4), the assembly jar file name refers to hadoop
v.1.0.4:
./assembly/target/scala-2.10/spark-assembly-1.3.0-SNAPSHOT-hadoop1.0.4.jar
Any idea why?
Alex
Re: Assembly jar file name does not match profile selection
Posted by Kousuke Saruta <sa...@oss.nttdata.co.jp>.
Hi Alessandro,
It's fixed by SPARK-3787 and will be applied to 1.2.1 and 1.3.0.
https://issues.apache.org/jira/browse/SPARK-3787
- Kousuke
(2014/12/27 11:15), Alessandro Baretta wrote:
> I am building spark with sbt off of branch 1.2. I'm using the following
> command:
>
> sbt/sbt -Pyarn -Phadoop-2.3 assembly
>
> (http://spark.apache.org/docs/latest/building-spark.html#building-with-sbt)
>
> Although the jar file I obtain does contain the proper version of the
> hadoop libraries (v. 2.4), the assembly jar file name refers to hadoop
> v.1.0.4:
>
> ./assembly/target/scala-2.10/spark-assembly-1.3.0-SNAPSHOT-hadoop1.0.4.jar
>
> Any idea why?
>
>
> Alex
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: Assembly jar file name does not match profile selection
Posted by Alessandro Baretta <al...@gmail.com>.
Here's what I get:
./assembly/target/scala-2.10/spark-assembly-1.3.0-SNAPSHOT-hadoop2.6.0.jar
Alex
On Fri, Dec 26, 2014 at 8:41 PM, Ted Yu <yu...@gmail.com> wrote:
> Can you try this command ?
>
> sbt/sbt -Pyarn -Phadoop-2.4 -Dhadoop.version=2.6.0 -Phive assembly
>
> On Fri, Dec 26, 2014 at 6:15 PM, Alessandro Baretta <alexbaretta@gmail.com
> > wrote:
>
>> I am building spark with sbt off of branch 1.2. I'm using the following
>> command:
>>
>> sbt/sbt -Pyarn -Phadoop-2.3 assembly
>>
>> (
>> http://spark.apache.org/docs/latest/building-spark.html#building-with-sbt
>> )
>>
>> Although the jar file I obtain does contain the proper version of the
>> hadoop libraries (v. 2.4), the assembly jar file name refers to hadoop
>> v.1.0.4:
>>
>> ./assembly/target/scala-2.10/spark-assembly-1.3.0-SNAPSHOT-hadoop1.0.4.jar
>>
>> Any idea why?
>>
>>
>> Alex
>>
>
>
Re: Assembly jar file name does not match profile selection
Posted by Ted Yu <yu...@gmail.com>.
Can you try this command ?
sbt/sbt -Pyarn -Phadoop-2.4 -Dhadoop.version=2.6.0 -Phive assembly
On Fri, Dec 26, 2014 at 6:15 PM, Alessandro Baretta <al...@gmail.com>
wrote:
> I am building spark with sbt off of branch 1.2. I'm using the following
> command:
>
> sbt/sbt -Pyarn -Phadoop-2.3 assembly
>
> (http://spark.apache.org/docs/latest/building-spark.html#building-with-sbt
> )
>
> Although the jar file I obtain does contain the proper version of the
> hadoop libraries (v. 2.4), the assembly jar file name refers to hadoop
> v.1.0.4:
>
> ./assembly/target/scala-2.10/spark-assembly-1.3.0-SNAPSHOT-hadoop1.0.4.jar
>
> Any idea why?
>
>
> Alex
>