You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Srikrishna S <sr...@gmail.com> on 2014/07/08 05:07:28 UTC
Spark Installation
Hi All,
Does anyone know what the command line arguments to mvn are to generate the
pre-built binary for spark on Hadoop 2-CHD5.
I would like to pull in a recent bug fix in spark-master and rebuild the
binaries in the exact same way that was used for that provided on the
website.
I have tried the following:
mvn install -Pyarn -Dhadoop.version=2.3.0-cdh5.0.1
And it doesn't quite work.
Any thoughts anyone?
Re: Spark Installation
Posted by Sean Owen <so...@cloudera.com>.
On Tue, Jul 8, 2014 at 4:07 AM, Srikrishna S <sr...@gmail.com>
wrote:
> Hi All,
>
> Does anyone know what the command line arguments to mvn are to generate
> the pre-built binary for spark on Hadoop 2-CHD5.
>
> I would like to pull in a recent bug fix in spark-master and rebuild the
> binaries in the exact same way that was used for that provided on the
> website.
>
> I have tried the following:
>
> mvn install -Pyarn -Dhadoop.version=2.3.0-cdh5.0.1
>
> And it doesn't quite work.
>
It would be a lot more helpful to say what didn't work exactly.
Re: Spark Installation
Posted by Jaideep Dhok <ja...@inmobi.com>.
Hi Srikrishna,
You can use the make-distribution script in Spark to generate the binary.
Example - ./make-distribution.sh --tgz --hadoop HADOOP_VERSION
The above script calls maven, so you can look into it to get the exact mvn
command too.
Thanks,
Jaideep
On Tue, Jul 8, 2014 at 8:37 AM, Srikrishna S <sr...@gmail.com>
wrote:
> Hi All,
>
> Does anyone know what the command line arguments to mvn are to generate
> the pre-built binary for spark on Hadoop 2-CHD5.
>
> I would like to pull in a recent bug fix in spark-master and rebuild the
> binaries in the exact same way that was used for that provided on the
> website.
>
> I have tried the following:
>
> mvn install -Pyarn -Dhadoop.version=2.3.0-cdh5.0.1
>
> And it doesn't quite work.
>
> Any thoughts anyone?
>
>
--
_____________________________________________________________
The information contained in this communication is intended solely for the
use of the individual or entity to whom it is addressed and others
authorized to receive it. It may contain confidential or legally privileged
information. If you are not the intended recipient you are hereby notified
that any disclosure, copying, distribution or taking any action in reliance
on the contents of this information is strictly prohibited and may be
unlawful. If you have received this communication in error, please notify
us immediately by responding to this email and then delete it from your
system. The firm is neither liable for the proper and complete transmission
of the information contained in this communication nor for any delay in its
receipt.
Re: Spark Installation
Posted by 田毅 <ti...@asiainfo.com>.
Hi Srikrishna
the reason to this issue is you had uploaded assembly jar to HDFS twice.
paste your command could be better diagnosis
田毅
===========================================
橘云平台产品线
大数据产品部
亚信联创科技（中国）有限公司
手机：13910177261
电话：010－82166322
传真：010－82166617
Q Q：20057509
MSN：yi.tian@hotmail.com
地址：北京市海淀区东北旺西路10号院东区 亚信联创大厦
===========================================
在 2014年7月9日，上午3:03，Srikrishna S <sr...@gmail.com> 写道：
> Hi All,
>
>
> I tried the make distribution script and it worked well. I was able to
> compile the spark binary on our CDH5 cluster. Once I compiled Spark, I
> copied over the binaries in the dist folder to all the other machines
> in the cluster.
>
> However, I run into an issue while submit a job in yarn-client mode. I
> get an error message that says the following
> Resource file:/opt/spark/spark-1.0.0-bin-hadoop2/lib/spark-assembly-1.1.0-SNAPSHOT-hadoop2.3.0.jar"
> changed on src filesystem (expected 1404845211000, was 1404845404000)
>
> My end goal is to submit a job (that uses MLLib) in our Yarn cluster.
>
> Any thoughts anyone?
>
> Regards,
> Krishna
>
>
>
> On Tue, Jul 8, 2014 at 9:49 AM, Sandy Ryza <sa...@cloudera.com> wrote:
>>
>> Hi Srikrishna,
>>
>> The binaries are built with something like
>> mvn package -Pyarn -Dhadoop.version=2.3.0-cdh5.0.1 -Dyarn.version=2.3.0-cdh5.0.1
>>
>> -Sandy
>>
>>
>> On Tue, Jul 8, 2014 at 3:14 AM, 田毅 <ti...@asiainfo.com> wrote:
>>>
>>> try this command:
>>>
>>> make-distribution.sh --hadoop 2.3.0-cdh5.0.0 --with-yarn --with-hive
>>>
>>>
>>>
>>>
>>> 田毅
>>> ===========================================
>>> 橘云平台产品线
>>> 大数据产品部
>>> 亚信联创科技（中国）有限公司
>>> 手机：13910177261
>>> 电话：010－82166322
>>> 传真：010－82166617
>>> Q Q：20057509
>>> MSN：yi.tian@hotmail.com
>>> 地址：北京市海淀区东北旺西路10号院东区 亚信联创大厦
>>>
>>>
>>> ===========================================
>>>
>>> 在 2014年7月8日，上午11:53，Krishna Sankar <ks...@gmail.com> 写道：
>>>
>>> Couldn't find any reference of CDH in pom.xml - profiles or the hadoop.version.Am also wondering how the cdh compatible artifact was compiled.
>>> Cheers
>>> <k/>
>>>
>>>
>>> On Mon, Jul 7, 2014 at 8:07 PM, Srikrishna S <sr...@gmail.com> wrote:
>>>>
>>>> Hi All,
>>>>
>>>> Does anyone know what the command line arguments to mvn are to generate the pre-built binary for spark on Hadoop 2-CHD5.
>>>>
>>>> I would like to pull in a recent bug fix in spark-master and rebuild the binaries in the exact same way that was used for that provided on the website.
>>>>
>>>> I have tried the following:
>>>>
>>>> mvn install -Pyarn -Dhadoop.version=2.3.0-cdh5.0.1
>>>>
>>>> And it doesn't quite work.
>>>>
>>>> Any thoughts anyone?
>>>>
>>>
>>>
>>
>
Re: Spark Installation
Posted by Srikrishna S <sr...@gmail.com>.
Hi All,
I tried the make distribution script and it worked well. I was able to
compile the spark binary on our CDH5 cluster. Once I compiled Spark, I
copied over the binaries in the dist folder to all the other machines
in the cluster.
However, I run into an issue while submit a job in yarn-client mode. I
get an error message that says the following
Resource file:/opt/spark/spark-1.0.0-bin-hadoop2/lib/spark-assembly-1.1.0-SNAPSHOT-hadoop2.3.0.jar"
changed on src filesystem (expected 1404845211000, was 1404845404000)
My end goal is to submit a job (that uses MLLib) in our Yarn cluster.
Any thoughts anyone?
Regards,
Krishna
On Tue, Jul 8, 2014 at 9:49 AM, Sandy Ryza <sa...@cloudera.com> wrote:
>
> Hi Srikrishna,
>
> The binaries are built with something like
> mvn package -Pyarn -Dhadoop.version=2.3.0-cdh5.0.1 -Dyarn.version=2.3.0-cdh5.0.1
>
> -Sandy
>
>
> On Tue, Jul 8, 2014 at 3:14 AM, 田毅 <ti...@asiainfo.com> wrote:
>>
>> try this command:
>>
>> make-distribution.sh --hadoop 2.3.0-cdh5.0.0 --with-yarn --with-hive
>>
>>
>>
>>
>> 田毅
>> ===========================================
>> 橘云平台产品线
>> 大数据产品部
>> 亚信联创科技（中国）有限公司
>> 手机：13910177261
>> 电话：010－82166322
>> 传真：010－82166617
>> Q Q：20057509
>> MSN：yi.tian@hotmail.com
>> 地址：北京市海淀区东北旺西路10号院东区 亚信联创大厦
>>
>>
>> ===========================================
>>
>> 在 2014年7月8日，上午11:53，Krishna Sankar <ks...@gmail.com> 写道：
>>
>> Couldn't find any reference of CDH in pom.xml - profiles or the hadoop.version.Am also wondering how the cdh compatible artifact was compiled.
>> Cheers
>> <k/>
>>
>>
>> On Mon, Jul 7, 2014 at 8:07 PM, Srikrishna S <sr...@gmail.com> wrote:
>>>
>>> Hi All,
>>>
>>> Does anyone know what the command line arguments to mvn are to generate the pre-built binary for spark on Hadoop 2-CHD5.
>>>
>>> I would like to pull in a recent bug fix in spark-master and rebuild the binaries in the exact same way that was used for that provided on the website.
>>>
>>> I have tried the following:
>>>
>>> mvn install -Pyarn -Dhadoop.version=2.3.0-cdh5.0.1
>>>
>>> And it doesn't quite work.
>>>
>>> Any thoughts anyone?
>>>
>>
>>
>
Re: Spark Installation
Posted by Sandy Ryza <sa...@cloudera.com>.
Hi Srikrishna,
The binaries are built with something like
mvn package -Pyarn -Dhadoop.version=2.3.0-cdh5.0.1
-Dyarn.version=2.3.0-cdh5.0.1
-Sandy
On Tue, Jul 8, 2014 at 3:14 AM, 田毅 <ti...@asiainfo.com> wrote:
> try this command:
>
> make-distribution.sh --hadoop 2.3.0-cdh5.0.0 --with-yarn --with-hive
>
>
>
>
> 田毅
> ===========================================
> 橘云平台产品线
> 大数据产品部
> 亚信联创科技（中国）有限公司
> 手机：13910177261
> 电话：010－82166322
> 传真：010－82166617
> Q Q：20057509
> MSN：yi.tian@hotmail.com
> 地址：北京市海淀区东北旺西路10号院东区 亚信联创大厦
>
>
> ===========================================
>
> 在 2014年7月8日，上午11:53，Krishna Sankar <ks...@gmail.com> 写道：
>
> Couldn't find any reference of CDH in pom.xml - profiles or the
> hadoop.version.Am <http://hadoop.version.am/> also wondering how the cdh
> compatible artifact was compiled.
> Cheers
> <k/>
>
>
> On Mon, Jul 7, 2014 at 8:07 PM, Srikrishna S <sr...@gmail.com>
> wrote:
>
>> Hi All,
>>
>> Does anyone know what the command line arguments to mvn are to generate
>> the pre-built binary for spark on Hadoop 2-CHD5.
>>
>> I would like to pull in a recent bug fix in spark-master and rebuild the
>> binaries in the exact same way that was used for that provided on the
>> website.
>>
>> I have tried the following:
>>
>> mvn install -Pyarn -Dhadoop.version=2.3.0-cdh5.0.1
>>
>> And it doesn't quite work.
>>
>> Any thoughts anyone?
>>
>>
>
>
Re: Spark Installation
Posted by 田毅 <ti...@asiainfo.com>.
try this command:
make-distribution.sh --hadoop 2.3.0-cdh5.0.0 --with-yarn --with-hive
田毅
===========================================
橘云平台产品线
大数据产品部
亚信联创科技（中国）有限公司
手机：13910177261
电话：010－82166322
传真：010－82166617
Q Q：20057509
MSN：yi.tian@hotmail.com
地址：北京市海淀区东北旺西路10号院东区 亚信联创大厦
===========================================
在 2014年7月8日，上午11:53，Krishna Sankar <ks...@gmail.com> 写道：
> Couldn't find any reference of CDH in pom.xml - profiles or the hadoop.version.Am also wondering how the cdh compatible artifact was compiled.
> Cheers
> <k/>
>
>
> On Mon, Jul 7, 2014 at 8:07 PM, Srikrishna S <sr...@gmail.com> wrote:
> Hi All,
>
> Does anyone know what the command line arguments to mvn are to generate the pre-built binary for spark on Hadoop 2-CHD5.
>
> I would like to pull in a recent bug fix in spark-master and rebuild the binaries in the exact same way that was used for that provided on the website.
>
> I have tried the following:
>
> mvn install -Pyarn -Dhadoop.version=2.3.0-cdh5.0.1
>
> And it doesn't quite work.
>
> Any thoughts anyone?
>
>
Re: Spark Installation
Posted by Krishna Sankar <ks...@gmail.com>.
Couldn't find any reference of CDH in pom.xml - profiles or the
hadoop.version.Am also wondering how the cdh compatible artifact was
compiled.
Cheers
<k/>
On Mon, Jul 7, 2014 at 8:07 PM, Srikrishna S <sr...@gmail.com>
wrote:
> Hi All,
>
> Does anyone know what the command line arguments to mvn are to generate
> the pre-built binary for spark on Hadoop 2-CHD5.
>
> I would like to pull in a recent bug fix in spark-master and rebuild the
> binaries in the exact same way that was used for that provided on the
> website.
>
> I have tried the following:
>
> mvn install -Pyarn -Dhadoop.version=2.3.0-cdh5.0.1
>
> And it doesn't quite work.
>
> Any thoughts anyone?
>
>