You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by guxiaobo1982 <gu...@qq.com> on 2014/12/24 05:00:33 UTC
How to build Spark against the latest
Hi,
The official pom.xml file only have profile for hadoop version 2.4 as the latest version, but I installed hadoop version 2.6.0 with ambari, how can I build spark against it, just using mvn -Dhadoop.version=2.6.0, or how to make a coresponding profile for it?
Regards,
Xiaobo
Re: How to build Spark against the latest
Posted by Sean Owen <so...@cloudera.com>.
Yes, it is just a warning but it can be ignored unless you are running old
Java 6 at runtime too.
On Dec 27, 2014 3:11 PM, "Ted Yu" <yu...@gmail.com> wrote:
> In make-distribution.sh, there is following check of Java version:
>
> if [[ ! "$JAVA_VERSION" =~ "1.6" && -z "$SKIP_JAVA_TEST" ]]; then
> echo "***NOTE***: JAVA_HOME is not set to a JDK 6 installation. The
> resulting"
>
> FYI
>
> On Sat, Dec 27, 2014 at 1:31 AM, Sean Owen <so...@cloudera.com> wrote:
>
>> Why do you need to skip java tests? I build the distro just fine with
>> Java 8.
>> On Dec 27, 2014 4:21 AM, "Ted Yu" <yu...@gmail.com> wrote:
>>
>>> In case jdk 1.7 or higher is used to build, --skip-java-test needs to be
>>> specifed.
>>>
>>> FYI
>>>
>>> On Thu, Dec 25, 2014 at 5:03 PM, guxiaobo1982 <gu...@qq.com>
>>> wrote:
>>>
>>>> The following command works
>>>>
>>>> ./make-distribution.sh --tgz -Pyarn -Dyarn.version=2.6.0 -Phadoop-2.4
>>>> -Dhadoop.version=2.6.0 -Phive -DskipTests
>>>>
>>>> ------------------ Original ------------------
>>>> *From: * "guxiaobo1982";<gu...@qq.com>;
>>>> *Send time:* Thursday, Dec 25, 2014 3:58 PM
>>>> *To:* ""<gu...@qq.com>; "Ted Yu"<yu...@gmail.com>;
>>>> *Cc:* "user@spark.apache.org"<us...@spark.apache.org>;
>>>> *Subject: * Re: How to build Spark against the latest
>>>>
>>>>
>>>> What options should I use when running the make-distribution.sh script,
>>>>
>>>> I tried ./make-distribution.sh --hadoop.version 2.6.0 --with-yarn
>>>> -with-hive --with-tachyon --tgz
>>>> with nothing came out.
>>>>
>>>> Regards
>>>>
>>>> ------------------ Original ------------------
>>>> *From: * "guxiaobo1982";<gu...@qq.com>;
>>>> *Send time:* Wednesday, Dec 24, 2014 6:52 PM
>>>> *To:* "Ted Yu"<yu...@gmail.com>;
>>>> *Cc:* "user@spark.apache.org"<us...@spark.apache.org>;
>>>> *Subject: * Re: How to build Spark against the latest
>>>>
>>>> Hi Ted,
>>>> The reference command works, but where I can get the deployable
>>>> binaries?
>>>>
>>>> Xiaobo Gu
>>>>
>>>>
>>>>
>>>>
>>>> ------------------ Original ------------------
>>>> *From: * "Ted Yu";<yu...@gmail.com>;
>>>> *Send time:* Wednesday, Dec 24, 2014 12:09 PM
>>>> *To:* ""<gu...@qq.com>;
>>>> *Cc:* "user@spark.apache.org"<us...@spark.apache.org>;
>>>> *Subject: * Re: How to build Spark against the latest
>>>>
>>>> See http://search-hadoop.com/m/JW1q5Cew0j
>>>>
>>>> On Tue, Dec 23, 2014 at 8:00 PM, guxiaobo1982 <gu...@qq.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>> The official pom.xml file only have profile for hadoop version 2.4 as
>>>>> the latest version, but I installed hadoop version 2.6.0 with ambari, how
>>>>> can I build spark against it, just using mvn -Dhadoop.version=2.6.0,
>>>>> or how to make a coresponding profile for it?
>>>>>
>>>>> Regards,
>>>>>
>>>>> Xiaobo
>>>>>
>>>>
>>>>
>>>
>
Re: How to build Spark against the latest
Posted by Ted Yu <yu...@gmail.com>.
In make-distribution.sh, there is following check of Java version:
if [[ ! "$JAVA_VERSION" =~ "1.6" && -z "$SKIP_JAVA_TEST" ]]; then
echo "***NOTE***: JAVA_HOME is not set to a JDK 6 installation. The
resulting"
FYI
On Sat, Dec 27, 2014 at 1:31 AM, Sean Owen <so...@cloudera.com> wrote:
> Why do you need to skip java tests? I build the distro just fine with Java
> 8.
> On Dec 27, 2014 4:21 AM, "Ted Yu" <yu...@gmail.com> wrote:
>
>> In case jdk 1.7 or higher is used to build, --skip-java-test needs to be
>> specifed.
>>
>> FYI
>>
>> On Thu, Dec 25, 2014 at 5:03 PM, guxiaobo1982 <gu...@qq.com>
>> wrote:
>>
>>> The following command works
>>>
>>> ./make-distribution.sh --tgz -Pyarn -Dyarn.version=2.6.0 -Phadoop-2.4
>>> -Dhadoop.version=2.6.0 -Phive -DskipTests
>>>
>>> ------------------ Original ------------------
>>> *From: * "guxiaobo1982";<gu...@qq.com>;
>>> *Send time:* Thursday, Dec 25, 2014 3:58 PM
>>> *To:* ""<gu...@qq.com>; "Ted Yu"<yu...@gmail.com>;
>>> *Cc:* "user@spark.apache.org"<us...@spark.apache.org>;
>>> *Subject: * Re: How to build Spark against the latest
>>>
>>>
>>> What options should I use when running the make-distribution.sh script,
>>>
>>> I tried ./make-distribution.sh --hadoop.version 2.6.0 --with-yarn
>>> -with-hive --with-tachyon --tgz
>>> with nothing came out.
>>>
>>> Regards
>>>
>>> ------------------ Original ------------------
>>> *From: * "guxiaobo1982";<gu...@qq.com>;
>>> *Send time:* Wednesday, Dec 24, 2014 6:52 PM
>>> *To:* "Ted Yu"<yu...@gmail.com>;
>>> *Cc:* "user@spark.apache.org"<us...@spark.apache.org>;
>>> *Subject: * Re: How to build Spark against the latest
>>>
>>> Hi Ted,
>>> The reference command works, but where I can get the deployable
>>> binaries?
>>>
>>> Xiaobo Gu
>>>
>>>
>>>
>>>
>>> ------------------ Original ------------------
>>> *From: * "Ted Yu";<yu...@gmail.com>;
>>> *Send time:* Wednesday, Dec 24, 2014 12:09 PM
>>> *To:* ""<gu...@qq.com>;
>>> *Cc:* "user@spark.apache.org"<us...@spark.apache.org>;
>>> *Subject: * Re: How to build Spark against the latest
>>>
>>> See http://search-hadoop.com/m/JW1q5Cew0j
>>>
>>> On Tue, Dec 23, 2014 at 8:00 PM, guxiaobo1982 <gu...@qq.com>
>>> wrote:
>>>
>>>> Hi,
>>>> The official pom.xml file only have profile for hadoop version 2.4 as
>>>> the latest version, but I installed hadoop version 2.6.0 with ambari, how
>>>> can I build spark against it, just using mvn -Dhadoop.version=2.6.0,
>>>> or how to make a coresponding profile for it?
>>>>
>>>> Regards,
>>>>
>>>> Xiaobo
>>>>
>>>
>>>
>>
Re: How to build Spark against the latest
Posted by Sean Owen <so...@cloudera.com>.
Why do you need to skip java tests? I build the distro just fine with Java
8.
On Dec 27, 2014 4:21 AM, "Ted Yu" <yu...@gmail.com> wrote:
> In case jdk 1.7 or higher is used to build, --skip-java-test needs to be
> specifed.
>
> FYI
>
> On Thu, Dec 25, 2014 at 5:03 PM, guxiaobo1982 <gu...@qq.com> wrote:
>
>> The following command works
>>
>> ./make-distribution.sh --tgz -Pyarn -Dyarn.version=2.6.0 -Phadoop-2.4
>> -Dhadoop.version=2.6.0 -Phive -DskipTests
>>
>> ------------------ Original ------------------
>> *From: * "guxiaobo1982";<gu...@qq.com>;
>> *Send time:* Thursday, Dec 25, 2014 3:58 PM
>> *To:* ""<gu...@qq.com>; "Ted Yu"<yu...@gmail.com>;
>> *Cc:* "user@spark.apache.org"<us...@spark.apache.org>;
>> *Subject: * Re: How to build Spark against the latest
>>
>>
>> What options should I use when running the make-distribution.sh script,
>>
>> I tried ./make-distribution.sh --hadoop.version 2.6.0 --with-yarn
>> -with-hive --with-tachyon --tgz
>> with nothing came out.
>>
>> Regards
>>
>> ------------------ Original ------------------
>> *From: * "guxiaobo1982";<gu...@qq.com>;
>> *Send time:* Wednesday, Dec 24, 2014 6:52 PM
>> *To:* "Ted Yu"<yu...@gmail.com>;
>> *Cc:* "user@spark.apache.org"<us...@spark.apache.org>;
>> *Subject: * Re: How to build Spark against the latest
>>
>> Hi Ted,
>> The reference command works, but where I can get the deployable
>> binaries?
>>
>> Xiaobo Gu
>>
>>
>>
>>
>> ------------------ Original ------------------
>> *From: * "Ted Yu";<yu...@gmail.com>;
>> *Send time:* Wednesday, Dec 24, 2014 12:09 PM
>> *To:* ""<gu...@qq.com>;
>> *Cc:* "user@spark.apache.org"<us...@spark.apache.org>;
>> *Subject: * Re: How to build Spark against the latest
>>
>> See http://search-hadoop.com/m/JW1q5Cew0j
>>
>> On Tue, Dec 23, 2014 at 8:00 PM, guxiaobo1982 <gu...@qq.com>
>> wrote:
>>
>>> Hi,
>>> The official pom.xml file only have profile for hadoop version 2.4 as
>>> the latest version, but I installed hadoop version 2.6.0 with ambari, how
>>> can I build spark against it, just using mvn -Dhadoop.version=2.6.0, or
>>> how to make a coresponding profile for it?
>>>
>>> Regards,
>>>
>>> Xiaobo
>>>
>>
>>
>
Re: How to build Spark against the latest
Posted by Ted Yu <yu...@gmail.com>.
In case jdk 1.7 or higher is used to build, --skip-java-test needs to be
specifed.
FYI
On Thu, Dec 25, 2014 at 5:03 PM, guxiaobo1982 <gu...@qq.com> wrote:
> The following command works
>
> ./make-distribution.sh --tgz -Pyarn -Dyarn.version=2.6.0 -Phadoop-2.4
> -Dhadoop.version=2.6.0 -Phive -DskipTests
>
> ------------------ Original ------------------
> *From: * "guxiaobo1982";<gu...@qq.com>;
> *Send time:* Thursday, Dec 25, 2014 3:58 PM
> *To:* ""<gu...@qq.com>; "Ted Yu"<yu...@gmail.com>;
> *Cc:* "user@spark.apache.org"<us...@spark.apache.org>;
> *Subject: * Re: How to build Spark against the latest
>
>
> What options should I use when running the make-distribution.sh script,
>
> I tried ./make-distribution.sh --hadoop.version 2.6.0 --with-yarn
> -with-hive --with-tachyon --tgz
> with nothing came out.
>
> Regards
>
> ------------------ Original ------------------
> *From: * "guxiaobo1982";<gu...@qq.com>;
> *Send time:* Wednesday, Dec 24, 2014 6:52 PM
> *To:* "Ted Yu"<yu...@gmail.com>;
> *Cc:* "user@spark.apache.org"<us...@spark.apache.org>;
> *Subject: * Re: How to build Spark against the latest
>
> Hi Ted,
> The reference command works, but where I can get the deployable
> binaries?
>
> Xiaobo Gu
>
>
>
>
> ------------------ Original ------------------
> *From: * "Ted Yu";<yu...@gmail.com>;
> *Send time:* Wednesday, Dec 24, 2014 12:09 PM
> *To:* ""<gu...@qq.com>;
> *Cc:* "user@spark.apache.org"<us...@spark.apache.org>;
> *Subject: * Re: How to build Spark against the latest
>
> See http://search-hadoop.com/m/JW1q5Cew0j
>
> On Tue, Dec 23, 2014 at 8:00 PM, guxiaobo1982 <gu...@qq.com> wrote:
>
>> Hi,
>> The official pom.xml file only have profile for hadoop version 2.4 as the
>> latest version, but I installed hadoop version 2.6.0 with ambari, how can I
>> build spark against it, just using mvn -Dhadoop.version=2.6.0, or how to
>> make a coresponding profile for it?
>>
>> Regards,
>>
>> Xiaobo
>>
>
>
Re: How to build Spark against the latest
Posted by guxiaobo1982 <gu...@qq.com>.
The following command works
./make-distribution.sh --tgz -Pyarn -Dyarn.version=2.6.0 -Phadoop-2.4 -Dhadoop.version=2.6.0 -Phive -DskipTests
------------------ Original ------------------
From: "guxiaobo1982";<gu...@qq.com>;
Send time: Thursday, Dec 25, 2014 3:58 PM
To: ""<gu...@qq.com>; "Ted Yu"<yu...@gmail.com>;
Cc: "user@spark.apache.org"<us...@spark.apache.org>;
Subject: Re: How to build Spark against the latest
What options should I use when running the make-distribution.sh script,
I tried ./make-distribution.sh --hadoop.version 2.6.0 --with-yarn -with-hive --with-tachyon --tgz
with nothing came out.
Regards
------------------ Original ------------------
From: "guxiaobo1982";<gu...@qq.com>;
Send time: Wednesday, Dec 24, 2014 6:52 PM
To: "Ted Yu"<yu...@gmail.com>;
Cc: "user@spark.apache.org"<us...@spark.apache.org>;
Subject: Re: How to build Spark against the latest
Hi Ted,
The reference command works, but where I can get the deployable binaries?
Xiaobo Gu
------------------ Original ------------------
From: "Ted Yu";<yu...@gmail.com>;
Send time: Wednesday, Dec 24, 2014 12:09 PM
To: ""<gu...@qq.com>;
Cc: "user@spark.apache.org"<us...@spark.apache.org>;
Subject: Re: How to build Spark against the latest
See http://search-hadoop.com/m/JW1q5Cew0j
On Tue, Dec 23, 2014 at 8:00 PM, guxiaobo1982 <gu...@qq.com> wrote:
Hi,
The official pom.xml file only have profile for hadoop version 2.4 as the latest version, but I installed hadoop version 2.6.0 with ambari, how can I build spark against it, just using mvn -Dhadoop.version=2.6.0, or how to make a coresponding profile for it?
Regards,
Xiaobo
Re: How to build Spark against the latest
Posted by guxiaobo1982 <gu...@qq.com>.
What options should I use when running the make-distribution.sh script,
I tried ./make-distribution.sh --hadoop.version 2.6.0 --with-yarn -with-hive --with-tachyon --tgz
with nothing came out.
Regards
------------------ Original ------------------
From: "guxiaobo1982";<gu...@qq.com>;
Send time: Wednesday, Dec 24, 2014 6:52 PM
To: "Ted Yu"<yu...@gmail.com>;
Cc: "user@spark.apache.org"<us...@spark.apache.org>;
Subject: Re: How to build Spark against the latest
Hi Ted,
The reference command works, but where I can get the deployable binaries?
Xiaobo Gu
------------------ Original ------------------
From: "Ted Yu";<yu...@gmail.com>;
Send time: Wednesday, Dec 24, 2014 12:09 PM
To: ""<gu...@qq.com>;
Cc: "user@spark.apache.org"<us...@spark.apache.org>;
Subject: Re: How to build Spark against the latest
See http://search-hadoop.com/m/JW1q5Cew0j
On Tue, Dec 23, 2014 at 8:00 PM, guxiaobo1982 <gu...@qq.com> wrote:
Hi,
The official pom.xml file only have profile for hadoop version 2.4 as the latest version, but I installed hadoop version 2.6.0 with ambari, how can I build spark against it, just using mvn -Dhadoop.version=2.6.0, or how to make a coresponding profile for it?
Regards,
Xiaobo
Re: How to build Spark against the latest
Posted by guxiaobo1982 <gu...@qq.com>.
Hi Ted,
The reference command works, but where I can get the deployable binaries?
Xiaobo Gu
------------------ Original ------------------
From: "Ted Yu";<yu...@gmail.com>;
Send time: Wednesday, Dec 24, 2014 12:09 PM
To: ""<gu...@qq.com>;
Cc: "user@spark.apache.org"<us...@spark.apache.org>;
Subject: Re: How to build Spark against the latest
See http://search-hadoop.com/m/JW1q5Cew0j
On Tue, Dec 23, 2014 at 8:00 PM, guxiaobo1982 <gu...@qq.com> wrote:
Hi,
The official pom.xml file only have profile for hadoop version 2.4 as the latest version, but I installed hadoop version 2.6.0 with ambari, how can I build spark against it, just using mvn -Dhadoop.version=2.6.0, or how to make a coresponding profile for it?
Regards,
Xiaobo
Re: How to build Spark against the latest
Posted by Ted Yu <yu...@gmail.com>.
See http://search-hadoop.com/m/JW1q5Cew0j
On Tue, Dec 23, 2014 at 8:00 PM, guxiaobo1982 <gu...@qq.com> wrote:
> Hi,
> The official pom.xml file only have profile for hadoop version 2.4 as the
> latest version, but I installed hadoop version 2.6.0 with ambari, how can I
> build spark against it, just using mvn -Dhadoop.version=2.6.0, or how to
> make a coresponding profile for it?
>
> Regards,
>
> Xiaobo
>