You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Yana Kadiyska <ya...@gmail.com> on 2015/10/26 16:34:30 UTC
Problem with make-distribution.sh
Hi folks,
building spark instructions (
http://spark.apache.org/docs/latest/building-spark.html) suggest that
./make-distribution.sh --name custom-spark --tgz -Phadoop-2.4 -Pyarn
should produce a distribution similar to the ones found on the "Downloads"
page.
I noticed that the tgz I built using the above command does not produce the
datanucleus jars which are included in the "boxed" spark distributions.
What is the best-practice advice here?
I would like my distribution to match the official one as closely as
possible.
Thanks
Re: Problem with make-distribution.sh
Posted by Yana Kadiyska <ya...@gmail.com>.
@Sean, here is where I think it's a little misleading (underlining is mine):
Building a Runnable Distribution
To create a *Spark distribution like those distributed by the Spark
Downloads <http://spark.apache.org/downloads.html> page*, and that is laid
out so as to be runnable, use make-distribution.sh in the project root
directory. It can be configured with Maven profile settings and so on like
the direct Maven build. Example:
Agreed that "like" doesn't necessarily imply "exactly the same". On the
other hand, if I go to the download page all I select is a hadoop version
and distribution, so it's not super-intuitive that -Phive was used to
produce these. I don't have a strong opinion on whether this should be a
fix to the script or the docs but now that it's bitten me twice I'm very
appreciative of either :)
Thanks
On Mon, Oct 26, 2015 at 1:29 PM, Sean Owen <so...@cloudera.com> wrote:
> I don't think the page suggests that gives you any of the tarballs on the
> downloads page, and -Phive does not by itself do so either.
>
> On Mon, Oct 26, 2015 at 4:58 PM, Ted Yu <yu...@gmail.com> wrote:
>
>> I logged SPARK-11318 with a PR.
>>
>> I verified that by adding -Phive the datanucleus jars are included:
>>
>> tar tzvf spark-1.6.0-SNAPSHOT-bin-custom-spark.tgz | grep datanucleus
>> -rw-r--r-- hbase/hadoop 1890075 2015-10-26 09:52
>> spark-1.6.0-SNAPSHOT-bin-custom-spark/lib/datanucleus-core-3.2.10.jar
>> -rw-r--r-- hbase/hadoop 339666 2015-10-26 09:52
>> spark-1.6.0-SNAPSHOT-bin-custom-spark/lib/datanucleus-api-jdo-3.2.6.jar
>> -rw-r--r-- hbase/hadoop 1809447 2015-10-26 09:52
>> spark-1.6.0-SNAPSHOT-bin-custom-spark/lib/datanucleus-rdbms-3.2.9.jar
>>
>> Cheers
>>
>> On Mon, Oct 26, 2015 at 8:52 AM, Yana Kadiyska <ya...@gmail.com>
>> wrote:
>>
>>> thank you so much! You are correct. This is the second time I've made
>>> this mistake :(
>>>
>>> On Mon, Oct 26, 2015 at 11:36 AM, java8964 <ja...@hotmail.com> wrote:
>>>
>>>> Maybe you need the Hive part?
>>>>
>>>> Yong
>>>>
>>>> ------------------------------
>>>> Date: Mon, 26 Oct 2015 11:34:30 -0400
>>>> Subject: Problem with make-distribution.sh
>>>> From: yana.kadiyska@gmail.com
>>>> To: user@spark.apache.org
>>>>
>>>>
>>>> Hi folks,
>>>>
>>>> building spark instructions (
>>>> http://spark.apache.org/docs/latest/building-spark.html) suggest that
>>>>
>>>>
>>>> ./make-distribution.sh --name custom-spark --tgz -Phadoop-2.4 -Pyarn
>>>>
>>>>
>>>>
>>>> should produce a distribution similar to the ones found on the
>>>> "Downloads" page.
>>>>
>>>> I noticed that the tgz I built using the above command does not produce
>>>> the datanucleus jars which are included in the "boxed" spark distributions.
>>>> What is the best-practice advice here?
>>>>
>>>> I would like my distribution to match the official one as closely as
>>>> possible.
>>>>
>>>> Thanks
>>>>
>>>
>>>
>>
>
Re: Problem with make-distribution.sh
Posted by Sean Owen <so...@cloudera.com>.
I don't think the page suggests that gives you any of the tarballs on the
downloads page, and -Phive does not by itself do so either.
On Mon, Oct 26, 2015 at 4:58 PM, Ted Yu <yu...@gmail.com> wrote:
> I logged SPARK-11318 with a PR.
>
> I verified that by adding -Phive the datanucleus jars are included:
>
> tar tzvf spark-1.6.0-SNAPSHOT-bin-custom-spark.tgz | grep datanucleus
> -rw-r--r-- hbase/hadoop 1890075 2015-10-26 09:52
> spark-1.6.0-SNAPSHOT-bin-custom-spark/lib/datanucleus-core-3.2.10.jar
> -rw-r--r-- hbase/hadoop 339666 2015-10-26 09:52
> spark-1.6.0-SNAPSHOT-bin-custom-spark/lib/datanucleus-api-jdo-3.2.6.jar
> -rw-r--r-- hbase/hadoop 1809447 2015-10-26 09:52
> spark-1.6.0-SNAPSHOT-bin-custom-spark/lib/datanucleus-rdbms-3.2.9.jar
>
> Cheers
>
> On Mon, Oct 26, 2015 at 8:52 AM, Yana Kadiyska <ya...@gmail.com>
> wrote:
>
>> thank you so much! You are correct. This is the second time I've made
>> this mistake :(
>>
>> On Mon, Oct 26, 2015 at 11:36 AM, java8964 <ja...@hotmail.com> wrote:
>>
>>> Maybe you need the Hive part?
>>>
>>> Yong
>>>
>>> ------------------------------
>>> Date: Mon, 26 Oct 2015 11:34:30 -0400
>>> Subject: Problem with make-distribution.sh
>>> From: yana.kadiyska@gmail.com
>>> To: user@spark.apache.org
>>>
>>>
>>> Hi folks,
>>>
>>> building spark instructions (
>>> http://spark.apache.org/docs/latest/building-spark.html) suggest that
>>>
>>>
>>> ./make-distribution.sh --name custom-spark --tgz -Phadoop-2.4 -Pyarn
>>>
>>>
>>>
>>> should produce a distribution similar to the ones found on the
>>> "Downloads" page.
>>>
>>> I noticed that the tgz I built using the above command does not produce
>>> the datanucleus jars which are included in the "boxed" spark distributions.
>>> What is the best-practice advice here?
>>>
>>> I would like my distribution to match the official one as closely as
>>> possible.
>>>
>>> Thanks
>>>
>>
>>
>
Re: Problem with make-distribution.sh
Posted by Ted Yu <yu...@gmail.com>.
I logged SPARK-11318 with a PR.
I verified that by adding -Phive the datanucleus jars are included:
tar tzvf spark-1.6.0-SNAPSHOT-bin-custom-spark.tgz | grep datanucleus
-rw-r--r-- hbase/hadoop 1890075 2015-10-26 09:52
spark-1.6.0-SNAPSHOT-bin-custom-spark/lib/datanucleus-core-3.2.10.jar
-rw-r--r-- hbase/hadoop 339666 2015-10-26 09:52
spark-1.6.0-SNAPSHOT-bin-custom-spark/lib/datanucleus-api-jdo-3.2.6.jar
-rw-r--r-- hbase/hadoop 1809447 2015-10-26 09:52
spark-1.6.0-SNAPSHOT-bin-custom-spark/lib/datanucleus-rdbms-3.2.9.jar
Cheers
On Mon, Oct 26, 2015 at 8:52 AM, Yana Kadiyska <ya...@gmail.com>
wrote:
> thank you so much! You are correct. This is the second time I've made this
> mistake :(
>
> On Mon, Oct 26, 2015 at 11:36 AM, java8964 <ja...@hotmail.com> wrote:
>
>> Maybe you need the Hive part?
>>
>> Yong
>>
>> ------------------------------
>> Date: Mon, 26 Oct 2015 11:34:30 -0400
>> Subject: Problem with make-distribution.sh
>> From: yana.kadiyska@gmail.com
>> To: user@spark.apache.org
>>
>>
>> Hi folks,
>>
>> building spark instructions (
>> http://spark.apache.org/docs/latest/building-spark.html) suggest that
>>
>>
>> ./make-distribution.sh --name custom-spark --tgz -Phadoop-2.4 -Pyarn
>>
>>
>>
>> should produce a distribution similar to the ones found on the
>> "Downloads" page.
>>
>> I noticed that the tgz I built using the above command does not produce
>> the datanucleus jars which are included in the "boxed" spark distributions.
>> What is the best-practice advice here?
>>
>> I would like my distribution to match the official one as closely as
>> possible.
>>
>> Thanks
>>
>
>
Re: Problem with make-distribution.sh
Posted by Yana Kadiyska <ya...@gmail.com>.
thank you so much! You are correct. This is the second time I've made this
mistake :(
On Mon, Oct 26, 2015 at 11:36 AM, java8964 <ja...@hotmail.com> wrote:
> Maybe you need the Hive part?
>
> Yong
>
> ------------------------------
> Date: Mon, 26 Oct 2015 11:34:30 -0400
> Subject: Problem with make-distribution.sh
> From: yana.kadiyska@gmail.com
> To: user@spark.apache.org
>
>
> Hi folks,
>
> building spark instructions (
> http://spark.apache.org/docs/latest/building-spark.html) suggest that
>
>
> ./make-distribution.sh --name custom-spark --tgz -Phadoop-2.4 -Pyarn
>
>
>
> should produce a distribution similar to the ones found on the "Downloads"
> page.
>
> I noticed that the tgz I built using the above command does not produce
> the datanucleus jars which are included in the "boxed" spark distributions.
> What is the best-practice advice here?
>
> I would like my distribution to match the official one as closely as
> possible.
>
> Thanks
>
RE: Problem with make-distribution.sh
Posted by java8964 <ja...@hotmail.com>.
Maybe you need the Hive part?
Yong
Date: Mon, 26 Oct 2015 11:34:30 -0400
Subject: Problem with make-distribution.sh
From: yana.kadiyska@gmail.com
To: user@spark.apache.org
Hi folks,
building spark instructions (http://spark.apache.org/docs/latest/building-spark.html) suggest that
./make-distribution.sh --name custom-spark --tgz -Phadoop-2.4 -Pyarn
should produce a distribution similar to the ones found on the "Downloads" page.
I noticed that the tgz I built using the above command does not produce the datanucleus jars which are included in the "boxed" spark distributions. What is the best-practice advice here?
I would like my distribution to match the official one as closely as possible.
Thanks