You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by Christian Tzolov <ch...@gmail.com> on 2015/07/12 20:20:43 UTC
How generate spark-rpm from a custom build Spark tarball?
OOB spark-rpm downloads the bigtop.mk#SPARK_BASE_VERSION Spark tarball form
the apache archive: http://archive.apache.org/dist/spark
So I can change the bigtop.mk#SPARK_BASE_VERSION to the latest 1.4.0 and
build proper RPMs for this version.
But if i need to create my own Spark tarball assembly that enables flags
like -Phive and -Phive-thriftserver ... then what is the best way to point
Bigtop to my custom assembly and generate rpms for it?
Would appreciate any suggestions!
Cheers,
Christian
Re: How generate spark-rpm from a custom build Spark tarball?
Posted by Jay Vyas <ja...@gmail.com>.
Thanks for bringing this up Christian.
We encourage people to open up bigtop and build their own distributions, so it's great to see you doing this. Let us know how it works for you and we'll be here to answer questions.
In general, For build options you often want to look at the do-component-build file, which is the part that compiles the maven artifacts.
For example https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/mahout/do-component-build for mahout and so on.
> On Jul 12, 2015, at 5:05 PM, Christian Tzolov <ch...@gmail.com> wrote:
>
> Thank Olaf, Looks like the bigtop-packages/src/common/spark/do-component-build would help
>
>> On Sun, Jul 12, 2015 at 10:26 PM, Olaf Flebbe <of...@oflebbe.de> wrote:
>> Hi.
>>
>> Yes you can edit the bigtop.mk file for changing versions. For enabling additional build options edit
>> bigtop-packages/src/common/spark/do-component-build
>>
>> If you need to patch the sources, please follow the instructions at
>> https://cwiki.apache.org/confluence/display/BIGTOP/Bigtop+Packaging
>> (see zookeeper for a demo if this functionality)
>>
>> Olaf
>>
>> > Am 12.07.2015 um 20:20 schrieb Christian Tzolov <ch...@gmail.com>:
>> >
>> > OOB spark-rpm downloads the bigtop.mk#SPARK_BASE_VERSION Spark tarball form the apache archive: http://archive.apache.org/dist/spark
>> >
>> > So I can change the bigtop.mk#SPARK_BASE_VERSION to the latest 1.4.0 and build proper RPMs for this version.
>> >
>> > But if i need to create my own Spark tarball assembly that enables flags like -Phive and -Phive-thriftserver ... then what is the best way to point Bigtop to my custom assembly and generate rpms for it?
>> >
>> > Would appreciate any suggestions!
>> >
>> > Cheers,
>> > Christian
>
Re: How generate spark-rpm from a custom build Spark tarball?
Posted by Christian Tzolov <ch...@gmail.com>.
Thank Olaf, Looks like the bigtop-packages/src/common/spark/do-component-build
would help
On Sun, Jul 12, 2015 at 10:26 PM, Olaf Flebbe <of...@oflebbe.de> wrote:
> Hi.
>
> Yes you can edit the bigtop.mk file for changing versions. For enabling
> additional build options edit
> bigtop-packages/src/common/spark/do-component-build
>
> If you need to patch the sources, please follow the instructions at
> https://cwiki.apache.org/confluence/display/BIGTOP/Bigtop+Packaging
> (see zookeeper for a demo if this functionality)
>
> Olaf
>
> > Am 12.07.2015 um 20:20 schrieb Christian Tzolov <
> christian.tzolov@gmail.com>:
> >
> > OOB spark-rpm downloads the bigtop.mk#SPARK_BASE_VERSION Spark tarball
> form the apache archive: http://archive.apache.org/dist/spark
> >
> > So I can change the bigtop.mk#SPARK_BASE_VERSION to the latest 1.4.0
> and build proper RPMs for this version.
> >
> > But if i need to create my own Spark tarball assembly that enables flags
> like -Phive and -Phive-thriftserver ... then what is the best way to point
> Bigtop to my custom assembly and generate rpms for it?
> >
> > Would appreciate any suggestions!
> >
> > Cheers,
> > Christian
>
>
Re: How generate spark-rpm from a custom build Spark tarball?
Posted by Olaf Flebbe <of...@oflebbe.de>.
Hi.
Yes you can edit the bigtop.mk file for changing versions. For enabling additional build options edit
bigtop-packages/src/common/spark/do-component-build
If you need to patch the sources, please follow the instructions at
https://cwiki.apache.org/confluence/display/BIGTOP/Bigtop+Packaging
(see zookeeper for a demo if this functionality)
Olaf
> Am 12.07.2015 um 20:20 schrieb Christian Tzolov <ch...@gmail.com>:
>
> OOB spark-rpm downloads the bigtop.mk#SPARK_BASE_VERSION Spark tarball form the apache archive: http://archive.apache.org/dist/spark
>
> So I can change the bigtop.mk#SPARK_BASE_VERSION to the latest 1.4.0 and build proper RPMs for this version.
>
> But if i need to create my own Spark tarball assembly that enables flags like -Phive and -Phive-thriftserver ... then what is the best way to point Bigtop to my custom assembly and generate rpms for it?
>
> Would appreciate any suggestions!
>
> Cheers,
> Christian