You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Rahul Singhal <Ra...@guavus.com> on 2014/04/04 09:10:11 UTC

How to create a RPM package

Hello Community,

This is my first mail to the list and I have a small question. The maven build page<http://spark.apache.org/docs/latest/building-with-maven.html#building-spark-debian-packages> mentions a way to create a debian package but I was wondering if there is a simple way (preferably through maven) to create a RPM package. Is there a script (which is probably used for spark releases) that I can get my hands on? Or should I write one on my own?

P.S. I don't want to use the "alien" software to convert a debian package to a RPM.

Thanks,
Rahul Singhal

Re: How to create a RPM package

Posted by Patrick Wendell <pw...@gmail.com>.

We might be able to incorporate the maven rpm plugin into our build. If
that can be done in an elegant way it would be nice to have that
distribution target for people who wanted to try this with arbitrary Spark
versions...

Personally I have no familiarity with that plug-in, so curious if anyone in
the community has feedback from trying this.

- Patrick


On Fri, Apr 4, 2014 at 12:43 PM, Rahul Singhal <Ra...@guavus.com>wrote:

>   Hi Christophe,
>
>  Thanks for your reply and the spec file. I have solved my issue for now.
> I didn't want to rely building spark using the spec file (%build section)
> as I don't want to be maintaining the list of files that need to be
> packaged. I ended up adding maven build support to make-distribution.sh.
> This script produces a tar ball which I can then use to create a RPM
> package.
>
>   Thanks,
> Rahul Singhal
>
>   From: Christophe Préaud <ch...@kelkoo.com>
> Reply-To: "user@spark.apache.org" <us...@spark.apache.org>
> Date: Friday 4 April 2014 7:55 PM
> To: "user@spark.apache.org" <us...@spark.apache.org>
> Subject: Re: How to create a RPM package
>
>   Hi Rahul,
>
> Spark will be available in Fedora 21 (see:
> https://fedoraproject.org/wiki/SIGs/bigdata/packaging/Spark), currently
> scheduled on 2014-10-14 but they already have produced spec files and
> source RPMs.
> If you are stuck with EL6 like me, you can have a look at the attached
> spec file, which you can probably adapt to your need.
>
> Christophe.
>
> On 04/04/2014 09:10, Rahul Singhal wrote:
>
>  Hello Community,
>
>  This is my first mail to the list and I have a small question. The maven
> build page<http://spark.apache.org/docs/latest/building-with-maven.html#building-spark-debian-packages> mentions
> a way to create a debian package but I was wondering if there is a simple
> way (preferably through maven) to create a RPM package. Is there a script
> (which is probably used for spark releases) that I can get my hands on? Or
> should I write one on my own?
>
>  P.S. I don't want to use the "alien" software to convert a debian
> package to a RPM.
>
>   Thanks,
>  Rahul Singhal
>
>
>
> ------------------------------
> Kelkoo SAS
> Société par Actions Simplifiée
> Au capital de EURO 4.168.964,30
> Siège social : 8, rue du Sentier 75002 Paris
> 425 093 069 RCS Paris
>
> Ce message et les pièces jointes sont confidentiels et établis à
> l'attention exclusive de leurs destinataires. Si vous n'êtes pas le
> destinataire de ce message, merci de le détruire et d'en avertir
> l'expéditeur.
>

Re: How to create a RPM package

Posted by Will Benton <wi...@redhat.com>.

> For issue #2 I was concerned that the build & packaging had to be
> internal. So I am using the already packaged make-distribution.sh
> (modified to use a maven build) to create a tar ball which I then package
> it using a RPM spec file.

Hi Rahul, so the issue for downstream operating system distributions is that they basically need to be able to build everything from source.  So while you're building Spark itself locally, a package that appears in Fedora or Debian will go a bit further and also build all of the library dependencies, compilers, etc., from sources (possibly with some exceptions for bootstrapping compilers that can't be built without themselves) rather than pulling down binaries from a public Maven or Ivy repository.

> Although on a side note, it would interesting to learn how the list of
> files does not need to be maintained in the spec file (the spec file that
> Christophe attached was using a explicit list).

If you take a look at the spec that is in Fedora (http://pkgs.fedoraproject.org/cgit/spark.git/tree/spark.spec, starting at line 257 in the current revision), you'll see the following %files section:

    %files -f .mfiles
    %dir %{_javadir}/%{name}

    %doc LICENSE README.md

    %files javadoc
    %{_javadocdir}/%{name}
    %doc LICENSE

As you can see, this list is pretty generic (and obviously doesn't include all the files in the package).  The list of JAR and POM files is provided in a file called .mfiles (which is automatically generated by macros) and we just have to specify the directories that the package owns (which aren't picked up by the macros at this time), the license, and the README.



best,
wb

Re: How to create a RPM package

Posted by Rahul Singhal <Ra...@guavus.com>.

Hi Will,

For issue #2 I was concerned that the build & packaging had to be
internal. So I am using the already packaged make-distribution.sh
(modified to use a maven build) to create a tar ball which I then package
it using a RPM spec file.

Although on a side note, it would interesting to learn how the list of
files does not need to be maintained in the spec file (the spec file that
Christophe attached was using a explicit list).

Thanks for the explanation and your help.


Thanks,
Rahul Singhal





On 05/04/14 9:48 PM, "Will Benton" <wi...@redhat.com> wrote:

>Hi Rahul,
>
>As Christophe pointed out, Spark has been in Fedora Rawhide (which will
>become Fedora 21) for a little while now.  (I haven't announced it here
>because Rawhide is a little too bleeding-edge for most end-users.)  With
>native packages of any kind, there are a couple of considerations:
>
>1.  do we have a particular sort of container for executable artifacts
>and dependency metadata?
>2.  do we have a package that meets certain criteria for system
>integration and repeatable builds?
>
>Many users (and installations) are only concerned about issue #1:  they
>would like a package that they can install and remove easily (perhaps
>automatically).  Most downstream operating system distributions (like
>Fedora and Debian) are also concerned with issue #2, which is really the
>hard part in a lot of cases.  (There are good reasons to care about both,
>of course!)
>
>If *you're* only worried about issue #1, then you have a couple of
>options:
>
>1.  The Apache Bigtop project does a great job of producing produces
>Spark binaries (as well as many Hadoop ecosystem projects) in RPM
>containers, or
>2.  I'm happy to give you some guidance for how to modify the Fedora
>Spark spec file to produce Spark binaries in an RPM container that
>doesn't meet Fedora packaging guidelines.  (Note that you don't have to
>maintain explicit file lists in a spec file in general, and certainly not
>in this case.)
>
>
>
>best,
>wb
>
>
>----- Original Message -----
>> From: "Rahul Singhal" <Ra...@guavus.com>
>> To: user@spark.apache.org
>> Sent: Friday, April 4, 2014 2:43:48 PM
>> Subject: Re: How to create a RPM package
>> 
>> Hi Christophe,
>> 
>> Thanks for your reply and the spec file. I have solved my issue for
>>now. I
>> didn't want to rely building spark using the spec file (%build section)
>>as I
>> don't want to be maintaining the list of files that need to be
>>packaged. I
>> ended up adding maven build support to make-distribution.sh. This script
>> produces a tar ball which I can then use to create a RPM package.
>> 
>> Thanks,
>> Rahul Singhal
>> 
>> From: Christophe Préaud
>> <ch...@kelkoo.com>>
>> Reply-To: "user@spark.apache.org<ma...@spark.apache.org>"
>> <us...@spark.apache.org>>
>> Date: Friday 4 April 2014 7:55 PM
>> To: "user@spark.apache.org<ma...@spark.apache.org>"
>> <us...@spark.apache.org>>
>> Subject: Re: How to create a RPM package
>> 
>> Hi Rahul,
>> 
>> Spark will be available in Fedora 21 (see:
>> https://fedoraproject.org/wiki/SIGs/bigdata/packaging/Spark), currently
>> scheduled on 2014-10-14 but they already have produced spec files and
>>source
>> RPMs.
>> If you are stuck with EL6 like me, you can have a look at the attached
>>spec
>> file, which you can probably adapt to your need.
>> 
>> Christophe.
>> 
>> On 04/04/2014 09:10, Rahul Singhal wrote:
>> Hello Community,
>> 
>> This is my first mail to the list and I have a small question. The maven
>> build
>> 
>>page<http://spark.apache.org/docs/latest/building-with-maven.html#buildin
>>g-spark-debian-packages>
>> mentions a way to create a debian package but I was wondering if there
>>is a
>> simple way (preferably through maven) to create a RPM package. Is there
>>a
>> script (which is probably used for spark releases) that I can get my
>>hands
>> on? Or should I write one on my own?
>> 
>> P.S. I don't want to use the "alien" software to convert a debian
>>package to
>> a RPM.
>> 
>> Thanks,
>> Rahul Singhal
>> 
>> 
>> ________________________________
>> Kelkoo SAS
>> Société par Actions Simplifiée
>> Au capital de € 4.168.964,30
>> Siège social : 8, rue du Sentier 75002 Paris
>> 425 093 069 RCS Paris
>> 
>> Ce message et les pièces jointes sont confidentiels et établis à
>>l'attention
>> exclusive de leurs destinataires. Si vous n'êtes pas le destinataire de
>>ce
>> message, merci de le détruire et d'en avertir l'expéditeur.
>>

Re: How to create a RPM package

Posted by Will Benton <wi...@redhat.com>.

Hi Rahul,

As Christophe pointed out, Spark has been in Fedora Rawhide (which will become Fedora 21) for a little while now.  (I haven't announced it here because Rawhide is a little too bleeding-edge for most end-users.)  With native packages of any kind, there are a couple of considerations:

1.  do we have a particular sort of container for executable artifacts and dependency metadata?
2.  do we have a package that meets certain criteria for system integration and repeatable builds?

Many users (and installations) are only concerned about issue #1:  they would like a package that they can install and remove easily (perhaps automatically).  Most downstream operating system distributions (like Fedora and Debian) are also concerned with issue #2, which is really the hard part in a lot of cases.  (There are good reasons to care about both, of course!)

If *you're* only worried about issue #1, then you have a couple of options:

1.  The Apache Bigtop project does a great job of producing produces Spark binaries (as well as many Hadoop ecosystem projects) in RPM containers, or
2.  I'm happy to give you some guidance for how to modify the Fedora Spark spec file to produce Spark binaries in an RPM container that doesn't meet Fedora packaging guidelines.  (Note that you don't have to maintain explicit file lists in a spec file in general, and certainly not in this case.)

best,
wb

----- Original Message -----
> From: "Rahul Singhal" <Ra...@guavus.com>
> To: user@spark.apache.org
> Sent: Friday, April 4, 2014 2:43:48 PM
> Subject: Re: How to create a RPM package
> 
> Hi Christophe,
> 
> Thanks for your reply and the spec file. I have solved my issue for now. I
> didn't want to rely building spark using the spec file (%build section) as I
> don't want to be maintaining the list of files that need to be packaged. I
> ended up adding maven build support to make-distribution.sh. This script
> produces a tar ball which I can then use to create a RPM package.
> 
> Thanks,
> Rahul Singhal
> 
> From: Christophe Préaud
> <ch...@kelkoo.com>>
> Reply-To: "user@spark.apache.org<ma...@spark.apache.org>"
> <us...@spark.apache.org>>
> Date: Friday 4 April 2014 7:55 PM
> To: "user@spark.apache.org<ma...@spark.apache.org>"
> <us...@spark.apache.org>>
> Subject: Re: How to create a RPM package
> 
> Hi Rahul,
> 
> Spark will be available in Fedora 21 (see:
> https://fedoraproject.org/wiki/SIGs/bigdata/packaging/Spark), currently
> scheduled on 2014-10-14 but they already have produced spec files and source
> RPMs.
> If you are stuck with EL6 like me, you can have a look at the attached spec
> file, which you can probably adapt to your need.
> 
> Christophe.
> 
> On 04/04/2014 09:10, Rahul Singhal wrote:
> Hello Community,
> 
> This is my first mail to the list and I have a small question. The maven
> build
> page<http://spark.apache.org/docs/latest/building-with-maven.html#building-spark-debian-packages>
> mentions a way to create a debian package but I was wondering if there is a
> simple way (preferably through maven) to create a RPM package. Is there a
> script (which is probably used for spark releases) that I can get my hands
> on? Or should I write one on my own?
> 
> P.S. I don't want to use the "alien" software to convert a debian package to
> a RPM.
> 
> Thanks,
> Rahul Singhal
> 
> 
> ________________________________
> Kelkoo SAS
> Société par Actions Simplifiée
> Au capital de € 4.168.964,30
> Siège social : 8, rue du Sentier 75002 Paris
> 425 093 069 RCS Paris
> 
> Ce message et les pièces jointes sont confidentiels et établis à l'attention
> exclusive de leurs destinataires. Si vous n'êtes pas le destinataire de ce
> message, merci de le détruire et d'en avertir l'expéditeur.
>

Re: How to create a RPM package

Posted by Rahul Singhal <Ra...@guavus.com>.

Hi Christophe,

Thanks for your reply and the spec file. I have solved my issue for now. I didn't want to rely building spark using the spec file (%build section) as I don't want to be maintaining the list of files that need to be packaged. I ended up adding maven build support to make-distribution.sh. This script produces a tar ball which I can then use to create a RPM package.

Thanks,
Rahul Singhal

From: Christophe Préaud <ch...@kelkoo.com>>
Reply-To: "user@spark.apache.org<ma...@spark.apache.org>" <us...@spark.apache.org>>
Date: Friday 4 April 2014 7:55 PM
To: "user@spark.apache.org<ma...@spark.apache.org>" <us...@spark.apache.org>>
Subject: Re: How to create a RPM package

Hi Rahul,

Spark will be available in Fedora 21 (see: https://fedoraproject.org/wiki/SIGs/bigdata/packaging/Spark), currently scheduled on 2014-10-14 but they already have produced spec files and source RPMs.
If you are stuck with EL6 like me, you can have a look at the attached spec file, which you can probably adapt to your need.

Christophe.

On 04/04/2014 09:10, Rahul Singhal wrote:
Hello Community,

This is my first mail to the list and I have a small question. The maven build page<http://spark.apache.org/docs/latest/building-with-maven.html#building-spark-debian-packages> mentions a way to create a debian package but I was wondering if there is a simple way (preferably through maven) to create a RPM package. Is there a script (which is probably used for spark releases) that I can get my hands on? Or should I write one on my own?

P.S. I don't want to use the "alien" software to convert a debian package to a RPM.

Thanks,
Rahul Singhal

________________________________
Kelkoo SAS
Société par Actions Simplifiée
Au capital de € 4.168.964,30
Siège social : 8, rue du Sentier 75002 Paris
425 093 069 RCS Paris

Ce message et les pièces jointes sont confidentiels et établis à l'attention exclusive de leurs destinataires. Si vous n'êtes pas le destinataire de ce message, merci de le détruire et d'en avertir l'expéditeur.

Re: How to create a RPM package

Posted by Christophe Préaud <ch...@kelkoo.com>.

Hi Rahul,

Spark will be available in Fedora 21 (see: https://fedoraproject.org/wiki/SIGs/bigdata/packaging/Spark), currently scheduled on 2014-10-14 but they already have produced spec files and source RPMs.
If you are stuck with EL6 like me, you can have a look at the attached spec file, which you can probably adapt to your need.

Christophe.

On 04/04/2014 09:10, Rahul Singhal wrote:
Hello Community,

This is my first mail to the list and I have a small question. The maven build page<http://spark.apache.org/docs/latest/building-with-maven.html#building-spark-debian-packages> mentions a way to create a debian package but I was wondering if there is a simple way (preferably through maven) to create a RPM package. Is there a script (which is probably used for spark releases) that I can get my hands on? Or should I write one on my own?

P.S. I don't want to use the "alien" software to convert a debian package to a RPM.

Thanks,
Rahul Singhal


________________________________
Kelkoo SAS
Société par Actions Simplifiée
Au capital de € 4.168.964,30
Siège social : 8, rue du Sentier 75002 Paris
425 093 069 RCS Paris

Ce message et les pièces jointes sont confidentiels et établis à l'attention exclusive de leurs destinataires. Si vous n'êtes pas le destinataire de ce message, merci de le détruire et d'en avertir l'expéditeur.