RPM spec file for 0.19.1

[Ian Soboroff <ia...@nist.gov>]

I created a JIRA (https://issues.apache.org/jira/browse/HADOOP-5615)
with a spec file for building a 0.19.1 RPM.

I like the idea of Cloudera's RPM file very much.  In particular, it has
nifty /etc/init.d scripts and RPM is nice for managing updates.
However, it's for an older, patched version of Hadoop.

This spec file is actually just Cloudera's, with suitable edits.  The
spec file does not contain an explicit license... if Cloudera have
strong feelings about it, let me know and I'll pull the JIRA attachment.

The JIRA includes instructions on how to roll the RPMs yourself.  I
would have attached the SRPM but they're too big for JIRA.  I can offer
noarch RPMs build with this spec file if someone wants to host them.

Ian

Re: RPM spec file for 0.19.1

[Ian Soboroff <ia...@nist.gov>]

If you guys want to spin RPMs for the community, that's great.  My main
motivation was that I wanted the current version rather than 0.18.3.

There is of course (as Steve points out) a larger discussion about if
you want RPMs, what should be in them.  In particular, some might want
to include the configuration in the RPMs.  That's a good reason to post
SRPMs, because then it's not so hard to re-roll the RPMs with different
configurations.

(Personally I wouldn't manage configs with RPM, it's just a pain to
propagate changes.  Instead, we are looking at using Puppet for general
cluster configuration needs, and RPMs for the basic "binaries".)

Ian

Christophe Bisciglia <ch...@cloudera.com> writes:

> Hey Ian, we are totally fine with this - the only reason we didn't
> contribute the SPEC file is that it is the output of our internal
> build system, and we don't have the bandwidth to properly maintain
> multiple RPMs.
>
> That said, we chatted about this a bit today, and were wondering if
> the community would like us to host RPMs for all releases in our
> "devel" repository. We can't stand behind these from a reliability
> angle the same way we can with our "blessed" RPMs, but it's a
> manageable amount of additional work to have our build system spit
> those out as well.
>
> If you'd like us to do this, please add a "me too" to this page:
> http://www.getsatisfaction.com/cloudera/topics/should_we_release_host_rpms_for_all_releases
>
> We could even skip the branding on the "devel" releases :-)
>
> Cheers,
> Christophe
>
> On Thu, Apr 2, 2009 at 12:46 PM, Ian Soboroff <ia...@nist.gov> wrote:
>>
>> I created a JIRA (https://issues.apache.org/jira/browse/HADOOP-5615)
>> with a spec file for building a 0.19.1 RPM.
>>
>> I like the idea of Cloudera's RPM file very much.  In particular, it has
>> nifty /etc/init.d scripts and RPM is nice for managing updates.
>> However, it's for an older, patched version of Hadoop.
>>
>> This spec file is actually just Cloudera's, with suitable edits.  The
>> spec file does not contain an explicit license... if Cloudera have
>> strong feelings about it, let me know and I'll pull the JIRA attachment.
>>
>> The JIRA includes instructions on how to roll the RPMs yourself.  I
>> would have attached the SRPM but they're too big for JIRA.  I can offer
>> noarch RPMs build with this spec file if someone wants to host them.
>>
>> Ian
>>
>>

Re: RPM spec file for 0.19.1

[Steve Loughran <st...@apache.org>]

Ian Soboroff wrote:
> Steve Loughran <st...@apache.org> writes:
> 
>> I think from your perpective it makes sense as it stops anyone getting
>> itchy fingers and doing their own RPMs. 
> 
> Um, what's wrong with that?
> 

It's reallly hard to do good RPM spec files. If cloudera are willing to 
pay Matt to do it, not only do I welcome it, but will see if I can help 
him with some of the automated test setup they'll need.

One thing which would be useful would be to package up all of the hadoop 
functional tests that need a live cluster up as its own JAR, so the test 
suite could be run against an RPM installation on different Virtual 
OS/JVM combos. I've just hit this problem with my own RPMs on Java6 
(java security related), so know that having the ability to use the 
entire existing test suite against an RPM installation would be be 
beneficial (both in my case and for hadoop RPMS)

Re: RPM spec file for 0.19.1

[Ian Soboroff <ia...@nist.gov>]

Simon Lewis <si...@lewis.li> writes:

> On 3 Apr 2009, at 15:11, Ian Soboroff wrote:
>> Steve Loughran <st...@apache.org> writes:
>>
>>> I think from your perpective it makes sense as it stops anyone
>>> getting
>>> itchy fingers and doing their own RPMs.
>>
>> Um, what's wrong with that?
>
> I would certainly like the ability to build RPMs from a source
> checkout, anyone thought of putting a standard spec file in with the
> source somewhere?

Another vote for a .spec file to be included in the standard
distribution as a contrib.

If it's ok with Cloudera (since my spec file just came from them), I
will edit my JIRA to offer that proposal.  If it's Cloudera's spec
that's included, we should also include the init.d script templates
(which are already Apache licensed).

Ian

Re: RPM spec file for 0.19.1

[Simon Lewis <si...@lewis.li>]

On 3 Apr 2009, at 15:11, Ian Soboroff wrote:
> Steve Loughran <st...@apache.org> writes:
>
>> I think from your perpective it makes sense as it stops anyone  
>> getting
>> itchy fingers and doing their own RPMs.
>
> Um, what's wrong with that?

I would certainly like the ability to build RPMs from a source  
checkout, anyone thought of putting a standard spec file in with the  
source somewhere?

I said yes to this: http://www.getsatisfaction.com/cloudera/topics/should_we_release_host_rpms_for_all_releases

Thanks,
Simon

Re: RPM spec file for 0.19.1

[Ian Soboroff <ia...@nist.gov>]

Steve Loughran <st...@apache.org> writes:

> I think from your perpective it makes sense as it stops anyone getting
> itchy fingers and doing their own RPMs. 

Um, what's wrong with that?

Ian

Re: RPM spec file for 0.19.1

[Steve Loughran <st...@apache.org>]

Christophe Bisciglia wrote:
> Hey Ian, we are totally fine with this - the only reason we didn't
> contribute the SPEC file is that it is the output of our internal
> build system, and we don't have the bandwidth to properly maintain
> multiple RPMs.
> 
> That said, we chatted about this a bit today, and were wondering if
> the community would like us to host RPMs for all releases in our
> "devel" repository. We can't stand behind these from a reliability
> angle the same way we can with our "blessed" RPMs, but it's a
> manageable amount of additional work to have our build system spit
> those out as well.


I think from your perpective it makes sense as it stops anyone getting 
itchy fingers and doing their own RPMs. At the same time, I think we do 
need to make it possible/easy to do RPMs *and have them consistent*. If 
hadoop-core makes RPMs that don't work with your settings rpms, you get 
to field to the support calls.

-steve

Re: RPM spec file for 0.19.1

[Christophe Bisciglia <ch...@cloudera.com>]

Hey Ian, we are totally fine with this - the only reason we didn't
contribute the SPEC file is that it is the output of our internal
build system, and we don't have the bandwidth to properly maintain
multiple RPMs.

That said, we chatted about this a bit today, and were wondering if
the community would like us to host RPMs for all releases in our
"devel" repository. We can't stand behind these from a reliability
angle the same way we can with our "blessed" RPMs, but it's a
manageable amount of additional work to have our build system spit
those out as well.

If you'd like us to do this, please add a "me too" to this page:
http://www.getsatisfaction.com/cloudera/topics/should_we_release_host_rpms_for_all_releases

We could even skip the branding on the "devel" releases :-)

Cheers,
Christophe

On Thu, Apr 2, 2009 at 12:46 PM, Ian Soboroff <ia...@nist.gov> wrote:
>
> I created a JIRA (https://issues.apache.org/jira/browse/HADOOP-5615)
> with a spec file for building a 0.19.1 RPM.
>
> I like the idea of Cloudera's RPM file very much.  In particular, it has
> nifty /etc/init.d scripts and RPM is nice for managing updates.
> However, it's for an older, patched version of Hadoop.
>
> This spec file is actually just Cloudera's, with suitable edits.  The
> spec file does not contain an explicit license... if Cloudera have
> strong feelings about it, let me know and I'll pull the JIRA attachment.
>
> The JIRA includes instructions on how to roll the RPMs yourself.  I
> would have attached the SRPM but they're too big for JIRA.  I can offer
> noarch RPMs build with this spec file if someone wants to host them.
>
> Ian
>
>

Re: RPM spec file for 0.19.1

[Ian Soboroff <ia...@nist.gov>]

Steve Loughran <st...@apache.org> writes:

> -RPM and deb packaging would be nice

Indeed.  The best thing would be to have the hadoop build system output
them, for some sensible subset of systems.

> -the jdk requirements are too harsh as it should run on openjdk's JRE
> or jrockit; no need for sun only. Too bad the only way to say that is
> leave off all jdk dependencies.

I haven't tried running Hadoop on anything but the Sun JDK, much less
built it from source (well, the rpmbuild did that so I guess I have).

> -I worry about how they patch the rc.d files. I can see why, but
> wonder what that does with the RPM ownership

Those are just fine: (from hadoop-init.tmpl)

#!/bin/bash
# 
# (c) Copyright 2009 Cloudera, Inc.
# 
#   Licensed under the Apache License, Version 2.0 (the "License");
#   you may not use this file except in compliance with the License.
...

Ian

Re: RPM spec file for 0.19.1

[Steve Loughran <st...@apache.org>]

Ian Soboroff wrote:
> I created a JIRA (https://issues.apache.org/jira/browse/HADOOP-5615)
> with a spec file for building a 0.19.1 RPM.
> 
> I like the idea of Cloudera's RPM file very much.  In particular, it has
> nifty /etc/init.d scripts and RPM is nice for managing updates.
> However, it's for an older, patched version of Hadoop.
> 
> This spec file is actually just Cloudera's, with suitable edits.  The
> spec file does not contain an explicit license... if Cloudera have
> strong feelings about it, let me know and I'll pull the JIRA attachment.
> 
> The JIRA includes instructions on how to roll the RPMs yourself.  I
> would have attached the SRPM but they're too big for JIRA.  I can offer
> noarch RPMs build with this spec file if someone wants to host them.
> 
> Ian
> 

-RPM and deb packaging would be nice

-the .spec file should be driven by ant properties to get dependencies 
from the ivy files
-the jdk requirements are too harsh as it should run on openjdk's JRE or 
jrockit; no need for sun only. Too bad the only way to say that is leave 
off all jdk dependencies.
-I worry about how they patch the rc.d files. I can see why, but wonder 
what that does with the RPM ownership

As someone whose software does get released as RPMs (and tar files 
containing everything needed to create your own), I can state with 
experience that RPMs are very hard to get right, and very hard to test. 
The hardest thing to get right (and to test) is live update of the RPMs 
while the app is running. I am happy for the cloudera team to have taken 
on this problem.