You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mesos.apache.org by Jason Dusek <ja...@gmail.com> on 2013/07/10 19:27:01 UTC
Merging in packaging tools for Mesos
Dear List,
I have been working on automated packaging for Mesos on Debian
and Ubuntu.
https://github.com/solidsnack/mesos-deb-packaging/
Much thanks is due to Tomas Barton, who got the ball rolling on
this and put together much of the code in the repo above.
The present collection of init scripts and build options is used
regularly for automated rollouts of Mesos at my firm, on Debian
and Ubuntu, in conjunction with tools using the Java API. We are
looking in to proper installation and setup of the Python
libraries.
The `build_mesos' script is useful for those wishing to build
and release internal forks of Mesos, too.
Support for RedHat/CentOS is important to us and is planned for
the near future. We'd like to see Mesos make its way in to the
repositories someday. What we have is a solid foundation and it
seems like integrating with mainline Mesos early is the best way
to make for a smooth transition in the future.
Would it be alright to add this material to
http://git.apache.org/incubator-mesos.git
under a new directory, "packaging"? This tooling will eventually
grow to encompass Homebrew as well as RPMs so its probably just
as well to keep the name distro agnostic.
--
Jason Dusek
pgp // solidsnack // C1EBC57DC55144F35460C8DF1FD4C6C1FED18A2B
Re: Merging in packaging tools for Mesos
Posted by Mathias Söderberg <ma...@burtcorp.com>.
Just ignore that last message from me. Fuck mailbox for not supporting the undo option for gmail.
On Fri, Jul 12, 2013 at 10:39 PM, Dave Lester <da...@ischool.berkeley.edu>
wrote:
> On Fri, Jul 12, 2013 at 12:13 PM, Jason Dusek <ja...@gmail.com> wrote:
>> This really seems right to me. The meta package manager might
>> get you something the installer will accept; but not something
>> the distro will accept.
>>
> Agreed, pursuing separate maintainers for different Mesos packages sounds
> like the easiest solution at this point. Also a great way to involve more
> members from the community. It would be ideal to have a point person for
> each package, so we know who to bug when a new release goes out.
> What are we missing? I'd love to see a Homebrew script for all the OSX
> users out there..
> Dave
Re: Merging in packaging tools for Mesos
Posted by Mathias Söderberg <ma...@burtcorp.com>.
On Fri, Jul 12, 2013 at 10:39 PM, Dave Lester <da...@ischool.berkeley.edu>
wrote:
> On Fri, Jul 12, 2013 at 12:13 PM, Jason Dusek <ja...@gmail.com> wrote:
>> This really seems right to me. The meta package manager might
>> get you something the installer will accept; but not something
>> the distro will accept.
>>
> Agreed, pursuing separate maintainers for different Mesos packages sounds
> like the easiest solution at this point. Also a great way to involve more
> members from the community. It would be ideal to have a point person for
> each package, so we know who to bug when a new release goes out.
> What are we missing? I'd love to see a Homebrew script for all the OSX
> users out there..
> Dave
Re: Merging in packaging tools for Mesos
Posted by Jason Dusek <ja...@gmail.com>.
2013/7/12 Dave Lester <da...@ischool.berkeley.edu>:
> On Fri, Jul 12, 2013 at 12:13 PM, Jason Dusek <ja...@gmail.com> wrote:
> Agreed, pursuing separate maintainers for different Mesos packages sounds
> like the easiest solution at this point. Also a great way to involve more
> members from the community. It would be ideal to have a point person for
> each package, so we know who to bug when a new release goes out.
>
> What are we missing? I'd love to see a Homebrew script for all the OSX
> users out there..
OP delivers:
https://gist.github.com/solidsnack/5962537/raw/0d203d94ab1a8ae950bc9203013199cfcf911867/mesos.rb
--
Jason Dusek
pgp // solidsnack // C1EBC57DC55144F35460C8DF1FD4C6C1FED18A2B
Re: Merging in packaging tools for Mesos
Posted by Dave Lester <da...@ischool.berkeley.edu>.
On Fri, Jul 12, 2013 at 12:13 PM, Jason Dusek <ja...@gmail.com> wrote:
> This really seems right to me. The meta package manager might
> get you something the installer will accept; but not something
> the distro will accept.
>
Agreed, pursuing separate maintainers for different Mesos packages sounds
like the easiest solution at this point. Also a great way to involve more
members from the community. It would be ideal to have a point person for
each package, so we know who to bug when a new release goes out.
What are we missing? I'd love to see a Homebrew script for all the OSX
users out there..
Dave
Re: Merging in packaging tools for Mesos
Posted by Tim St Clair <ts...@redhat.com>.
Feel free to ping fedora-java mailing list on *this.
I have my own thoughts, but they are the source on java policy thus would answer the question best.
Cheers,
Tim
----- Original Message -----
> From: "Jason Dusek" <ja...@gmail.com>
> To: dev@mesos.apache.org, "Bernardo Gomez Palacio" <be...@gmail.com>
> Sent: Tuesday, September 24, 2013 1:23:20 PM
> Subject: Re: Merging in packaging tools for Mesos
>
> Hi Everybody,
>
> I am reviving this old thread, to talk about builds specifically
> on RHEL and CentOS. I am not sure how to ensure reliably that
> Mesos finds libjvm.so while allowing flexibility in the choice
> of JDK.
>
> On Debian and Ubuntu, one can depend on an abstract package for
> Java and even depend on one or another package. So for the
> builds I do at present, the Deb package depends on:
>
> java7-runtime-headless | java6-runtime-headless
>
> On Debian and Ubuntu, as long as the default-jre-headless is
> installed, then Mesos finds libjvm.so without any problems; and
> it's indifferent to whether it's a Java 7 JRE or Java 6 JRE.
>
> But on RedHat and CentOS, there does not seem to be any way to
> specify an "abstract" Java dependency and alternatives are not
> part of the RedHat dependency specification language. Two
> suggestions I've seen online are:
>
> * Create ones own abstract Java package (not sure how this
> would work) and depend on that.
>
> * Just don't depend on Java and allow the user to install it
> (seems to be what CDH does).
>
> I've taken the latter approach.
>
> So now we come to the problem of libjvm.so -- Mesos doesn't find
> it. The only reliable thing I've got going is using `locate' and
> symlinking the version from the server JRE in to /usr/lib. But
> one hesitates to do this from a package post-install script.
>
> It is also notable that the Java library paths aren't set up
> right for many of the packages, so one can't simply query Java
> for the location of libjvm.so:
>
> https://bugzilla.redhat.com/show_bug.cgi?id=740762
>
> Has anyone else encountered difficulty with discovery of
> libjvm.so on RedHat and CentOS when running internal RPMs? I'd
> like to find a way to ensure the packages work without any
> fiddling. It may be that the ultimate solution is to provide
> separate JDK 1.6 and JDK 1.7 versions of the Mesos package for
> these distros and modify LD_LIBRARY_PATH from the init scripts.
>
> --
> Jason Dusek
>
> --
> Jason Dusek
> pgp // solidsnack // C1EBC57DC55144F35460C8DF1FD4C6C1FED18A2B
>
>
> 2013/7/12 Bernardo Gomez Palacio <be...@gmail.com>:
> > At least in master, 0.14.0, file based configuration with the `--conf`
> > options is not supported. That said imho we could follow the approach
> > stated bellow.
> >
> > Within the execution of the init.d script source a default env shell. (
> > extracted from mesos-profile-env.sh )
> >
> > # Source the default environment variables.
> > DEFAULT_ENV="$PRGDIR/mesos-profile-env.sh"
> > if [ -f "$DEFAULT_ENV"]
> > source $DEFAULT_ENV
> > fi
> >
> > And also source a profile.d shell if available
> >
> > # Source the profile environment variables if available.
> > PROFILE_ENV="/etc/profile.d/mesos-profile-env.sh"
> > if [ -f "$PROFILE_ENV" ]; then
> > source "$PROFILE_ENV"
> > fi
> >
> > Then support a configuration file, which basically defines what can be set
> > as options, such that you can again have a default or override for a
> > specific package (local, master, slave)
> >
> > # Setup Configuration Directory
> > if [ -f "/etc/mesos/mesos-master.conf" ]; then
> > CONFIGFILE="/etc/mesos/mesos-master.conf"
> > else
> > CONFIGFILE="/etc/mesos/mesos.conf"
> > fi
> >
> > Apply the conf file elements as arguments/options to either the master,
> > slave, or local.
> >
> > # Initialize the _Command_ options.
> > OPTIONS="--log_dir=$LOG_DIR"
> > if [ -n "$CONFIGFILE" ]; then
> > OPTIONS=`paste -d " " <(cat /etc/mesos/master-master.conf | grep -e
> > "^[--]" )`
> > fi
> >
> >
> > Example references.
> > 1. "mesos-profile-env.sh"
> > https://github.com/Guavus/incubator-mesos/blob/support/guavus-master/src/deploy/mesos-profile-env.sh.in
> > 2. "mesos-master init.d script for rhl"
> > https://github.com/Guavus/incubator-mesos/blob/support/guavus-master/src/deploy/mesos-masterd.sh.in
> >
> > Best
> > Bernardo
> >
> >
> > On Jul 12, 2013, at 12:13 PM, Jason Dusek <ja...@gmail.com> wrote:
> >
> >> 2013/7/11 Daniel Lundin <dl...@eintr.org>:
> >>> As an option, I've attached the debian packaging patch I've
> >>> used for packaging Mesos for Ubuntu (Precise) to MESOS-74. It
> >>> does not use a third party / unified packager, rather just the
> >>> standard toolchain - A debian directory in the source tree
> >>> with all the trimmings. Just run "debuild ..." at the top and,
> >>> ideally, you're good to go.
> >>>
> >>> Overall, in my experience it's hard to maintain packages for
> >>> multiple platforms using a "meta-packager" scheme on a
> >>> sizeable/multi-language project. It's usually easier to just
> >>> go with the romans, whomever they may be at that particular
> >>> juncture. Different systems have different policies for
> >>> packages, how to split them up (libs, dev, language-specific
> >>> bits), dealing with multi-architecture, python versions, etc.
> >>> I've always ended up with *more* work trying to unify system
> >>> packaging than just doing it once per platform. :)
> >>
> >> This really seems right to me. The meta package manager might
> >> get you something the installer will accept; but not something
> >> the distro will accept.
> >>
> >>> It might be worth considering it can be easier to find people
> >>> experienced, interested and opinionated in platform-specific
> >>> packaging, and the work tends to be low-hanging fruit for
> >>> contributing to a project.
> >>>
> >>> In any case, hope the patch comes in handy.
> >>
> >> In its pretty form, this is pretty hard to work with. If there
> >> is anyway you could create branch on GitHub...
> >>
> >> Of course I will still try to read it over the weekend, anyways.
> >>
> >> An open question for the project is how we want to manage Mesos
> >> configuration. In the branch I'm working on, it's all
> >> environment variables; and /etc/mesos/zk is loaded into an
> >> environment variable and is the only "config file". We could
> >> have a `mesos.conf' but as far as I understand it, file-based
> >> config is slated for removal.
> >>
> >> --
> >> Jason Dusek
> >> pgp // solidsnack // C1EBC57DC55144F35460C8DF1FD4C6C1FED18A2B
> >
>
--
Cheers,
Tim
Re: Merging in packaging tools for Mesos
Posted by Jason Dusek <ja...@gmail.com>.
Hi Everybody,
I am reviving this old thread, to talk about builds specifically
on RHEL and CentOS. I am not sure how to ensure reliably that
Mesos finds libjvm.so while allowing flexibility in the choice
of JDK.
On Debian and Ubuntu, one can depend on an abstract package for
Java and even depend on one or another package. So for the
builds I do at present, the Deb package depends on:
java7-runtime-headless | java6-runtime-headless
On Debian and Ubuntu, as long as the default-jre-headless is
installed, then Mesos finds libjvm.so without any problems; and
it's indifferent to whether it's a Java 7 JRE or Java 6 JRE.
But on RedHat and CentOS, there does not seem to be any way to
specify an "abstract" Java dependency and alternatives are not
part of the RedHat dependency specification language. Two
suggestions I've seen online are:
* Create ones own abstract Java package (not sure how this
would work) and depend on that.
* Just don't depend on Java and allow the user to install it
(seems to be what CDH does).
I've taken the latter approach.
So now we come to the problem of libjvm.so -- Mesos doesn't find
it. The only reliable thing I've got going is using `locate' and
symlinking the version from the server JRE in to /usr/lib. But
one hesitates to do this from a package post-install script.
It is also notable that the Java library paths aren't set up
right for many of the packages, so one can't simply query Java
for the location of libjvm.so:
https://bugzilla.redhat.com/show_bug.cgi?id=740762
Has anyone else encountered difficulty with discovery of
libjvm.so on RedHat and CentOS when running internal RPMs? I'd
like to find a way to ensure the packages work without any
fiddling. It may be that the ultimate solution is to provide
separate JDK 1.6 and JDK 1.7 versions of the Mesos package for
these distros and modify LD_LIBRARY_PATH from the init scripts.
--
Jason Dusek
--
Jason Dusek
pgp // solidsnack // C1EBC57DC55144F35460C8DF1FD4C6C1FED18A2B
2013/7/12 Bernardo Gomez Palacio <be...@gmail.com>:
> At least in master, 0.14.0, file based configuration with the `--conf` options is not supported. That said imho we could follow the approach stated bellow.
>
> Within the execution of the init.d script source a default env shell. ( extracted from mesos-profile-env.sh )
>
> # Source the default environment variables.
> DEFAULT_ENV="$PRGDIR/mesos-profile-env.sh"
> if [ -f "$DEFAULT_ENV"]
> source $DEFAULT_ENV
> fi
>
> And also source a profile.d shell if available
>
> # Source the profile environment variables if available.
> PROFILE_ENV="/etc/profile.d/mesos-profile-env.sh"
> if [ -f "$PROFILE_ENV" ]; then
> source "$PROFILE_ENV"
> fi
>
> Then support a configuration file, which basically defines what can be set as options, such that you can again have a default or override for a specific package (local, master, slave)
>
> # Setup Configuration Directory
> if [ -f "/etc/mesos/mesos-master.conf" ]; then
> CONFIGFILE="/etc/mesos/mesos-master.conf"
> else
> CONFIGFILE="/etc/mesos/mesos.conf"
> fi
>
> Apply the conf file elements as arguments/options to either the master, slave, or local.
>
> # Initialize the _Command_ options.
> OPTIONS="--log_dir=$LOG_DIR"
> if [ -n "$CONFIGFILE" ]; then
> OPTIONS=`paste -d " " <(cat /etc/mesos/master-master.conf | grep -e "^[--]" )`
> fi
>
>
> Example references.
> 1. "mesos-profile-env.sh" https://github.com/Guavus/incubator-mesos/blob/support/guavus-master/src/deploy/mesos-profile-env.sh.in
> 2. "mesos-master init.d script for rhl" https://github.com/Guavus/incubator-mesos/blob/support/guavus-master/src/deploy/mesos-masterd.sh.in
>
> Best
> Bernardo
>
>
> On Jul 12, 2013, at 12:13 PM, Jason Dusek <ja...@gmail.com> wrote:
>
>> 2013/7/11 Daniel Lundin <dl...@eintr.org>:
>>> As an option, I've attached the debian packaging patch I've
>>> used for packaging Mesos for Ubuntu (Precise) to MESOS-74. It
>>> does not use a third party / unified packager, rather just the
>>> standard toolchain - A debian directory in the source tree
>>> with all the trimmings. Just run "debuild ..." at the top and,
>>> ideally, you're good to go.
>>>
>>> Overall, in my experience it's hard to maintain packages for
>>> multiple platforms using a "meta-packager" scheme on a
>>> sizeable/multi-language project. It's usually easier to just
>>> go with the romans, whomever they may be at that particular
>>> juncture. Different systems have different policies for
>>> packages, how to split them up (libs, dev, language-specific
>>> bits), dealing with multi-architecture, python versions, etc.
>>> I've always ended up with *more* work trying to unify system
>>> packaging than just doing it once per platform. :)
>>
>> This really seems right to me. The meta package manager might
>> get you something the installer will accept; but not something
>> the distro will accept.
>>
>>> It might be worth considering it can be easier to find people
>>> experienced, interested and opinionated in platform-specific
>>> packaging, and the work tends to be low-hanging fruit for
>>> contributing to a project.
>>>
>>> In any case, hope the patch comes in handy.
>>
>> In its pretty form, this is pretty hard to work with. If there
>> is anyway you could create branch on GitHub...
>>
>> Of course I will still try to read it over the weekend, anyways.
>>
>> An open question for the project is how we want to manage Mesos
>> configuration. In the branch I'm working on, it's all
>> environment variables; and /etc/mesos/zk is loaded into an
>> environment variable and is the only "config file". We could
>> have a `mesos.conf' but as far as I understand it, file-based
>> config is slated for removal.
>>
>> --
>> Jason Dusek
>> pgp // solidsnack // C1EBC57DC55144F35460C8DF1FD4C6C1FED18A2B
>
Re: Merging in packaging tools for Mesos
Posted by Bernardo Gomez Palacio <be...@gmail.com>.
Its trivial to create the few files you mention, its more on keeping behavior consistent across cookbooks. That said I'll align to what makes more sense to everyone.
On Jul 15, 2013, at 12:35 PM, Jason Dusek <ja...@gmail.com> wrote:
> 2013/7/15 Bernardo Gomez Palacio <be...@gmail.com>:
>> IMHO from a devops perspective its easier to have a file (i
>> can have that file as a Opscode Chef cookbook template).
>
> For people who have that, it's great -- but not everyone has
> Chef or Puppet and when you think about configuration from
> debconf settings and the like, templating in to files is pretty
> rough.
>
> Is a file really much easier in Chef than creating a few files?
> It's been awhile since I've used Chef but I'm sure it's just a
> for loop; and that has the nice effect of pushing things out of
> templates and into code.
>
> --
> Jason Dusek
> pgp // solidsnack // C1EBC57DC55144F35460C8DF1FD4C6C1FED18A2B
Re: Merging in packaging tools for Mesos
Posted by Jason Dusek <ja...@gmail.com>.
2013/7/15 Bernardo Gomez Palacio <be...@gmail.com>:
> IMHO from a devops perspective its easier to have a file (i
> can have that file as a Opscode Chef cookbook template).
For people who have that, it's great -- but not everyone has
Chef or Puppet and when you think about configuration from
debconf settings and the like, templating in to files is pretty
rough.
Is a file really much easier in Chef than creating a few files?
It's been awhile since I've used Chef but I'm sure it's just a
for loop; and that has the nice effect of pushing things out of
templates and into code.
--
Jason Dusek
pgp // solidsnack // C1EBC57DC55144F35460C8DF1FD4C6C1FED18A2B
Re: Merging in packaging tools for Mesos
Posted by Bernardo Gomez Palacio <be...@gmail.com>.
IMHO from a devops perspective its easier to have a file (i can have that file as a Opscode Chef cookbook template).
Best
Bernardo.
On Jul 14, 2013, at 3:58 PM, Jason Dusek <ja...@gmail.com> wrote:
> What do you all think about an "envdir" like setup, but for
> options? The motivation is ease of automated configuration
> management: cat and echo are easier than sed. So the conf
> directory would look like:
>
> /etc/
> mesos/
> zk
> log_dir
> ...
>
> Then we'd construct the command line thusly:
>
> args=()
> cd /etc/mesos
> for f in *
> do args=( "${args[@]:+${args[@]}}" --"$f" "$(cat "$f")" )
> done
>
> This implementation requires a Bash feature, namely arrays, but
> Bash is fairly widespread now.
>
> --
> Jason Dusek
> pgp // solidsnack // C1EBC57DC55144F35460C8DF1FD4C6C1FED18A2B
>
>
> 2013/7/12 Bernardo Gomez Palacio <be...@gmail.com>:
>> Amending
>>
>> On Fri, Jul 12, 2013 at 12:38 PM, Bernardo Gomez Palacio <
>> bernardo.gomezpalacio@gmail.com> wrote:
>>
>>> # Initialize the _Command_ options.
>>> OPTIONS="--log_dir=$LOG_DIR"
>>> if [ -n "$CONFIGFILE" ]; then
>>> OPTIONS=`paste -d " " <(cat $CONFIGFILE | grep -e "^[--]" )`
>>> fi
>>>
Re: Merging in packaging tools for Mesos
Posted by Jason Dusek <ja...@gmail.com>.
What do you all think about an "envdir" like setup, but for
options? The motivation is ease of automated configuration
management: cat and echo are easier than sed. So the conf
directory would look like:
/etc/
mesos/
zk
log_dir
...
Then we'd construct the command line thusly:
args=()
cd /etc/mesos
for f in *
do args=( "${args[@]:+${args[@]}}" --"$f" "$(cat "$f")" )
done
This implementation requires a Bash feature, namely arrays, but
Bash is fairly widespread now.
--
Jason Dusek
pgp // solidsnack // C1EBC57DC55144F35460C8DF1FD4C6C1FED18A2B
2013/7/12 Bernardo Gomez Palacio <be...@gmail.com>:
> Amending
>
> On Fri, Jul 12, 2013 at 12:38 PM, Bernardo Gomez Palacio <
> bernardo.gomezpalacio@gmail.com> wrote:
>
>> # Initialize the _Command_ options.
>> OPTIONS="--log_dir=$LOG_DIR"
>> if [ -n "$CONFIGFILE" ]; then
>> OPTIONS=`paste -d " " <(cat $CONFIGFILE | grep -e "^[--]" )`
>> fi
>>
Re: Merging in packaging tools for Mesos
Posted by Bernardo Gomez Palacio <be...@gmail.com>.
Amending
On Fri, Jul 12, 2013 at 12:38 PM, Bernardo Gomez Palacio <
bernardo.gomezpalacio@gmail.com> wrote:
> # Initialize the _Command_ options.
> OPTIONS="--log_dir=$LOG_DIR"
> if [ -n "$CONFIGFILE" ]; then
> OPTIONS=`paste -d " " <(cat $CONFIGFILE | grep -e "^[--]" )`
> fi
>
Re: Merging in packaging tools for Mesos
Posted by Bernardo Gomez Palacio <be...@gmail.com>.
At least in master, 0.14.0, file based configuration with the `--conf` options is not supported. That said imho we could follow the approach stated bellow.
Within the execution of the init.d script source a default env shell. ( extracted from mesos-profile-env.sh )
# Source the default environment variables.
DEFAULT_ENV="$PRGDIR/mesos-profile-env.sh"
if [ -f "$DEFAULT_ENV"]
source $DEFAULT_ENV
fi
And also source a profile.d shell if available
# Source the profile environment variables if available.
PROFILE_ENV="/etc/profile.d/mesos-profile-env.sh"
if [ -f "$PROFILE_ENV" ]; then
source "$PROFILE_ENV"
fi
Then support a configuration file, which basically defines what can be set as options, such that you can again have a default or override for a specific package (local, master, slave)
# Setup Configuration Directory
if [ -f "/etc/mesos/mesos-master.conf" ]; then
CONFIGFILE="/etc/mesos/mesos-master.conf"
else
CONFIGFILE="/etc/mesos/mesos.conf"
fi
Apply the conf file elements as arguments/options to either the master, slave, or local.
# Initialize the _Command_ options.
OPTIONS="--log_dir=$LOG_DIR"
if [ -n "$CONFIGFILE" ]; then
OPTIONS=`paste -d " " <(cat /etc/mesos/master-master.conf | grep -e "^[--]" )`
fi
Example references.
1. "mesos-profile-env.sh" https://github.com/Guavus/incubator-mesos/blob/support/guavus-master/src/deploy/mesos-profile-env.sh.in
2. "mesos-master init.d script for rhl" https://github.com/Guavus/incubator-mesos/blob/support/guavus-master/src/deploy/mesos-masterd.sh.in
Best
Bernardo
On Jul 12, 2013, at 12:13 PM, Jason Dusek <ja...@gmail.com> wrote:
> 2013/7/11 Daniel Lundin <dl...@eintr.org>:
>> As an option, I've attached the debian packaging patch I've
>> used for packaging Mesos for Ubuntu (Precise) to MESOS-74. It
>> does not use a third party / unified packager, rather just the
>> standard toolchain - A debian directory in the source tree
>> with all the trimmings. Just run "debuild ..." at the top and,
>> ideally, you're good to go.
>>
>> Overall, in my experience it's hard to maintain packages for
>> multiple platforms using a "meta-packager" scheme on a
>> sizeable/multi-language project. It's usually easier to just
>> go with the romans, whomever they may be at that particular
>> juncture. Different systems have different policies for
>> packages, how to split them up (libs, dev, language-specific
>> bits), dealing with multi-architecture, python versions, etc.
>> I've always ended up with *more* work trying to unify system
>> packaging than just doing it once per platform. :)
>
> This really seems right to me. The meta package manager might
> get you something the installer will accept; but not something
> the distro will accept.
>
>> It might be worth considering it can be easier to find people
>> experienced, interested and opinionated in platform-specific
>> packaging, and the work tends to be low-hanging fruit for
>> contributing to a project.
>>
>> In any case, hope the patch comes in handy.
>
> In its pretty form, this is pretty hard to work with. If there
> is anyway you could create branch on GitHub...
>
> Of course I will still try to read it over the weekend, anyways.
>
> An open question for the project is how we want to manage Mesos
> configuration. In the branch I'm working on, it's all
> environment variables; and /etc/mesos/zk is loaded into an
> environment variable and is the only "config file". We could
> have a `mesos.conf' but as far as I understand it, file-based
> config is slated for removal.
>
> --
> Jason Dusek
> pgp // solidsnack // C1EBC57DC55144F35460C8DF1FD4C6C1FED18A2B
Re: Merging in packaging tools for Mesos
Posted by Daniel Lundin <dl...@eintr.org>.
> In its pretty form, this is pretty hard to work with. If there
> is anyway you could create branch on GitHub...
Absolutely, here's the very branch whence it came:
https://github.com/dln/incubator-mesos/tree/debian-package
Should be a bit easier to read. :)
I noticed when double-checking today it's a bit incomplete, specifically
missing a dependency on a jdk (for libjvm). I'll go ahead and fix this.
> An open question for the project is how we want to manage Mesos
> configuration.
+1. Config file, env vars *and* flags is probably two options too many.
Personally, I'm all for only using environment vars and removing
"/etc/mesos/mesos.conf" altogether. Maybe have defaults in
/etc/default/mesos and also sourcing /etc/mesos/mesos-env.sh for "user
settings"?
/d
On Fri, Jul 12, 2013 at 9:13 PM, Jason Dusek <ja...@gmail.com> wrote:
> 2013/7/11 Daniel Lundin <dl...@eintr.org>:
> > As an option, I've attached the debian packaging patch I've
> > used for packaging Mesos for Ubuntu (Precise) to MESOS-74. It
> > does not use a third party / unified packager, rather just the
> > standard toolchain - A debian directory in the source tree
> > with all the trimmings. Just run "debuild ..." at the top and,
> > ideally, you're good to go.
> >
> > Overall, in my experience it's hard to maintain packages for
> > multiple platforms using a "meta-packager" scheme on a
> > sizeable/multi-language project. It's usually easier to just
> > go with the romans, whomever they may be at that particular
> > juncture. Different systems have different policies for
> > packages, how to split them up (libs, dev, language-specific
> > bits), dealing with multi-architecture, python versions, etc.
> > I've always ended up with *more* work trying to unify system
> > packaging than just doing it once per platform. :)
>
> This really seems right to me. The meta package manager might
> get you something the installer will accept; but not something
> the distro will accept.
>
> > It might be worth considering it can be easier to find people
> > experienced, interested and opinionated in platform-specific
> > packaging, and the work tends to be low-hanging fruit for
> > contributing to a project.
> >
> > In any case, hope the patch comes in handy.
>
> In its pretty form, this is pretty hard to work with. If there
> is anyway you could create branch on GitHub...
>
> Of course I will still try to read it over the weekend, anyways.
>
> An open question for the project is how we want to manage Mesos
> configuration. In the branch I'm working on, it's all
> environment variables; and /etc/mesos/zk is loaded into an
> environment variable and is the only "config file". We could
> have a `mesos.conf' but as far as I understand it, file-based
> config is slated for removal.
>
> --
> Jason Dusek
> pgp // solidsnack // C1EBC57DC55144F35460C8DF1FD4C6C1FED18A2B
>
Re: Merging in packaging tools for Mesos
Posted by Jason Dusek <ja...@gmail.com>.
2013/7/11 Daniel Lundin <dl...@eintr.org>:
> As an option, I've attached the debian packaging patch I've
> used for packaging Mesos for Ubuntu (Precise) to MESOS-74. It
> does not use a third party / unified packager, rather just the
> standard toolchain - A debian directory in the source tree
> with all the trimmings. Just run "debuild ..." at the top and,
> ideally, you're good to go.
>
> Overall, in my experience it's hard to maintain packages for
> multiple platforms using a "meta-packager" scheme on a
> sizeable/multi-language project. It's usually easier to just
> go with the romans, whomever they may be at that particular
> juncture. Different systems have different policies for
> packages, how to split them up (libs, dev, language-specific
> bits), dealing with multi-architecture, python versions, etc.
> I've always ended up with *more* work trying to unify system
> packaging than just doing it once per platform. :)
This really seems right to me. The meta package manager might
get you something the installer will accept; but not something
the distro will accept.
> It might be worth considering it can be easier to find people
> experienced, interested and opinionated in platform-specific
> packaging, and the work tends to be low-hanging fruit for
> contributing to a project.
>
> In any case, hope the patch comes in handy.
In its pretty form, this is pretty hard to work with. If there
is anyway you could create branch on GitHub...
Of course I will still try to read it over the weekend, anyways.
An open question for the project is how we want to manage Mesos
configuration. In the branch I'm working on, it's all
environment variables; and /etc/mesos/zk is loaded into an
environment variable and is the only "config file". We could
have a `mesos.conf' but as far as I understand it, file-based
config is slated for removal.
--
Jason Dusek
pgp // solidsnack // C1EBC57DC55144F35460C8DF1FD4C6C1FED18A2B
Re: Merging in packaging tools for Mesos
Posted by Tim St Clair <ts...@redhat.com>.
re: Red Hat packaging.
I can help Sheppard it, and have opened MESOS-543 as a start. When the time comes we can kick-start the channels (Fedora || EPEL) when it gets further down the road. Ideally helping to promote adherence to certain guidelines enables down-streams to easily package, and we can promote co-maintainers for the channels as they are typically community driven.
Cheers,
Tim [Red Hat "Roman" ;-)]
----- Original Message -----
> From: "Daniel Lundin" <dl...@eintr.org>
> To: dev@mesos.apache.org
> Sent: Thursday, July 11, 2013 3:47:17 PM
> Subject: Re: Merging in packaging tools for Mesos
>
> Hey guys,
>
> As an option, I've attached the debian packaging patch I've used for
> packaging Mesos for Ubuntu (Precise) to MESOS-74. It does not use a third
> party / unified packager, rather just the standard toolchain - A debian
> directory in the source tree with all the trimmings. Just run "debuild ..."
> at the top and, ideally, you're good to go.
>
> Overall, in my experience it's hard to maintain packages for multiple
> platforms using a "meta-packager" scheme on a sizeable/multi-language
> project. It's usually easier to just go with the romans, whomever they may
> be at that particular juncture. Different systems have different policies
> for packages, how to split them up (libs, dev, language-specific bits),
> dealing with multi-architecture, python versions, etc. I've always ended up
> with *more* work trying to unify system packaging than just doing it once
> per platform. :)
>
> It might be worth considering it can be easier to find people experienced,
> interested and opinionated in platform-specific packaging, and the work
> tends to be low-hanging fruit for contributing to a project.
>
> In any case, hope the patch comes in handy.
>
> Cheers,
> /d
>
>
>
> On Wed, Jul 10, 2013 at 7:50 PM, Benjamin Hindman
> <be...@eecs.berkeley.edu>wrote:
>
> > +1!
> >
> >
> > On Wed, Jul 10, 2013 at 10:48 AM, Vinod Kone <vi...@gmail.com> wrote:
> >
> > > Hey Jason and Tomas,
> > >
> > > Love it! We definitely like to include this in the mesos repo. Would you
> > > mind sending out the patch(es) via Apache ReviewBoard? That is our
> > > preferred way to review changes before committing them.
> > >
> > > P.S: Since mesos graduated, the new repo location is:
> > > https://git-wip-us.apache.org/repos/asf/mesos.git
> > >
> > > Thanks,
> > >
> > >
> > > On Wed, Jul 10, 2013 at 10:27 AM, Jason Dusek <ja...@gmail.com>
> > > wrote:
> > >
> > > > Dear List,
> > > >
> > > > I have been working on automated packaging for Mesos on Debian
> > > > and Ubuntu.
> > > >
> > > > https://github.com/solidsnack/mesos-deb-packaging/
> > > >
> > > > Much thanks is due to Tomas Barton, who got the ball rolling on
> > > > this and put together much of the code in the repo above.
> > > >
> > > > The present collection of init scripts and build options is used
> > > > regularly for automated rollouts of Mesos at my firm, on Debian
> > > > and Ubuntu, in conjunction with tools using the Java API. We are
> > > > looking in to proper installation and setup of the Python
> > > > libraries.
> > > >
> > > > The `build_mesos' script is useful for those wishing to build
> > > > and release internal forks of Mesos, too.
> > > >
> > > > Support for RedHat/CentOS is important to us and is planned for
> > > > the near future. We'd like to see Mesos make its way in to the
> > > > repositories someday. What we have is a solid foundation and it
> > > > seems like integrating with mainline Mesos early is the best way
> > > > to make for a smooth transition in the future.
> > > >
> > > > Would it be alright to add this material to
> > > >
> > > > http://git.apache.org/incubator-mesos.git
> > > >
> > > > under a new directory, "packaging"? This tooling will eventually
> > > > grow to encompass Homebrew as well as RPMs so its probably just
> > > > as well to keep the name distro agnostic.
> > > >
> > > > --
> > > > Jason Dusek
> > > > pgp // solidsnack // C1EBC57DC55144F35460C8DF1FD4C6C1FED18A2B
> > > >
> > >
> >
>
Re: Merging in packaging tools for Mesos
Posted by Daniel Lundin <dl...@eintr.org>.
Hey guys,
As an option, I've attached the debian packaging patch I've used for
packaging Mesos for Ubuntu (Precise) to MESOS-74. It does not use a third
party / unified packager, rather just the standard toolchain - A debian
directory in the source tree with all the trimmings. Just run "debuild ..."
at the top and, ideally, you're good to go.
Overall, in my experience it's hard to maintain packages for multiple
platforms using a "meta-packager" scheme on a sizeable/multi-language
project. It's usually easier to just go with the romans, whomever they may
be at that particular juncture. Different systems have different policies
for packages, how to split them up (libs, dev, language-specific bits),
dealing with multi-architecture, python versions, etc. I've always ended up
with *more* work trying to unify system packaging than just doing it once
per platform. :)
It might be worth considering it can be easier to find people experienced,
interested and opinionated in platform-specific packaging, and the work
tends to be low-hanging fruit for contributing to a project.
In any case, hope the patch comes in handy.
Cheers,
/d
On Wed, Jul 10, 2013 at 7:50 PM, Benjamin Hindman <be...@eecs.berkeley.edu>wrote:
> +1!
>
>
> On Wed, Jul 10, 2013 at 10:48 AM, Vinod Kone <vi...@gmail.com> wrote:
>
> > Hey Jason and Tomas,
> >
> > Love it! We definitely like to include this in the mesos repo. Would you
> > mind sending out the patch(es) via Apache ReviewBoard? That is our
> > preferred way to review changes before committing them.
> >
> > P.S: Since mesos graduated, the new repo location is:
> > https://git-wip-us.apache.org/repos/asf/mesos.git
> >
> > Thanks,
> >
> >
> > On Wed, Jul 10, 2013 at 10:27 AM, Jason Dusek <ja...@gmail.com>
> > wrote:
> >
> > > Dear List,
> > >
> > > I have been working on automated packaging for Mesos on Debian
> > > and Ubuntu.
> > >
> > > https://github.com/solidsnack/mesos-deb-packaging/
> > >
> > > Much thanks is due to Tomas Barton, who got the ball rolling on
> > > this and put together much of the code in the repo above.
> > >
> > > The present collection of init scripts and build options is used
> > > regularly for automated rollouts of Mesos at my firm, on Debian
> > > and Ubuntu, in conjunction with tools using the Java API. We are
> > > looking in to proper installation and setup of the Python
> > > libraries.
> > >
> > > The `build_mesos' script is useful for those wishing to build
> > > and release internal forks of Mesos, too.
> > >
> > > Support for RedHat/CentOS is important to us and is planned for
> > > the near future. We'd like to see Mesos make its way in to the
> > > repositories someday. What we have is a solid foundation and it
> > > seems like integrating with mainline Mesos early is the best way
> > > to make for a smooth transition in the future.
> > >
> > > Would it be alright to add this material to
> > >
> > > http://git.apache.org/incubator-mesos.git
> > >
> > > under a new directory, "packaging"? This tooling will eventually
> > > grow to encompass Homebrew as well as RPMs so its probably just
> > > as well to keep the name distro agnostic.
> > >
> > > --
> > > Jason Dusek
> > > pgp // solidsnack // C1EBC57DC55144F35460C8DF1FD4C6C1FED18A2B
> > >
> >
>
Re: Merging in packaging tools for Mesos
Posted by Benjamin Mahler <bm...@twitter.com>.
Great work Jason and Thomas!
On Wed, Jul 10, 2013 at 10:50 AM, Benjamin Hindman
<be...@eecs.berkeley.edu>wrote:
> +1!
>
>
> On Wed, Jul 10, 2013 at 10:48 AM, Vinod Kone <vi...@gmail.com> wrote:
>
> > Hey Jason and Tomas,
> >
> > Love it! We definitely like to include this in the mesos repo. Would you
> > mind sending out the patch(es) via Apache ReviewBoard? That is our
> > preferred way to review changes before committing them.
> >
> > P.S: Since mesos graduated, the new repo location is:
> > https://git-wip-us.apache.org/repos/asf/mesos.git
> >
> > Thanks,
> >
> >
> > On Wed, Jul 10, 2013 at 10:27 AM, Jason Dusek <ja...@gmail.com>
> > wrote:
> >
> > > Dear List,
> > >
> > > I have been working on automated packaging for Mesos on Debian
> > > and Ubuntu.
> > >
> > > https://github.com/solidsnack/mesos-deb-packaging/
> > >
> > > Much thanks is due to Tomas Barton, who got the ball rolling on
> > > this and put together much of the code in the repo above.
> > >
> > > The present collection of init scripts and build options is used
> > > regularly for automated rollouts of Mesos at my firm, on Debian
> > > and Ubuntu, in conjunction with tools using the Java API. We are
> > > looking in to proper installation and setup of the Python
> > > libraries.
> > >
> > > The `build_mesos' script is useful for those wishing to build
> > > and release internal forks of Mesos, too.
> > >
> > > Support for RedHat/CentOS is important to us and is planned for
> > > the near future. We'd like to see Mesos make its way in to the
> > > repositories someday. What we have is a solid foundation and it
> > > seems like integrating with mainline Mesos early is the best way
> > > to make for a smooth transition in the future.
> > >
> > > Would it be alright to add this material to
> > >
> > > http://git.apache.org/incubator-mesos.git
> > >
> > > under a new directory, "packaging"? This tooling will eventually
> > > grow to encompass Homebrew as well as RPMs so its probably just
> > > as well to keep the name distro agnostic.
> > >
> > > --
> > > Jason Dusek
> > > pgp // solidsnack // C1EBC57DC55144F35460C8DF1FD4C6C1FED18A2B
> > >
> >
>
Re: Merging in packaging tools for Mesos
Posted by Benjamin Hindman <be...@eecs.berkeley.edu>.
+1!
On Wed, Jul 10, 2013 at 10:48 AM, Vinod Kone <vi...@gmail.com> wrote:
> Hey Jason and Tomas,
>
> Love it! We definitely like to include this in the mesos repo. Would you
> mind sending out the patch(es) via Apache ReviewBoard? That is our
> preferred way to review changes before committing them.
>
> P.S: Since mesos graduated, the new repo location is:
> https://git-wip-us.apache.org/repos/asf/mesos.git
>
> Thanks,
>
>
> On Wed, Jul 10, 2013 at 10:27 AM, Jason Dusek <ja...@gmail.com>
> wrote:
>
> > Dear List,
> >
> > I have been working on automated packaging for Mesos on Debian
> > and Ubuntu.
> >
> > https://github.com/solidsnack/mesos-deb-packaging/
> >
> > Much thanks is due to Tomas Barton, who got the ball rolling on
> > this and put together much of the code in the repo above.
> >
> > The present collection of init scripts and build options is used
> > regularly for automated rollouts of Mesos at my firm, on Debian
> > and Ubuntu, in conjunction with tools using the Java API. We are
> > looking in to proper installation and setup of the Python
> > libraries.
> >
> > The `build_mesos' script is useful for those wishing to build
> > and release internal forks of Mesos, too.
> >
> > Support for RedHat/CentOS is important to us and is planned for
> > the near future. We'd like to see Mesos make its way in to the
> > repositories someday. What we have is a solid foundation and it
> > seems like integrating with mainline Mesos early is the best way
> > to make for a smooth transition in the future.
> >
> > Would it be alright to add this material to
> >
> > http://git.apache.org/incubator-mesos.git
> >
> > under a new directory, "packaging"? This tooling will eventually
> > grow to encompass Homebrew as well as RPMs so its probably just
> > as well to keep the name distro agnostic.
> >
> > --
> > Jason Dusek
> > pgp // solidsnack // C1EBC57DC55144F35460C8DF1FD4C6C1FED18A2B
> >
>
Re: Merging in packaging tools for Mesos
Posted by Vinod Kone <vi...@gmail.com>.
Hey Jason and Tomas,
Love it! We definitely like to include this in the mesos repo. Would you
mind sending out the patch(es) via Apache ReviewBoard? That is our
preferred way to review changes before committing them.
P.S: Since mesos graduated, the new repo location is:
https://git-wip-us.apache.org/repos/asf/mesos.git
Thanks,
On Wed, Jul 10, 2013 at 10:27 AM, Jason Dusek <ja...@gmail.com> wrote:
> Dear List,
>
> I have been working on automated packaging for Mesos on Debian
> and Ubuntu.
>
> https://github.com/solidsnack/mesos-deb-packaging/
>
> Much thanks is due to Tomas Barton, who got the ball rolling on
> this and put together much of the code in the repo above.
>
> The present collection of init scripts and build options is used
> regularly for automated rollouts of Mesos at my firm, on Debian
> and Ubuntu, in conjunction with tools using the Java API. We are
> looking in to proper installation and setup of the Python
> libraries.
>
> The `build_mesos' script is useful for those wishing to build
> and release internal forks of Mesos, too.
>
> Support for RedHat/CentOS is important to us and is planned for
> the near future. We'd like to see Mesos make its way in to the
> repositories someday. What we have is a solid foundation and it
> seems like integrating with mainline Mesos early is the best way
> to make for a smooth transition in the future.
>
> Would it be alright to add this material to
>
> http://git.apache.org/incubator-mesos.git
>
> under a new directory, "packaging"? This tooling will eventually
> grow to encompass Homebrew as well as RPMs so its probably just
> as well to keep the name distro agnostic.
>
> --
> Jason Dusek
> pgp // solidsnack // C1EBC57DC55144F35460C8DF1FD4C6C1FED18A2B
>