You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by F21 <f2...@gmail.com> on 2015/09/18 01:33:04 UTC
Building portable binaries
Is there anyway to build portable binaries for mesos?
Currently, I have tried building my own libsvn, libsasl2, libcurl,
libapr and then built mesos using the following:
../configure CC=gcc-4.8 CXX=g++-4.8
LD_LIBRARY_PATH=/tmp/mesos-build/sasl2/lib
SASL_PATH=/tmp/mesos-build/sasl2/lib/sasl2
--prefix=/tmp/mesos-build/mesos --with-svn=/tmp/mesos-build/svn
--with-apr=/tmp/mesos-build/apr --with-sasl=/tmp/mesos-build/sasl2/
--with-curl=/tmp/mesos-build/curl
make
make install
I then compress /tmp/mesos-build/mesos into an archive and distribute it
to my machines. The only problem is that the build seems to be buggy.
For example, I've been experiencing a containerization issues where the
executors will crash, but not output anything useful to stderr and
stdout. See https://github.com/mesosphere/hdfs/issues/194
Is there a definite way to build portable binaries that I can easily
copy to another machine to run?
Re: Building portable binaries
Posted by CCAAT <cc...@tampabay.rr.com>.
First, here is a very cool gentoo vm with btrfs in a raid-one config.
https://docs.google.com/document/d/1VJlJyYLTZScta9a81xgKOIBjYsG3_VfxxmUSxG23Uxg/edit?pli=1
I'll clean up the ebuild and post tomorrow. I got an old spark and
zookeeper is floating around. I got sidetracked into working on
a rapid install semantic for gentoo....
Tomorrow.
James
On 09/17/2015 11:09 PM, F21 wrote:
> That sounds really interesting! I am just in the process of spinning up
> a gentoo vm.
>
> Would you mind sharing your ebuild for mesos-0.22.0 via a gist on Github?
>
> On 18/09/2015 12:58 PM, CCAAT wrote:
>> On 09/17/2015 06:33 PM, F21 wrote:
>>> Is there anyway to build portable binaries for mesos?
>>
>>
>> You should try out gentoo linux, everything is built from sources.
>>
>> Ebuilds guide the process. My (hack) ebuild for mesos-0.22.0 was
>> 61 lines. That's it. I will roll out a 0.24 ebuild, in a few weeks
>> or less.
>>
>> Gentoo is designed from the ground up to build form sources. We have
>> a rich 'cross-compile' environment for things like aarch64; so building
>> mesos for arm64 is mostly trivial, once the 0.24 ebuild is rolled out.
>>
>>
>> There a bit of reading, but the gentoo 'devmanual' pretty much guides
>> you through the process [1]. Gentoo also has a great package manager.
>> Here is a (very profane) rant/comparison of some common package managers
>> and their inherent weaknesses [2]. If you want to see how simple the
>> gentoo ebuild for mesos-0.22 is just ask. It fetches, unpacks, compiles
>> and installs the package, very neatly. And there is lots of help and
>> encouragement from a long list of talented devs.
>>
>> Gentoo is not for the weak minded or folks that do not wish to master
>> the deep details of linux. caveat emptor. CoreOS uses much of gentoo
>> in it's build/management, if that option helps.
>>
>>
>> hth,
>> James
>>
>>
>>
>>
>>
>> [1] https://devmanual.gentoo.org/
>>
>>
>> [2]
>> http://michael.orlitzky.com/articles/motherfuckers_need_package_management.php
>>
>
>
Re: Building portable binaries
Posted by CCAAT <cc...@tampabay.rr.com>.
On 09/19/2015 07:06 PM, F21 wrote:
> Hey James,
>
> Thanks for sharing the ebuild! I spun up gentoo as a docker container
> and copied the ebuild to
> /usr/local/portage/clustering/mesos/mesos-0.22.0.ebuild.
Hello F21,
First, apologies for not setting up a more appropriate repo. Github
is on my 'todo' list. I code command line and work mostly on
isolated systems, as an old school coder.
I'm no VM nor container concierge so for those types of setups, you are
going to have to find expertise, documents and such elsewhere. You would
be best advise to join the gentoo-user community to access a very deep
and wide bench of experts, all centric to gentoo. There are numerous
forums and irc channels too.
Traditionally, Gentoo puts the portage tree into /usr/portage. The
ebuilds tarballs are downloaded to /usr/portage/distfiles for raw
storage by the ebuild. Some are now using /var/portage/.
I use /usr/local/portage to hack new ebuilds before allowing others
to test. With gentoo you can put things where you like, but some
adjustments need to be made. When I roll all of this out it will be
much easier to just point to an overlay and install from there. But,
I have yet to put an overlay up. Here is one for zookeeper::
[U] sys-cluster/zookeeper
Available versions: (~)3.4.6^mb[3] (~)3.4.6[2] (~)3.4.6^mb[5]
(~)3.4.6-r1^mb[1] {ELIBC="FreeBSD" PYTHON_TARGETS="python2_7"}
Installed versions: 3.4.6^mb[2](02:35:35 PM
09/19/2015)(ELIBC="-FreeBSD" PYTHON_TARGETS="python2_7")
Homepage: http://zookeeper.apache.org/
Description: ZooKeeper is a high-performance coordination
service for distributed applications.
* sys-cluster/apache-zookeeper [4]
Available versions: ~3.3.6^mb {ELIBC="FreeBSD"}
Homepage: http://hadoop.apache.org/
Description: ZooKeeper is a high-performance coordination
service for distributed applications.
[1] "jackslap" /usr/local/portage
[2] "ultrabug" /var/lib/layman/ultrabug
[3] "fw-overlay" layman/fw-overlay
[4] "gentoo-zh" layman/gentoo-zh
[5] "ultrabug" layman/ultrabug
You can see others are hacking on creating a 100% build from source
solution for clustering. 'jackslap' is my local repo, which I just email
to you. I also subscribe to ultrabug's overlay repo, using the layman
tools to installation. Overlays have their own tree locations that can
vary from actual gentoo maintained servers, to github, to private
machines for developers, hackers and any other folks that code.
Gentoo has many derivative distros, some roll binaries, others are on a
rolling release, just like gentoo. Since the 'systemd wars', the fact
that gentoo has a robust "openrc" alternative to systemd means that
part of our developer community is growing very fast with new talent
and old coders alike. As more folks roll out clusters, kernel tuning,
a lost art for regular users at all major distros, is going to become
quintessentially important to get the best performance out of your
cluster/cloud. The way gentoo is set up, you can use an endless variety
of boot/kernel semantics to fit your needs. We have very robust tools
for kernel performance studies available with gentoo.
I work strictly on real hardware. I'm a 'bare metal' kind of EE hack.
That said, if you are serious about gentoo, my recommendation' would
be to use older hardware until you get everything working and figured
out what you want to do. Then VM, Container, Cloud or get new/cheap
cluster gear like aarch64 (arm64). YMMV depending on your goals.
> However, when I run ebuild mesos-0.22.0.ebuild manifest, it throws an
> error about the metadata:
>
> Appending /usr/local/portage to PORTDIR_OVERLAY...
> Error(s) in metadata for 'clustering/mesos-0.22.0':
> DEPEND: Invalid atom (.keep), token 12
> RDEPEND: Invalid atom (.keep), token 7
Make sure you do not have simple errors do to command line mistakes
in your copy process. Those errors are explained more fully in the
gentoo Devmanual. Depends are codes that are needed to compile your
mesos. RDEPENDS are runtime dependencies. Some codes are both.
> Any ideas why this might be happening?
Gentoo is not for the faint of heart. It is for those robust users
or for folks that really want to learn deeply how things work. The
great news about gentoo, although many will argue negatively, is that
in clusters, you can get 2-10% performance increase by minimizing codes,
particularly stripped and optimized kernels. Over hundreds of CPUs
that's going to be a huge performance difference. Gentoo had a very rich
echo system of distributed processing years ago. Most of those folks
took there expertise to corporations for exploit and let gentoo's
position in clustering languish. What gentoo needs now more than ever
are java coders, as the team is excellent, but under staffed, atm.
Now there is a very robust team of arm64 devs, a veritable who's who,
working bring gentoo to arm64. Others like myself are more focused on
the bare metal approach that works well for clusters, distributed
embedded systems, rock solid security and superior performance per watt
of power consumed.
You are most welcome to join us. Have you ever installed gentoo, on a
actual system? Right now, the pedantic approach dominants at Gentoo
via the arduous 'gentoo handbook'. I'd strongly suggest you endure
that pain, to become functionally literate with Gentoo. Several folks
are working on rapid install semantics for Gentoo on a myriad for
hardware architectures.
wwr,
James
>
> Cheers!
>
> On 20/09/2015 4:35 AM, CCAAT wrote:
>> Hey F21,
>>
>> The ebuild is attached. Hopefully I'll be setting up github for this
>> and other ebuilds I have been hacking on. I'm not a dev, but debugging
>> these ebuilds in not difficult [1].
>>
>>
>> Please let me know how this ebuild works for you. I never tested it
>> extensively....
>>
>>
>> James
>>
>>
>>
>>
>> On 09/17/2015 11:09 PM, F21 wrote:
>>> That sounds really interesting! I am just in the process of spinning up
>>> a gentoo vm.
>>>
>>> Would you mind sharing your ebuild for mesos-0.22.0 via a gist on
>>> Github?
>>>
>>> On 18/09/2015 12:58 PM, CCAAT wrote:
>>>> On 09/17/2015 06:33 PM, F21 wrote:
>>>>> Is there anyway to build portable binaries for mesos?
>>>>
>>>>
>>>> You should try out gentoo linux, everything is built from sources.
>>>>
>>>> Ebuilds guide the process. My (hack) ebuild for mesos-0.22.0 was
>>>> 61 lines. That's it. I will roll out a 0.24 ebuild, in a few weeks
>>>> or less.
>>>>
>>>> Gentoo is designed from the ground up to build form sources. We have
>>>> a rich 'cross-compile' environment for things like aarch64; so building
>>>> mesos for arm64 is mostly trivial, once the 0.24 ebuild is rolled out.
>>>>
>>>>
>>>> There a bit of reading, but the gentoo 'devmanual' pretty much guides
>>>> you through the process [1]. Gentoo also has a great package manager.
>>>> Here is a (very profane) rant/comparison of some common package
>>>> managers
>>>> and their inherent weaknesses [2]. If you want to see how simple the
>>>> gentoo ebuild for mesos-0.22 is just ask. It fetches, unpacks, compiles
>>>> and installs the package, very neatly. And there is lots of help and
>>>> encouragement from a long list of talented devs.
>>>>
>>>> Gentoo is not for the weak minded or folks that do not wish to master
>>>> the deep details of linux. caveat emptor. CoreOS uses much of gentoo
>>>> in it's build/management, if that option helps.
>>>>
>>>>
>>>> hth,
>>>> James
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> [1] https://devmanual.gentoo.org/
>>>>
>>>>
>>>> [2]
>>>> http://michael.orlitzky.com/articles/motherfuckers_need_package_management.php
>>>>
>>>>
>>>
>>>
>>
>
>
Re: Building portable binaries
Posted by F21 <f2...@gmail.com>.
Hey James,
Thanks for sharing the ebuild! I spun up gentoo as a docker container
and copied the ebuild to
/usr/local/portage/clustering/mesos/mesos-0.22.0.ebuild.
However, when I run ebuild mesos-0.22.0.ebuild manifest, it throws an
error about the metadata:
Appending /usr/local/portage to PORTDIR_OVERLAY...
Error(s) in metadata for 'clustering/mesos-0.22.0':
DEPEND: Invalid atom (.keep), token 12
RDEPEND: Invalid atom (.keep), token 7
Any ideas why this might be happening?
Cheers!
On 20/09/2015 4:35 AM, CCAAT wrote:
> Hey F21,
>
> The ebuild is attached. Hopefully I'll be setting up github for this
> and other ebuilds I have been hacking on. I'm not a dev, but debugging
> these ebuilds in not difficult [1].
>
>
> Please let me know how this ebuild works for you. I never tested it
> extensively....
>
>
> James
>
>
>
>
> On 09/17/2015 11:09 PM, F21 wrote:
>> That sounds really interesting! I am just in the process of spinning up
>> a gentoo vm.
>>
>> Would you mind sharing your ebuild for mesos-0.22.0 via a gist on
>> Github?
>>
>> On 18/09/2015 12:58 PM, CCAAT wrote:
>>> On 09/17/2015 06:33 PM, F21 wrote:
>>>> Is there anyway to build portable binaries for mesos?
>>>
>>>
>>> You should try out gentoo linux, everything is built from sources.
>>>
>>> Ebuilds guide the process. My (hack) ebuild for mesos-0.22.0 was
>>> 61 lines. That's it. I will roll out a 0.24 ebuild, in a few weeks
>>> or less.
>>>
>>> Gentoo is designed from the ground up to build form sources. We have
>>> a rich 'cross-compile' environment for things like aarch64; so building
>>> mesos for arm64 is mostly trivial, once the 0.24 ebuild is rolled out.
>>>
>>>
>>> There a bit of reading, but the gentoo 'devmanual' pretty much guides
>>> you through the process [1]. Gentoo also has a great package manager.
>>> Here is a (very profane) rant/comparison of some common package
>>> managers
>>> and their inherent weaknesses [2]. If you want to see how simple the
>>> gentoo ebuild for mesos-0.22 is just ask. It fetches, unpacks, compiles
>>> and installs the package, very neatly. And there is lots of help and
>>> encouragement from a long list of talented devs.
>>>
>>> Gentoo is not for the weak minded or folks that do not wish to master
>>> the deep details of linux. caveat emptor. CoreOS uses much of gentoo
>>> in it's build/management, if that option helps.
>>>
>>>
>>> hth,
>>> James
>>>
>>>
>>>
>>>
>>>
>>> [1] https://devmanual.gentoo.org/
>>>
>>>
>>> [2]
>>> http://michael.orlitzky.com/articles/motherfuckers_need_package_management.php
>>>
>>>
>>
>>
>
Re: Building portable binaries
Posted by CCAAT <cc...@tampabay.rr.com>.
Hey F21,
The ebuild is attached. Hopefully I'll be setting up github for this and
other ebuilds I have been hacking on. I'm not a dev, but debugging these
ebuilds in not difficult [1].
Please let me know how this ebuild works for you. I never tested it
extensively....
James
On 09/17/2015 11:09 PM, F21 wrote:
> That sounds really interesting! I am just in the process of spinning up
> a gentoo vm.
>
> Would you mind sharing your ebuild for mesos-0.22.0 via a gist on Github?
>
> On 18/09/2015 12:58 PM, CCAAT wrote:
>> On 09/17/2015 06:33 PM, F21 wrote:
>>> Is there anyway to build portable binaries for mesos?
>>
>>
>> You should try out gentoo linux, everything is built from sources.
>>
>> Ebuilds guide the process. My (hack) ebuild for mesos-0.22.0 was
>> 61 lines. That's it. I will roll out a 0.24 ebuild, in a few weeks
>> or less.
>>
>> Gentoo is designed from the ground up to build form sources. We have
>> a rich 'cross-compile' environment for things like aarch64; so building
>> mesos for arm64 is mostly trivial, once the 0.24 ebuild is rolled out.
>>
>>
>> There a bit of reading, but the gentoo 'devmanual' pretty much guides
>> you through the process [1]. Gentoo also has a great package manager.
>> Here is a (very profane) rant/comparison of some common package managers
>> and their inherent weaknesses [2]. If you want to see how simple the
>> gentoo ebuild for mesos-0.22 is just ask. It fetches, unpacks, compiles
>> and installs the package, very neatly. And there is lots of help and
>> encouragement from a long list of talented devs.
>>
>> Gentoo is not for the weak minded or folks that do not wish to master
>> the deep details of linux. caveat emptor. CoreOS uses much of gentoo
>> in it's build/management, if that option helps.
>>
>>
>> hth,
>> James
>>
>>
>>
>>
>>
>> [1] https://devmanual.gentoo.org/
>>
>>
>> [2]
>> http://michael.orlitzky.com/articles/motherfuckers_need_package_management.php
>>
>
>
Re: Building portable binaries
Posted by F21 <f2...@gmail.com>.
That sounds really interesting! I am just in the process of spinning up
a gentoo vm.
Would you mind sharing your ebuild for mesos-0.22.0 via a gist on Github?
On 18/09/2015 12:58 PM, CCAAT wrote:
> On 09/17/2015 06:33 PM, F21 wrote:
>> Is there anyway to build portable binaries for mesos?
>
>
> You should try out gentoo linux, everything is built from sources.
>
> Ebuilds guide the process. My (hack) ebuild for mesos-0.22.0 was
> 61 lines. That's it. I will roll out a 0.24 ebuild, in a few weeks
> or less.
>
> Gentoo is designed from the ground up to build form sources. We have
> a rich 'cross-compile' environment for things like aarch64; so building
> mesos for arm64 is mostly trivial, once the 0.24 ebuild is rolled out.
>
>
> There a bit of reading, but the gentoo 'devmanual' pretty much guides
> you through the process [1]. Gentoo also has a great package manager.
> Here is a (very profane) rant/comparison of some common package managers
> and their inherent weaknesses [2]. If you want to see how simple the
> gentoo ebuild for mesos-0.22 is just ask. It fetches, unpacks, compiles
> and installs the package, very neatly. And there is lots of help and
> encouragement from a long list of talented devs.
>
> Gentoo is not for the weak minded or folks that do not wish to master
> the deep details of linux. caveat emptor. CoreOS uses much of gentoo
> in it's build/management, if that option helps.
>
>
> hth,
> James
>
>
>
>
>
> [1] https://devmanual.gentoo.org/
>
>
> [2]
> http://michael.orlitzky.com/articles/motherfuckers_need_package_management.php
Re: Building portable binaries
Posted by CCAAT <cc...@tampabay.rr.com>.
On 09/17/2015 06:33 PM, F21 wrote:
> Is there anyway to build portable binaries for mesos?
You should try out gentoo linux, everything is built from sources.
Ebuilds guide the process. My (hack) ebuild for mesos-0.22.0 was
61 lines. That's it. I will roll out a 0.24 ebuild, in a few weeks
or less.
Gentoo is designed from the ground up to build form sources. We have
a rich 'cross-compile' environment for things like aarch64; so building
mesos for arm64 is mostly trivial, once the 0.24 ebuild is rolled out.
There a bit of reading, but the gentoo 'devmanual' pretty much guides
you through the process [1]. Gentoo also has a great package manager.
Here is a (very profane) rant/comparison of some common package managers
and their inherent weaknesses [2]. If you want to see how simple the
gentoo ebuild for mesos-0.22 is just ask. It fetches, unpacks, compiles
and installs the package, very neatly. And there is lots of help and
encouragement from a long list of talented devs.
Gentoo is not for the weak minded or folks that do not wish to master
the deep details of linux. caveat emptor. CoreOS uses much of gentoo
in it's build/management, if that option helps.
hth,
James
[1] https://devmanual.gentoo.org/
[2]
http://michael.orlitzky.com/articles/motherfuckers_need_package_management.php
Re: Building portable binaries
Posted by James Peach <jo...@gmail.com>.
> On Sep 17, 2015, at 4:33 PM, F21 <f2...@gmail.com> wrote:
>
> Is there anyway to build portable binaries for mesos?
>
> Currently, I have tried building my own libsvn, libsasl2, libcurl, libapr and then built mesos using the following:
>
> ../configure CC=gcc-4.8 CXX=g++-4.8 LD_LIBRARY_PATH=/tmp/mesos-build/sasl2/lib SASL_PATH=/tmp/mesos-build/sasl2/lib/sasl2 --prefix=/tmp/mesos-build/mesos --with-svn=/tmp/mesos-build/svn --with-apr=/tmp/mesos-build/apr --with-sasl=/tmp/mesos-build/sasl2/ --with-curl=/tmp/mesos-build/curl
> make
> make install
>
> I then compress /tmp/mesos-build/mesos into an archive and distribute it to my machines. The only problem is that the build seems to be buggy. For example, I've been experiencing a containerization issues where the executors will crash, but not output anything useful to stderr and stdout. See https://github.com/mesosphere/hdfs/issues/194
>
> Is there a definite way to build portable binaries that I can easily copy to another machine to run?
You could do a statically linked build by doing configure --enable-static --disable-shared. I don't know whether that is supported in the Mesos build, but it is a standard automake feature, so if it fails it should be fixable.
J
Re: Building portable binaries
Posted by Alexander Gallego <ag...@concord.io>.
My build system is a HUGE PITA.
Right now it is a set of 77 ansible roles. For configuring. half of them
are for mesos.
The reason to have ansible do it exactly because of what you are seeing,
but it gets more complex for framework authors (combination of operating
systems types, versions, libraries, system installed libs, etc)
We build all the transitive dependencies from source too (as many as we can)
So, the flags are exactly what you're using and adding
--with-glog=/usr/local/lib
We just ` make ` and the ansible scripts move things to the right
directories.
Basically we found that trying to give all of the transitive dependencies a
custom installation path and custom LD_LIBRARY_PATH was hard and it kept
breaking.
So, the way we build it is:
1. Only run make for all deps (never make install - though, some libs
actually perform an install with make, just FYI)
2. Manually link or move the libraries to /usr/local/lib
3. fix your ldconfig
Packaging contents for my deployments is simpler as I deploy only on docker
images. Effectively:
```
curl -L https:...... | tar -xzf -
```
And it's not an issue because we've placed things in the default
directories.
- alex
On Thu, Sep 17, 2015 at 8:16 PM, F21 <f2...@gmail.com> wrote:
> Thanks for your pointers! I will give that a try to see if I can track
> down the problem.
>
> Do you mind sharing a bit more about your build process?
>
> What do your ./configure flags look like? Also, do you do a `make install`
> and then package up the contents?
>
> Cheers!
>
>
> On 18/09/2015 10:06 AM, Alexander Gallego wrote:
>
> I run a custom build of mesos and it works pretty reliably, so I'll try to
> share what I do in the hopes that it helps you.
>
> The symptom you are seeing of exiting before anything is output to the
> logs has been usually an issue with the LD_LIBRARY_PATH being screwed up. I
> would try to ssh into the machines and run the command manually.
>
> Native code is a bit hard to debug, but this is what I do for a cluster to
> figure out mesos issues.
>
> 1. Recompile with -ggdb and ship dwarf symbols (if you don't want to,
> you'll have to symbolize the binary later)
> 2. echo 'core.%h.%e.%t' > /proc/sys/kernel/core_pattern on all your
> machines
> 3. wrap your mesos executors (if native) as well as mesos-master and
> mesos-slave in a bash script that you control
> 4. In this bash script, set the filehandle limits from the soft to the hard
> 5. In this bash script, set the core dump limits to unlimited
>
> When it crashes, it will be pretty straight forward to debug with the core
> dump, it will almost tell you the exact assembly instruction, or exception
> in the code.
>
> I've never had crashes with mesos after compiling with
>
> ```
> -mtune=generic
> ```
>
> I had something similar to my rocksdb build (
> <https://github.com/facebook/rocksdb/issues/690>
> https://github.com/facebook/rocksdb/issues/690) happened to my mesos
> build when I was first starting and compiling with -mtune=native
>
> If you are linking the libmesos.so into another process (i.e.: you wrote
> your own native executor) then make sure you do something similar to the
> core dump limits.
>
> Note: If you are running dockerized/containerized deployments you won't be
> able to modify the /proc/sys/* , so this will have to be done on the host
> computers.
>
> I've been meaning to write a blog post about this, but this is the TL;DR.
>
> Hope this helps.
>
> - Alex
>
>
>
>
>
>
>
> On Thu, Sep 17, 2015 at 7:33 PM, F21 <f2...@gmail.com> wrote:
>
>> Is there anyway to build portable binaries for mesos?
>>
>> Currently, I have tried building my own libsvn, libsasl2, libcurl, libapr
>> and then built mesos using the following:
>>
>> ../configure CC=gcc-4.8 CXX=g++-4.8
>> LD_LIBRARY_PATH=/tmp/mesos-build/sasl2/lib
>> SASL_PATH=/tmp/mesos-build/sasl2/lib/sasl2 --prefix=/tmp/mesos-build/mesos
>> --with-svn=/tmp/mesos-build/svn --with-apr=/tmp/mesos-build/apr
>> --with-sasl=/tmp/mesos-build/sasl2/ --with-curl=/tmp/mesos-build/curl
>> make
>> make install
>>
>> I then compress /tmp/mesos-build/mesos into an archive and distribute it
>> to my machines. The only problem is that the build seems to be buggy. For
>> example, I've been experiencing a containerization issues where the
>> executors will crash, but not output anything useful to stderr and stdout.
>> See <https://github.com/mesosphere/hdfs/issues/194>
>> https://github.com/mesosphere/hdfs/issues/194
>>
>> Is there a definite way to build portable binaries that I can easily copy
>> to another machine to run?
>>
>
>
>
> --
>
>
>
>
>
> Alexander Gallego | <http://concord.io>http://concord.io
>
>
>
--
Alexander Gallego | http://concord.io
Re: Building portable binaries
Posted by F21 <f2...@gmail.com>.
Thanks for your pointers! I will give that a try to see if I can track
down the problem.
Do you mind sharing a bit more about your build process?
What do your ./configure flags look like? Also, do you do a `make
install` and then package up the contents?
Cheers!
On 18/09/2015 10:06 AM, Alexander Gallego wrote:
> I run a custom build of mesos and it works pretty reliably, so I'll
> try to share what I do in the hopes that it helps you.
>
> The symptom you are seeing of exiting before anything is output to the
> logs has been usually an issue with the LD_LIBRARY_PATH being screwed
> up. I would try to ssh into the machines and run the command manually.
>
> Native code is a bit hard to debug, but this is what I do for a
> cluster to figure out mesos issues.
>
> 1. Recompile with -ggdb and ship dwarf symbols (if you don't want to,
> you'll have to symbolize the binary later)
> 2. echo 'core.%h.%e.%t' > /proc/sys/kernel/core_pattern on all your
> machines
> 3. wrap your mesos executors (if native) as well as mesos-master and
> mesos-slave in a bash script that you control
> 4. In this bash script, set the filehandle limits from the soft to the
> hard
> 5. In this bash script, set the core dump limits to unlimited
>
> When it crashes, it will be pretty straight forward to debug with the
> core dump, it will almost tell you the exact assembly instruction, or
> exception in the code.
>
> I've never had crashes with mesos after compiling with
>
> ```
> -mtune=generic
> ```
>
> I had something similar to my rocksdb build
> (https://github.com/facebook/rocksdb/issues/690) happened to my mesos
> build when I was first starting and compiling with -mtune=native
>
> If you are linking the libmesos.so into another process (i.e.: you
> wrote your own native executor) then make sure you do something
> similar to the core dump limits.
>
> Note: If you are running dockerized/containerized deployments you
> won't be able to modify the /proc/sys/* , so this will have to be done
> on the host computers.
>
> I've been meaning to write a blog post about this, but this is the TL;DR.
>
> Hope this helps.
>
> - Alex
>
>
>
>
>
>
>
> On Thu, Sep 17, 2015 at 7:33 PM, F21 <f21.groups@gmail.com
> <ma...@gmail.com>> wrote:
>
> Is there anyway to build portable binaries for mesos?
>
> Currently, I have tried building my own libsvn, libsasl2, libcurl,
> libapr and then built mesos using the following:
>
> ../configure CC=gcc-4.8 CXX=g++-4.8
> LD_LIBRARY_PATH=/tmp/mesos-build/sasl2/lib
> SASL_PATH=/tmp/mesos-build/sasl2/lib/sasl2
> --prefix=/tmp/mesos-build/mesos --with-svn=/tmp/mesos-build/svn
> --with-apr=/tmp/mesos-build/apr
> --with-sasl=/tmp/mesos-build/sasl2/ --with-curl=/tmp/mesos-build/curl
> make
> make install
>
> I then compress /tmp/mesos-build/mesos into an archive and
> distribute it to my machines. The only problem is that the build
> seems to be buggy. For example, I've been experiencing a
> containerization issues where the executors will crash, but not
> output anything useful to stderr and stdout. See
> https://github.com/mesosphere/hdfs/issues/194
>
> Is there a definite way to build portable binaries that I can
> easily copy to another machine to run?
>
>
>
>
> --
>
>
>
>
>
> Alexander Gallego | http://concord.io
Re: Building portable binaries
Posted by Alexander Gallego <ag...@concord.io>.
I run a custom build of mesos and it works pretty reliably, so I'll try to
share what I do in the hopes that it helps you.
The symptom you are seeing of exiting before anything is output to the logs
has been usually an issue with the LD_LIBRARY_PATH being screwed up. I
would try to ssh into the machines and run the command manually.
Native code is a bit hard to debug, but this is what I do for a cluster to
figure out mesos issues.
1. Recompile with -ggdb and ship dwarf symbols (if you don't want to,
you'll have to symbolize the binary later)
2. echo 'core.%h.%e.%t' > /proc/sys/kernel/core_pattern on all your machines
3. wrap your mesos executors (if native) as well as mesos-master and
mesos-slave in a bash script that you control
4. In this bash script, set the filehandle limits from the soft to the hard
5. In this bash script, set the core dump limits to unlimited
When it crashes, it will be pretty straight forward to debug with the core
dump, it will almost tell you the exact assembly instruction, or exception
in the code.
I've never had crashes with mesos after compiling with
```
-mtune=generic
```
I had something similar to my rocksdb build (
https://github.com/facebook/rocksdb/issues/690) happened to my mesos build
when I was first starting and compiling with -mtune=native
If you are linking the libmesos.so into another process (i.e.: you wrote
your own native executor) then make sure you do something similar to the
core dump limits.
Note: If you are running dockerized/containerized deployments you won't be
able to modify the /proc/sys/* , so this will have to be done on the host
computers.
I've been meaning to write a blog post about this, but this is the TL;DR.
Hope this helps.
- Alex
On Thu, Sep 17, 2015 at 7:33 PM, F21 <f2...@gmail.com> wrote:
> Is there anyway to build portable binaries for mesos?
>
> Currently, I have tried building my own libsvn, libsasl2, libcurl, libapr
> and then built mesos using the following:
>
> ../configure CC=gcc-4.8 CXX=g++-4.8
> LD_LIBRARY_PATH=/tmp/mesos-build/sasl2/lib
> SASL_PATH=/tmp/mesos-build/sasl2/lib/sasl2 --prefix=/tmp/mesos-build/mesos
> --with-svn=/tmp/mesos-build/svn --with-apr=/tmp/mesos-build/apr
> --with-sasl=/tmp/mesos-build/sasl2/ --with-curl=/tmp/mesos-build/curl
> make
> make install
>
> I then compress /tmp/mesos-build/mesos into an archive and distribute it
> to my machines. The only problem is that the build seems to be buggy. For
> example, I've been experiencing a containerization issues where the
> executors will crash, but not output anything useful to stderr and stdout.
> See https://github.com/mesosphere/hdfs/issues/194
>
> Is there a definite way to build portable binaries that I can easily copy
> to another machine to run?
>
--
Alexander Gallego | http://concord.io