You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by max square <ma...@gmail.com> on 2015/02/20 19:51:51 UTC

Error when creating a new HA cluster on EC2

Hi all,

I am new to Mesos. I am trying to create a HA configuration using the
provided EC2 script. When I do so, I create the machines, but there is no
mesos process running on any of the machines. Is this a known limitation?
It seems to be working fine when I create a single master with many slaves
cluster. However if I do so, I can't run other frameworks that depend on
zookeeper, since the EC2 script does not install or start a ZK server.

Would it be better just to start from scratch to create a cluster? Is there
any other suggested method for AWS? (I am able to create small clusters in
my infrastructure).

Any suggestions or help is greatly appreciated.

Sergio Daniel

Re: Error when creating a new HA cluster on EC2

Posted by CCAAT <cc...@tampabay.rr.com>.
Mesos & related codes from 100% sources:

I understand your sentiment. However, I disagree, strongly. I have
been using gentoo for a long time. It has THE solution to 'dependency 
hell' built into how how ebuilds are constructed. [1]

I have mesos working and spark (mostly). These other codes that
are necessary just take a bit of effort. It's a cultural approach
rather than accumulating binaries. As (mesos) clustering matures
it's going to be increasingly difficult to debug problems, without
building from sources, imho. A little pain now, great rewards later.
Keeping projects on the open source track does not prevent anyone
(vendors included) from packaging up for quick evaluational types
of installs, or formal usage.

But, if that is all the community develops, then after a while it does
become too much of an effort. At that point, it is no longer an open 
source project, but a binary offering. That may bode well for the
short term business interests of some, but long term, it will kill
the project. ymmv.


For example, the ability to diagnose deep problems with resource 
utilization will necessitate kernel level debugging with such tools
as ftrace, trace-cmd and Kernelshark, as discussed in these REDhat
documents: [2]


I'd be most interested in associating with folks that have a desire to
build from 100% sources. Afterall, openstack, slurm and many other 
packages do this and they are quite easy to support (as there are 
ebuilds for them) in gentoo. Often other groups use gentoo ebuilds
as a guide to source compilations on other distros.  Gentoo was forked 
to create ChromeOS, which was forked again to create CoreOS. Some other 
linux distros employ this semantic of building from sources, but, gentoo
linux is the most robust, imho. This source build approach also will
tremendously facilitate porting of mesos and related codes to different 
and newer architectures, such as Arm64.


So shall we start with an ansible playbook for building mesos from 
sources? WE can add other codes from there and let folks pick and choose
to build as they like. Surely this is not even a difficult task if 
organized at the developer level with some miniscule level of 
support/encouragement by the developers of mesos. Perhaps a ticket is 
needed for comment? This sort of organization will attract more coders 
and still encourage a basis for vendors to package up binaries
for more focused needs, including targeted platform support. After all 
this is exactly what CoreOS is doing: [3,4,5]

Comments are most welcome!

James



[1] https://devmanual.gentoo.org/

[2] 
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_MRG/1.3/html/Realtime_Tuning_Guide/sect-Realtime_Tuning_Guide-Realtime_Specific_Tuning-Latency_Tracing_Using_trace_cmd.html

[3] 
https://coreos.com/docs/sdk-distributors/sdk/building-development-images/

[4] https://chromium.googlesource.com/chromiumos/overlays/portage/

[5] https://en.wikipedia.org/wiki/CoreOS





On 02/22/2015 03:38 PM, Alexander Gallego wrote:
> James,
>
> It is incredibly challenging to build everything from scratch.
>
> To build mesos from 100% sources as you propose you'll need to find
> **all** transitive dependencies first. Each in turn will have transitive
> deps.
>
> In fact, if you look at the build system the reason leveldb.patch exist
> is to attempt and reduce that complexity.
>
> ```
> +# Removed tests for TCMALLOC and SNAPPY to prevent additional dependencies.
> ```
>
> Most linux packages exist **exactly** for that reason. It is not easy to
> build native dependencies and have it work for your operating system.
> That's what apt-* is for ubuntu for exapmle.
>
>
> Now building **just mesos** from sources is quite trivial.
>
>
> Here is my entire ansible build for my local dev box
>
> ```
>
>
> - name: configure mesos
>    shell: "LIBS=-lglog LDFLAGS=-L/usr/local/lib
>      {{bolt_cache_dir}}/{{mesos_dir}}/configure
>      --with-glog=/usr/local
>      --disable-python
>      --disable-java"
>
>    args:
>      creates: '{{bolt_cache_dir}}/{{mesos_dir}}/build/Makefile'
>      chdir: '{{bolt_cache_dir}}/{{mesos_dir}}/build'
>
> - name: Make mesos
>    sudo: True
>    shell:
>      "ldconfig && make -j{{ansible_processor_vcpus}}"
>    args:
>      chdir: '{{bolt_cache_dir}}/{{mesos_dir}}/build'
>      creates: '{{bolt_cache_dir}}/{{mesos_dir}}/build/src/libmesos.la
> <http://libmesos.la>'
>
>
>
> ```
>
>
> If you ever participated in large scale native builds, you might know
> that compiler flags and linker flags have a huge effect on whether the
> software will work at runtime (i.e: symbols stripped away, extern
> functions not being exported for weird reasons, ABI incompatibility, etc).
>
> To be honest, building everything from source sounds impractical unless
> you have a team of developers optimizing it for your platform. Your
> dependencies would be an explosion of build system system deps - CMAKE,
> autoconf, Scons, at the top of my head for transitive deps.
>
> As I said, that is exactly my ansible task for building mesos locally.
> Hope it helps.
>
> -ag
>
>
>
>
>
>
> Sincerely,
> Alexander Gallego
>
>
> On Sun, Feb 22, 2015 at 3:11 PM, CCAAT <ccaat@tampabay.rr.com
> <ma...@tampabay.rr.com>> wrote:
>
>     On 02/22/2015 06:35 AM, info@roybos.nl <ma...@roybos.nl> wrote:
>
>         Last friday i put some Ansible scripts on github for provisoning
>         a multi
>         AZ cluster on AWS.
>         You could have a look at it
>         https://github.com/roybos/aws-__mesos-marathon-cluster
>         <https://github.com/roybos/aws-mesos-marathon-cluster> and maybe
>         it helps you.
>
>         It basically creates a VPC within an AWS region and setups 1
>         node in 3 AZ's.
>         All nodes are currently equal(mesos master and slave are on the same
>         machine) which is fine for smaller clusters for let's say 3 to
>         30 nodes.
>
>         Roy Bos
>
>
>     Very cool. I'm new to ansible, but I like what I've learns so far.
>     What would be cool is if some thing like this ansible example was for
>     just 3 (generic linux ) system that are installed  from sources. No
>     binaries, no distro specific packages, just pure, raw sources. That
>     way it would provide a baseline for installation on all sorts of linux
>     (embedded, vm, container, uncommon_distro, different architectures
>     like arm64 etc etc) based systems.
>
>
>     Any ansible guides for generic mesos installs from sources would be
>     of keen interests to many folks.
>
>     After that in-house from sources methodology is viable, I'm quite
>     certain companies will want to augment their local (in-house)
>     cluster resources, with resources from vendors, particularly in a
>     dynamic mode of utilization. Therefore, within the  in-house
>     resources mixed (supplemented) with vendor resources is the future
>     of cluster computing. ymmv.
>
>     Sadly, I see little progress on this open systems approach, and that
>     concerns me greatly for the viability of the mesos cluster. Is it
>     indeed going to be limited to large corporations trying to sell
>     something nobody wants; or are there going to be open source
>     methodologies developed that streamline the installation of mesos
>     inside of a medium size company with modest resources.
>
>     Ansible is the key technology here combined with building up a mesos
>     cluster FROM 100% SOURCES.
>
>
>     Put that in your presentation at mesos con(artists?)
>
>
>     James
>
>
>
>
>


Re: Error when creating a new HA cluster on EC2

Posted by Alexander Gallego <ag...@rbonut.com>.
James,

It is incredibly challenging to build everything from scratch.

To build mesos from 100% sources as you propose you'll need to find **all**
transitive dependencies first. Each in turn will have transitive deps.

In fact, if you look at the build system the reason leveldb.patch exist is
to attempt and reduce that complexity.

```
+# Removed tests for TCMALLOC and SNAPPY to prevent additional dependencies.
```

Most linux packages exist **exactly** for that reason. It is not easy to
build native dependencies and have it work for your operating system.
That's what apt-* is for ubuntu for exapmle.


Now building **just mesos** from sources is quite trivial.


Here is my entire ansible build for my local dev box

```


- name: configure mesos
  shell: "LIBS=-lglog LDFLAGS=-L/usr/local/lib
    {{bolt_cache_dir}}/{{mesos_dir}}/configure
    --with-glog=/usr/local
    --disable-python
    --disable-java"

  args:
    creates: '{{bolt_cache_dir}}/{{mesos_dir}}/build/Makefile'
    chdir: '{{bolt_cache_dir}}/{{mesos_dir}}/build'

- name: Make mesos
  sudo: True
  shell:
    "ldconfig && make -j{{ansible_processor_vcpus}}"
  args:
    chdir: '{{bolt_cache_dir}}/{{mesos_dir}}/build'
    creates: '{{bolt_cache_dir}}/{{mesos_dir}}/build/src/libmesos.la'



```


If you ever participated in large scale native builds, you might know that
compiler flags and linker flags have a huge effect on whether the software
will work at runtime (i.e: symbols stripped away, extern functions not
being exported for weird reasons, ABI incompatibility, etc).

To be honest, building everything from source sounds impractical unless you
have a team of developers optimizing it for your platform. Your
dependencies would be an explosion of build system system deps - CMAKE,
autoconf, Scons, at the top of my head for transitive deps.

As I said, that is exactly my ansible task for building mesos locally. Hope
it helps.

-ag






Sincerely,
Alexander Gallego


On Sun, Feb 22, 2015 at 3:11 PM, CCAAT <cc...@tampabay.rr.com> wrote:

> On 02/22/2015 06:35 AM, info@roybos.nl wrote:
>
>  Last friday i put some Ansible scripts on github for provisoning a multi
>> AZ cluster on AWS.
>> You could have a look at it
>> https://github.com/roybos/aws-mesos-marathon-cluster and maybe it helps
>> you.
>>
>> It basically creates a VPC within an AWS region and setups 1 node in 3
>> AZ's.
>> All nodes are currently equal(mesos master and slave are on the same
>> machine) which is fine for smaller clusters for let's say 3 to 30 nodes.
>>
>> Roy Bos
>>
>
> Very cool. I'm new to ansible, but I like what I've learns so far.
> What would be cool is if some thing like this ansible example was for
> just 3 (generic linux ) system that are installed  from sources. No
> binaries, no distro specific packages, just pure, raw sources. That
> way it would provide a baseline for installation on all sorts of linux
> (embedded, vm, container, uncommon_distro, different architectures like
> arm64 etc etc) based systems.
>
>
> Any ansible guides for generic mesos installs from sources would be
> of keen interests to many folks.
>
> After that in-house from sources methodology is viable, I'm quite certain
> companies will want to augment their local (in-house) cluster resources,
> with resources from vendors, particularly in a dynamic mode of utilization.
> Therefore, within the  in-house resources mixed (supplemented) with vendor
> resources is the future of cluster computing. ymmv.
>
> Sadly, I see little progress on this open systems approach, and that
> concerns me greatly for the viability of the mesos cluster. Is it indeed
> going to be limited to large corporations trying to sell something nobody
> wants; or are there going to be open source methodologies developed that
> streamline the installation of mesos
> inside of a medium size company with modest resources.
>
> Ansible is the key technology here combined with building up a mesos
> cluster FROM 100% SOURCES.
>
>
> Put that in your presentation at mesos con(artists?)
>
>
> James
>
>
>
>
>

Re: Error when creating a new HA cluster on EC2

Posted by CCAAT <cc...@tampabay.rr.com>.
On 02/22/2015 06:35 AM, info@roybos.nl wrote:

> Last friday i put some Ansible scripts on github for provisoning a multi
> AZ cluster on AWS.
> You could have a look at it
> https://github.com/roybos/aws-mesos-marathon-cluster and maybe it helps you.
>
> It basically creates a VPC within an AWS region and setups 1 node in 3 AZ's.
> All nodes are currently equal(mesos master and slave are on the same
> machine) which is fine for smaller clusters for let's say 3 to 30 nodes.
>
> Roy Bos

Very cool. I'm new to ansible, but I like what I've learns so far.
What would be cool is if some thing like this ansible example was for
just 3 (generic linux ) system that are installed  from sources. No
binaries, no distro specific packages, just pure, raw sources. That
way it would provide a baseline for installation on all sorts of linux
(embedded, vm, container, uncommon_distro, different architectures like 
arm64 etc etc) based systems.


Any ansible guides for generic mesos installs from sources would be
of keen interests to many folks.

After that in-house from sources methodology is viable, I'm quite 
certain companies will want to augment their local (in-house) cluster 
resources, with resources from vendors, particularly in a dynamic mode 
of utilization. Therefore, within the  in-house resources mixed 
(supplemented) with vendor resources is the future of cluster computing. 
ymmv.

Sadly, I see little progress on this open systems approach, and that 
concerns me greatly for the viability of the mesos cluster. Is it indeed 
going to be limited to large corporations trying to sell something 
nobody wants; or are there going to be open source methodologies 
developed that streamline the installation of mesos
inside of a medium size company with modest resources.

Ansible is the key technology here combined with building up a mesos
cluster FROM 100% SOURCES.


Put that in your presentation at mesos con(artists?)


James





Re: Error when creating a new HA cluster on EC2

Posted by "info@roybos.nl" <in...@roybos.nl>.
Hi Sergio,

Last friday i put some Ansible scripts on github for provisoning a multi AZ cluster on AWS.
You could have a look at it https://github.com/roybos/aws-mesos-marathon-cluster and maybe it helps you.

It basically creates a VPC within an AWS region and setups 1 node in 3 AZ's.
All nodes are currently equal(mesos master and slave are on the same machine) which is fine for smaller clusters for let's say 3 to 30 nodes.

Roy Bos


From: max square <ma...@gmail.com>
Reply: user@mesos.apache.org <us...@mesos.apache.org>>
Date: 20 Feb 2015 at 19:52:19
To: user@mesos.apache.org <us...@mesos.apache.org>>
Subject:  Error when creating a new HA cluster on EC2  

Hi all,

I am new to Mesos. I am trying to create a HA configuration using the provided EC2 script. When I do so, I create the machines, but there is no mesos process running on any of the machines. Is this a known limitation? It seems to be working fine when I create a single master with many slaves cluster. However if I do so, I can't run other frameworks that depend on zookeeper, since the EC2 script does not install or start a ZK server. 

Would it be better just to start from scratch to create a cluster? Is there any other suggested method for AWS? (I am able to create small clusters in my infrastructure).

Any suggestions or help is greatly appreciated.

Sergio Daniel

Re: Error when creating a new HA cluster on EC2

Posted by max square <ma...@gmail.com>.
Adam,

Thanks for the info. Initially I was using the OOTB script that comes with
Mesos in the ec2 folder
<https://github.com/apache/mesos/blob/master/ec2/mesos-ec2>which is
basically a Python script that lets you add an configure the cluster. I
have configured and executed successfully the cloudformation scripts.

Have you used them? For some reason I am not able to create more than 3
instances of anything when I'm running marathon. Tasks for some reason are
stuck as STAGING even when the cluster has more than enough resources.

Thanks in advance!

Sergio Daniel

On Sun, Feb 22, 2015 at 3:56 AM, Adam Bordelon <ad...@mesosphere.io> wrote:

> Check out https://github.com/mesosphere/cloudformation-mesos
> What "EC2 script" were you trying?
>
> On Fri, Feb 20, 2015 at 10:51 AM, max square <ma...@gmail.com>
> wrote:
>
>> Hi all,
>>
>> I am new to Mesos. I am trying to create a HA configuration using the
>> provided EC2 script. When I do so, I create the machines, but there is no
>> mesos process running on any of the machines. Is this a known limitation?
>> It seems to be working fine when I create a single master with many slaves
>> cluster. However if I do so, I can't run other frameworks that depend on
>> zookeeper, since the EC2 script does not install or start a ZK server.
>>
>> Would it be better just to start from scratch to create a cluster? Is
>> there any other suggested method for AWS? (I am able to create small
>> clusters in my infrastructure).
>>
>> Any suggestions or help is greatly appreciated.
>>
>> Sergio Daniel
>>
>
>

Re: Error when creating a new HA cluster on EC2

Posted by Adam Bordelon <ad...@mesosphere.io>.
Check out https://github.com/mesosphere/cloudformation-mesos
What "EC2 script" were you trying?

On Fri, Feb 20, 2015 at 10:51 AM, max square <ma...@gmail.com>
wrote:

> Hi all,
>
> I am new to Mesos. I am trying to create a HA configuration using the
> provided EC2 script. When I do so, I create the machines, but there is no
> mesos process running on any of the machines. Is this a known limitation?
> It seems to be working fine when I create a single master with many slaves
> cluster. However if I do so, I can't run other frameworks that depend on
> zookeeper, since the EC2 script does not install or start a ZK server.
>
> Would it be better just to start from scratch to create a cluster? Is
> there any other suggested method for AWS? (I am able to create small
> clusters in my infrastructure).
>
> Any suggestions or help is greatly appreciated.
>
> Sergio Daniel
>