introduction: Adam Monsen

[Adam Monsen <ha...@gmail.com>]

Hello,

I'm the ultimate newcomer in here. I'm new to both Mesos and cluster
computing. I'm interested in learning more about both and possibly
volunteering.

I'm not going to be much help coding. I've done very little C++ and I
don't have a strong classical CS/math background. Not that I couldn't
learn, it just probably isn't in the cards for me right now. But there
are many other ways I can help.

I'll start with a few questions.

* Anyone have a pointer to a decent "hello world" for Mesos and/or
cluster computing?

* Are there any folks local to Seattle using (or, better yet,
customizing) Mesos? (if so, I'd like to meet)

* What are some of the near-term and far-term goals for Mesos? (for
instance, I've seen mention of graduating from the Apache incubator)

* Looks like the authoritative source code repository is
https://svn.apache.org/repos/asf/incubator/mesos/trunk . Is anyone using
git-svn locally? (just curious)

* Is there an IRC channel or other chatroom that Mesos developers frequent?

* Is there an extant list of bite-size Mesos [volunteer] tasks?

Thanks!

--
Adam Monsen
http://adammonsen.com

Re: MESOS-347 cgroups test failure

[Adam Monsen <ha...@gmail.com>]

Yes, that repro breaks for me exactly as you describe.

On 02/12/2013 05:12 PM, Benjamin Hindman wrote:
> It looks like we've reproduced locally. You can confirm by doing your
> cleanup, then running (repeatedly) with
> --gtest_filter='CgroupsNoHierarchyTest.*'. Now try running with
> --gtest_filter='Cgroups*BalloonFramework'. Try
> with --gtest_filter='CgroupsNoHierarchyTest.*' again, and if it fails we
> have a winner!

Re: MESOS-347 cgroups test failure (was: introduction: Adam Monsen)

[Benjamin Hindman <be...@eecs.berkeley.edu>]

It looks like we've reproduced locally. You can confirm by doing your
cleanup, then running (repeatedly) with
--gtest_filter='CgroupsNoHierarchyTest.*'. Now try running with
--gtest_filter='Cgroups*BalloonFramework'. Try
with --gtest_filter='CgroupsNoHierarchyTest.*' again, and if it fails we
have a winner!


On Tue, Feb 12, 2013 at 4:41 PM, Adam Monsen <ha...@gmail.com> wrote:

> On 02/11/2013 02:11 PM, Benjamin Hindman wrote:
> > The error implies you still have a mounted hierarchy somewhere. Or we
> have
> > a bug and haven't properly cleaned up our own hierarchies causing this
> test
> > to fail.
>
> I think I did proper cleanup, so I'm guessing the latter.
>
> https://issues.apache.org/jira/browse/MESOS-347
>
> Can anyone else (on Linux) repro this bug?
>
>

MESOS-347 cgroups test failure (was: introduction: Adam Monsen)

[Adam Monsen <ha...@gmail.com>]

On 02/11/2013 02:11 PM, Benjamin Hindman wrote:
> The error implies you still have a mounted hierarchy somewhere. Or we have
> a bug and haven't properly cleaned up our own hierarchies causing this test
> to fail.

I think I did proper cleanup, so I'm guessing the latter.

https://issues.apache.org/jira/browse/MESOS-347

Can anyone else (on Linux) repro this bug?

Re: introduction: Adam Monsen

[Benjamin Hindman <be...@eecs.berkeley.edu>]

> [----------] 1 test from CgroupsNoHierarchyTest
> [ RUN      ] CgroupsNoHierarchyTest.ROOT_CGROUPS_MountUnmountHierarchy
> tests/cgroups_tests.cpp:294: Failure
> cgroups::mount(HIERARCHY, "cpu,memory"): 'cpu' is already attached to
> another hierarchy
> [  FAILED  ] CgroupsNoHierarchyTest.ROOT_CGROUPS_MountUnmountHierarchy (0
> ms)
> [----------] 1 test from CgroupsNoHierarchyTest (0 ms total)
>

The error implies you still have a mounted hierarchy somewhere. Or we have
a bug and haven't properly cleaned up our own hierarchies causing this test
to fail.

Re: introduction: Adam Monsen

[Adam Monsen <ha...@gmail.com>]

Ben H. wrote:
>Adam wrote:
>> I've got it down to 10 (consistently) failing tests, but they don't
>> seem too critical for my current experiments. Looks like all the test
>> failures (except for MESOS-335, which doesn't repro reliably) are
>> related to Linux cgroups.
>
> Is this on trunk?

Yep. I'm using a git clone of git://github.com/apache/mesos.git,
checked out to the "trunk" branch. git commit b8d38f0476833.
git-svn-id shows trunk r1444294.

> What are the error messages of the failed tests?

Update: I *had* 10 failed tests when running `sudo make check`, but
after cleaning up what appears to be some left over cgroups detritus
(using `umount /cgroup` and `sudo cgclear`), I now consistently get
just one failed test, CgroupsNoHierarchyTest (again, using `sudo make
check`):

- 8< -
[----------] 1 test from CgroupsNoHierarchyTest
[ RUN      ] CgroupsNoHierarchyTest.ROOT_CGROUPS_MountUnmountHierarchy
tests/cgroups_tests.cpp:294: Failure
cgroups::mount(HIERARCHY, "cpu,memory"): 'cpu' is already attached to
another hierarchy
[  FAILED  ] CgroupsNoHierarchyTest.ROOT_CGROUPS_MountUnmountHierarchy (0 ms)
[----------] 1 test from CgroupsNoHierarchyTest (0 ms total)
- >8 -

Interestingly, this test (CgroupsNoHierarchyTest) passes when I run it
stand-alone (using the bin/mesos-tests syntax you provided below).

> ./bin/mesos-tests --gtest_filter=CgroupsNoHierarchyTest.*
>
> Likewise, you can run all tests but the CgroupsNoHierarchyTests with:
>
> ./bin/mesos-tests --gtest_filter=-CgroupsNoHierarchyTest.*

Perfect, thank you.

Re: introduction: Adam Monsen

[Benjamin Hindman <be...@eecs.berkeley.edu>]

> I've got it down to 10 (consistently) failing tests, but they don't
> seem too critical for my current experiments. Looks like all the test
> failures (except for MESOS-335, which doesn't repro reliably) are
> related to Linux cgroups.
>

Is this on trunk? What are the error messages of the failed tests?


> Semi-related question: how do I run a single unit test? Say, only
> CgroupsNoHierarchyTest?
>

./bin/mesos-tests --gtest_filter=CgroupsNoHierarchyTest.*

Likewise, you can run all tests but the CgroupsNoHierarchyTests with:

./bin/mesos-tests --gtest_filter=-CgroupsNoHierarchyTest.*

Re: introduction: Adam Monsen

[Adam Monsen <ha...@gmail.com>]

I've got it down to 10 (consistently) failing tests, but they don't
seem too critical for my current experiments. Looks like all the test
failures (except for MESOS-335, which doesn't repro reliably) are
related to Linux cgroups.

https://github.com/mesos/mesos/wiki/Using-Linux-Containers was
helpful! It was bit out of date, so I updated it. With regard to the
cgroups unit tests, it said "These are disabled by default", but `make
check` runs cgroups tests for me. It also mentioned a script/program
called "alltests" that no longer exists.

Semi-related question: how do I run a single unit test? Say, only
CgroupsNoHierarchyTest?

I wrote:
> I've done little C/C++ hacking on Linux.

Really I should have said I've done little C/C++ hacking, period. But
I get the basics. :)

Re: introduction: Adam Monsen

[Adam Monsen <ha...@gmail.com>]

On Fri, Feb 8, 2013 at 2:08 PM, Benjamin Mahler <bm...@twitter.com> wrote:
> For a dev environment, I'd recommend:
>
> Ubuntu 10.04 (64 bit), or
> Ubuntu 12.04 (64 bit)
>
> with our current release, we don't support gcc 4.7
> We should be compiling on gcc-4.2.

I went with 64-but Ubuntu 12.04 desktop.

I didn't find a package for gcc-4.2. Looks like the oldest available
in the default Ubuntu 12.04 repositories is gcc-4.4. I recall it being
a pain to manually build an old gcc (or maybe the pain is building the
whole GNU autotools toolchain... anyway) so I just tried the one it
came with (gcc-4.6). `make` worked fine, but `make check` didn't.

The first time I ran `make check` I ran into MESOS-335, but I've had
different results from subsequent tries... sometimes 19 tests fail,
sometimes 18.

I'm usually seeing something exactly like what is documented in
http://mail-archives.apache.org/mod_mbox/incubator-mesos-dev/201211.mbox/%3C1110106306.2231.1353966687280.JavaMail.hudson@aegis%3E
, for example: https://gist.github.com/meonkeys/4748152

I can give gcc-4.2 a shot too.

> Feel free to communicate your experiments, including any things you found
> difficult, lacking in documentation, or confusing. We love getting this kind
> of visibility into how people are using mesos.

Will do. Fair warning, I've done little C/C++ hacking on Linux.

Re: introduction: Adam Monsen

[Benjamin Mahler <bm...@twitter.com>]

For a dev environment, I'd recommend:

Ubuntu 10.04 (64 bit), or
Ubuntu 12.04 (64 bit)

Unfortunately, with our current release, we don't support gcc 4.7. I
realize this is annoying, and this is something that will be fixed as part
of one of the next releases. So, using 12.04 will mean installing an older
version of gcc (for now). We should be compiling on gcc-4.2.

Feel free to communicate your experiments, including any things you found
difficult, lacking in documentation, or confusing. We love getting this
kind of visibility into how people are using mesos.

P.S. So you're also a careful curator of your email! Once we're in a direct
conversation, feel free to leave me in to/cc headers, that helps me keep
track of it :)


On Fri, Feb 8, 2013 at 1:34 PM, Adam Monsen <ha...@gmail.com> wrote:

> What would you guys recommend for a dev environment? I'll set up a
> 64-bit OS for my Mesos experiments. I'm most familiar with Ubuntu
> Desktop lately, so I'd prefer to stick with that. I'm thinking Ubuntu
> 12.10 64-bit desktop, and something other than gcc-4.7. Do I need a
> particular version of gcc/g++?
>
> I finished a simple Hadoop hello world, so look out, uh, world! :)
>
> Oh and please let me know if you guys would rather *not* be explicitly
> listed in To/Cc headers. I've been doing that ever since corresponding a
> bit on the git mailing list. I prefer it (since I rarely read all emails
> to mailing lists unless I'm in a To/Cc), but I understand if you don't.
>
>

Re: introduction: Adam Monsen

[Adam Monsen <ha...@gmail.com>]

What would you guys recommend for a dev environment? I'll set up a
64-bit OS for my Mesos experiments. I'm most familiar with Ubuntu
Desktop lately, so I'd prefer to stick with that. I'm thinking Ubuntu
12.10 64-bit desktop, and something other than gcc-4.7. Do I need a
particular version of gcc/g++?

I finished a simple Hadoop hello world, so look out, uh, world! :)

Oh and please let me know if you guys would rather *not* be explicitly
listed in To/Cc headers. I've been doing that ever since corresponding a
bit on the git mailing list. I prefer it (since I rarely read all emails
to mailing lists unless I'm in a To/Cc), but I understand if you don't.

Re: introduction: Adam Monsen

[Adam Monsen <ha...@gmail.com>]

Ben, Vinod, thanks again for your help!

I'm busy for the next couple days, but I'll take a look at this again
soon. More replies inline, below.

On 01/30/2013 02:39 PM, Benjamin Mahler wrote:
> Hi Adam, the build issue on trunk that your seeing is because we don't
> yet support building with GCC 4.7.
> 
> I've been keeping track of the particular boost issue here:
> https://issues.apache.org/jira/browse/MESOS-271
> https://svn.boost.org/trac/boost/ticket/6854
> 
> If your looking to use git, the svn repo is mirrored here:
> git://git.apache.org/mesos.git <http://git.apache.org/mesos.git>
> 
> Note that you should track "trunk" instead of "master".

Great info, thank you!

> P.S. If you're interested in Hadoop on mesos, you may also be interested
> in Storm or Spark on Mesos, which are maintained externally:
> 
> https://github.com/nathanmarz/storm-mesos
> https://github.com/mesos/spark

Wow, both of these projects look very compelling! I just started
watching Nathan Marz's talk on Storm. I think I might try that out
before Hadoop.

faster git mirror (was: introduction: Adam Monsen)

[Adam Monsen <ha...@gmail.com>]

On 01/30/2013 02:39 PM, Benjamin Mahler wrote:
> If your looking to use git, the svn repo is mirrored here:
> git://git.apache.org/mesos.git

I found git://github.com/apache/mesos.git to be much faster.


To change mirrors, I used:

git remote set-url origin git://github.com/apache/mesos.git

Re: introduction: Adam Monsen

[Benjamin Mahler <bm...@twitter.com>]

Hi Adam, the build issue on trunk that your seeing is because we don't yet
support building with GCC 4.7.

I've been keeping track of the particular boost issue here:
https://issues.apache.org/jira/browse/MESOS-271
https://svn.boost.org/trac/boost/ticket/6854

If your looking to use git, the svn repo is mirrored here:
git://git.apache.org/mesos.git

Note that you should track "trunk" instead of "master".

P.S. If you're interested in Hadoop on mesos, you may also be interested in
Storm or Spark on Mesos, which are maintained externally:

https://github.com/nathanmarz/storm-mesos
https://github.com/mesos/spark

On Wed, Jan 30, 2013 at 1:45 PM, Vinod Kone <vi...@twitter.com> wrote:

> Hey Adam,
>
> We do most of the Mesos development on 64-bit machines, and we rarely, if
> ever, test on 32-bit machines. So, we recommend building Mesos on 64-bit
> machines.
>
> The specific issue you have encountered with 0.9.0 is well known and the
> fix has been applied in 0.10.0 release. You can
> grab the distribution from: http://people.apache.org/~benh/mesos-0.10.0
> -incubating-RC2/mesos-0.10.0-incubating.tar.gz<http://people.apache.org/~benh/mesos-0.10.0-incubating-RC2/mesos-0.10.0-incubating.tar.gz>
>
> Having said that, you might still encounter issues when you are building on
> 32-bit. (https://issues.apache.org/jira/browse/MESOS-267).
>
> Vinod
>
>
> @vinodkone
>
>
> On Wed, Jan 30, 2013 at 10:56 AM, Adam Monsen <ha...@gmail.com> wrote:
>
> > Hi Vinod,
> >
> > Thanks for your reply.
> >
> > > To run the C++ framework for example, you would do
> >
> > I ran into compile errors with some 3rd party libraries (I think).
> >
> > Here's what happened trying to compile trunk r1439229:
> > https://gist.github.com/4675618
> >
> > Here's what happened trying to compile Mesos 0.9.0:
> > https://gist.github.com/4675694
> >
> > I'm doing this on an Ubuntu 12.10 32-bit Desktop.
> >
> > I'll check out those EC2 scripts, too. That seems like an easier way to
> > just try out Mesos/Hadoop.
> >
> > > You might want to look at issues tagged as "Minor"/"Trivial", if you
> are
> > > looking for some place to start!
> >
> > I'll check those out once I get oriented, thanks.
> >
> > I just started a MapReduce tutorial, we'll see how it goes. :)
> >
> > -Adam
> >
> >
>

Re: introduction: Adam Monsen

[Vinod Kone <vi...@twitter.com>]

Hey Adam,

We do most of the Mesos development on 64-bit machines, and we rarely, if
ever, test on 32-bit machines. So, we recommend building Mesos on 64-bit
machines.

The specific issue you have encountered with 0.9.0 is well known and the
fix has been applied in 0.10.0 release. You can
grab the distribution from: http://people.apache.org/~benh/mesos-0.10.0
-incubating-RC2/mesos-0.10.0-incubating.tar.gz

Having said that, you might still encounter issues when you are building on
32-bit. (https://issues.apache.org/jira/browse/MESOS-267).

Vinod


@vinodkone


On Wed, Jan 30, 2013 at 10:56 AM, Adam Monsen <ha...@gmail.com> wrote:

> Hi Vinod,
>
> Thanks for your reply.
>
> > To run the C++ framework for example, you would do
>
> I ran into compile errors with some 3rd party libraries (I think).
>
> Here's what happened trying to compile trunk r1439229:
> https://gist.github.com/4675618
>
> Here's what happened trying to compile Mesos 0.9.0:
> https://gist.github.com/4675694
>
> I'm doing this on an Ubuntu 12.10 32-bit Desktop.
>
> I'll check out those EC2 scripts, too. That seems like an easier way to
> just try out Mesos/Hadoop.
>
> > You might want to look at issues tagged as "Minor"/"Trivial", if you are
> > looking for some place to start!
>
> I'll check those out once I get oriented, thanks.
>
> I just started a MapReduce tutorial, we'll see how it goes. :)
>
> -Adam
>
>

Re: introduction: Adam Monsen

[Adam Monsen <ha...@gmail.com>]

Hi Vinod,

Thanks for your reply.

> To run the C++ framework for example, you would do

I ran into compile errors with some 3rd party libraries (I think).

Here's what happened trying to compile trunk r1439229:
https://gist.github.com/4675618

Here's what happened trying to compile Mesos 0.9.0:
https://gist.github.com/4675694

I'm doing this on an Ubuntu 12.10 32-bit Desktop.

I'll check out those EC2 scripts, too. That seems like an easier way to
just try out Mesos/Hadoop.

> You might want to look at issues tagged as "Minor"/"Trivial", if you are
> looking for some place to start!

I'll check those out once I get oriented, thanks.

I just started a MapReduce tutorial, we'll see how it goes. :)

-Adam

Re: introduction: Adam Monsen

[Vinod Kone <vi...@twitter.com>]

Hi Adam,

Glad to hear that you are interested in Mesos. Please see my answers to
some of your questions inline.

* Anyone have a pointer to a decent "hello world" for Mesos and/or
> cluster computing?
>
>
We have several examples under "src/examples" that implement very simple
frameworks (C++, Java, Python) that run on top of Mesos. To run the C++
framework for example, you would do

$ cd mesos
$ mkdir -p build && cd build # optional
$ ../configure && make
$ MESOS_SOURCE_DIR=../`pwd` MESOS_BUILD_DIR=`pwd`
../src/tests/test_framework_test.sh


* What are some of the near-term and far-term goals for Mesos? (for
> instance, I've seen mention of graduating from the Apache incubator)
>
>
We want to be the de-facto cluster management framework in the industry :)


> * Is there an IRC channel or other chatroom that Mesos developers frequent?
>
>
AFAIK, we don't maintain any IRC channel. This mailing list is the primary
way users/developers interact and
ask questions.



> * Is there an extant list of bite-size Mesos [volunteer] tasks?
>
> The authoritative list of issues are tracked at:
https://issues.apache.org/jira/browse/MESOS#selectedTab=com.atlassian.jira.plugin.system.project%3Aissues-panel

You might want to look at issues tagged as "Minor"/"Trivial", if you are
looking for some place to start!

HTH,
Vinod

Thanks!
>
> --
> Adam Monsen
> http://adammonsen.com
>
>