You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@mesos.apache.org by Jim Donahue <jd...@adobe.com> on 2012/11/07 02:05:21 UTC

Setting up a Mesos cluster to run Hadoop

I'm working on the Hadoop setup for a Mesos cluster.  Here's what I take from the documentation -- please correct if I'm wrong:

1.  Each node (master and slaves) needs the following (replicated) configuration files:
	Mapred-site.xml (plugging in the URL of the Mesos master)
	Hadoop-env.sh (with JAVA_HOME, PROTOBUF_JAR, MESOS_JAR, MESOS_NATIVE_LIBRARY and HADOOP_CLASSPATH set)
	Core-site.sh
	And a "master" file and a "slaves" file (with the host names or IP addresses of the master and slaves)

2.  On the master, I should start a jobtracker when the cluster comes up.

3.  Ports 9000 and 9001 need to be open on all instances.

I'm running the same AMI on all instances, so at least I know that the various software components are all installed in the same place on all the machines in the cluster.

Am I missing anything?

Thanks,

Jim Donahue
Adobe Systems