You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mesos.apache.org by Charles Reiss <wo...@gmail.com> on 2013/06/17 22:25:15 UTC
Review Request: Get EC2 scripts into workable state
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11914/
-----------------------------------------------------------
Review request for mesos.
Description
-------
This is a rather big change that updates the EC2 scripts to deal with changes in Mesos over the past 18 months (or so) and to be usable under the assumption that no one will be maintaining an "official" Mesos AMI.
The big changes are:
- users are expected to supply their own AMI, there is no default;
- the webui is firewalled away as exposing the libprocess port is dangerous (AFAIK), instead users are instructed to use the SSH-forwarding SOCKS proxy;
- instructions and a script for building an AMI (which should work on many Debian- or Fedora-like Linux distros). The resulting AMI should be similar to the one that was originally supplied, including installations of Hadoop (setup to work with Mesos) and Spark;
- use the deploy scripts to launch mesos;
- drop support for downloading Mesos from git and rebuilding and redeploying it;
- drop support for Torque/Hypertable/haproxy in the AMI;
This addresses bug MESOS-500.
https://issues.apache.org/jira/browse/MESOS-500
Diffs
-----
ec2/Makefile.am 8c64f485888df1599697eb181fc76aa83206da07
ec2/deploy.amazon64/root/ephemeral-hdfs/conf/core-site.xml
ec2/deploy.amazon64/root/ephemeral-hdfs/conf/hadoop-env.sh 4e1e6991591e09f8860ab130948b0e787fce2b42
ec2/deploy.amazon64/root/ephemeral-hdfs/conf/hdfs-site.xml
ec2/deploy.amazon64/root/ephemeral-hdfs/conf/mapred-site.xml
ec2/deploy.amazon64/root/ephemeral-hdfs/conf/masters
ec2/deploy.amazon64/root/ephemeral-hdfs/conf/slaves
ec2/deploy.amazon64/root/mesos-ec2/cluster-url
ec2/deploy.amazon64/root/mesos-ec2/copy-dir
ec2/deploy.amazon64/root/mesos-ec2/create-swap
ec2/deploy.amazon64/root/mesos-ec2/hadoop-framework-conf/core-site.xml
ec2/deploy.amazon64/root/mesos-ec2/hadoop-framework-conf/hadoop-env.sh d8483140546fe00d6d17494c14ef4b09ae368496
ec2/deploy.amazon64/root/mesos-ec2/hadoop-framework-conf/mapred-site.xml 0ffa92f115d66adc9e370f4030850216545ade38
ec2/deploy.amazon64/root/mesos-ec2/haproxy+apache/haproxy.config.template 957c3f6a6b2a2f658e076337d88e69447b2a3341
ec2/deploy.amazon64/root/mesos-ec2/hypertable/Capfile 8b50912745977cb71232ba1dfa77f8bb0d60191e
ec2/deploy.amazon64/root/mesos-ec2/hypertable/hypertable.cfg b4d5b7475fab3d4a842e4b0f459abc5ca316996a
ec2/deploy.amazon64/root/mesos-ec2/masters
ec2/deploy.amazon64/root/mesos-ec2/mesos-daemon bed27657f8718eecafb83cca5d29e0612a87f129
ec2/deploy.amazon64/root/mesos-ec2/redeploy-mesos 941d783d82f0708a6da0f4677c3364537dfded63
ec2/deploy.amazon64/root/mesos-ec2/setup b6b736091d4d5be431c8da29cdb98360a1df2d29
ec2/deploy.amazon64/root/mesos-ec2/setup-slave 436f417bc5a746ad74cc88c27e630a91d55b0b23
ec2/deploy.amazon64/root/mesos-ec2/setup-torque 2ac8fd3546063d3ba391147383de53b7824c7c8c
ec2/deploy.amazon64/root/mesos-ec2/slaves
ec2/deploy.amazon64/root/mesos-ec2/ssh-no-keychecking
ec2/deploy.amazon64/root/mesos-ec2/start-hypertable af16c2d7bd615cb0c98f6ba65ff5c69859678850
ec2/deploy.amazon64/root/mesos-ec2/start-mesos 0f551db396fa7ffebef880dca0232c22808ff7cc
ec2/deploy.amazon64/root/mesos-ec2/stop-hypertable 7280dc11bfc53ae84b7ecaba34c84810461ed7f4
ec2/deploy.amazon64/root/mesos-ec2/stop-mesos 9fdb8753dffc5115f94582753e0860538be6232b
ec2/deploy.amazon64/root/mesos-ec2/zoo
ec2/deploy.amazon64/root/persistent-hdfs/conf/core-site.xml
ec2/deploy.amazon64/root/persistent-hdfs/conf/hadoop-env.sh b38ba01817e3b9c9715a476ecb692bac68983f50
ec2/deploy.amazon64/root/persistent-hdfs/conf/hdfs-site.xml
ec2/deploy.amazon64/root/persistent-hdfs/conf/mapred-site.xml
ec2/deploy.amazon64/root/persistent-hdfs/conf/masters
ec2/deploy.amazon64/root/persistent-hdfs/conf/slaves
ec2/deploy.amazon64/root/spark/conf/spark-env.sh 6b331ff2257d1b88c8ebe2e8095887a3e2cf0d54
ec2/deploy.generic/root/mesos-ec2/ec2-variables.sh 1f76e61b2feffa4eadc53745caa5033dde182531
ec2/deploy.generic/root/mesos-ec2/mesos-master-env.sh PRE-CREATION
ec2/deploy.generic/root/mesos-ec2/mesos-slave-env.sh PRE-CREATION
ec2/deploy.generic/root/mesos-ec2/redeploy-mesos PRE-CREATION
ec2/deploy.generic/root/spark/conf/spark-env.sh PRE-CREATION
ec2/mesos-ec2 3bc5d6307b2759cec0cea430b44baa2809a6f2d2
ec2/mesos_ec2.py 94fd7b2181d138c97da609fb0ed2ebdc6206cfd6
Diff: https://reviews.apache.org/r/11914/diff/
Testing
-------
Not enough yet; I've gotten this to a point where it starts a Mesos cluster, but I haven't made sure that the Hadoop and Spark installation work.
Thanks,
Charles Reiss
Re: Review Request: Get EC2 scripts into workable state
Posted by Charles Reiss <wo...@gmail.com>.
> On June 20, 2013, 9:48 p.m., Ben Mahler wrote:
> > I'm happy to give a ship it on this so you can make progress, since it seems no one "owns" the EC2 scripts?
That's my impression. This is definitely not at the "ship it" point yet... I didn't include a couple files in the code review (most notably, the script to create a new AMI was missing), and I want to make sure that the HDFS instance on the image actually works with the Hadoop+Mesos and Spark instances on the image.
- Charles
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11914/#review22192
-----------------------------------------------------------
On June 17, 2013, 8:25 p.m., Charles Reiss wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/11914/
> -----------------------------------------------------------
>
> (Updated June 17, 2013, 8:25 p.m.)
>
>
> Review request for mesos.
>
>
> Description
> -------
>
> This is a rather big change that updates the EC2 scripts to deal with changes in Mesos over the past 18 months (or so) and to be usable under the assumption that no one will be maintaining an "official" Mesos AMI.
>
> The big changes are:
> - users are expected to supply their own AMI, there is no default;
> - the webui is firewalled away as exposing the libprocess port is dangerous (AFAIK), instead users are instructed to use the SSH-forwarding SOCKS proxy;
> - instructions and a script for building an AMI (which should work on many Debian- or Fedora-like Linux distros). The resulting AMI should be similar to the one that was originally supplied, including installations of Hadoop (setup to work with Mesos) and Spark;
> - use the deploy scripts to launch mesos;
> - drop support for downloading Mesos from git and rebuilding and redeploying it;
> - drop support for Torque/Hypertable/haproxy in the AMI;
>
>
> This addresses bug MESOS-500.
> https://issues.apache.org/jira/browse/MESOS-500
>
>
> Diffs
> -----
>
> ec2/Makefile.am 8c64f485888df1599697eb181fc76aa83206da07
> ec2/deploy.amazon64/root/ephemeral-hdfs/conf/core-site.xml
> ec2/deploy.amazon64/root/ephemeral-hdfs/conf/hadoop-env.sh 4e1e6991591e09f8860ab130948b0e787fce2b42
> ec2/deploy.amazon64/root/ephemeral-hdfs/conf/hdfs-site.xml
> ec2/deploy.amazon64/root/ephemeral-hdfs/conf/mapred-site.xml
> ec2/deploy.amazon64/root/ephemeral-hdfs/conf/masters
> ec2/deploy.amazon64/root/ephemeral-hdfs/conf/slaves
> ec2/deploy.amazon64/root/mesos-ec2/cluster-url
> ec2/deploy.amazon64/root/mesos-ec2/copy-dir
> ec2/deploy.amazon64/root/mesos-ec2/create-swap
> ec2/deploy.amazon64/root/mesos-ec2/hadoop-framework-conf/core-site.xml
> ec2/deploy.amazon64/root/mesos-ec2/hadoop-framework-conf/hadoop-env.sh d8483140546fe00d6d17494c14ef4b09ae368496
> ec2/deploy.amazon64/root/mesos-ec2/hadoop-framework-conf/mapred-site.xml 0ffa92f115d66adc9e370f4030850216545ade38
> ec2/deploy.amazon64/root/mesos-ec2/haproxy+apache/haproxy.config.template 957c3f6a6b2a2f658e076337d88e69447b2a3341
> ec2/deploy.amazon64/root/mesos-ec2/hypertable/Capfile 8b50912745977cb71232ba1dfa77f8bb0d60191e
> ec2/deploy.amazon64/root/mesos-ec2/hypertable/hypertable.cfg b4d5b7475fab3d4a842e4b0f459abc5ca316996a
> ec2/deploy.amazon64/root/mesos-ec2/masters
> ec2/deploy.amazon64/root/mesos-ec2/mesos-daemon bed27657f8718eecafb83cca5d29e0612a87f129
> ec2/deploy.amazon64/root/mesos-ec2/redeploy-mesos 941d783d82f0708a6da0f4677c3364537dfded63
> ec2/deploy.amazon64/root/mesos-ec2/setup b6b736091d4d5be431c8da29cdb98360a1df2d29
> ec2/deploy.amazon64/root/mesos-ec2/setup-slave 436f417bc5a746ad74cc88c27e630a91d55b0b23
> ec2/deploy.amazon64/root/mesos-ec2/setup-torque 2ac8fd3546063d3ba391147383de53b7824c7c8c
> ec2/deploy.amazon64/root/mesos-ec2/slaves
> ec2/deploy.amazon64/root/mesos-ec2/ssh-no-keychecking
> ec2/deploy.amazon64/root/mesos-ec2/start-hypertable af16c2d7bd615cb0c98f6ba65ff5c69859678850
> ec2/deploy.amazon64/root/mesos-ec2/start-mesos 0f551db396fa7ffebef880dca0232c22808ff7cc
> ec2/deploy.amazon64/root/mesos-ec2/stop-hypertable 7280dc11bfc53ae84b7ecaba34c84810461ed7f4
> ec2/deploy.amazon64/root/mesos-ec2/stop-mesos 9fdb8753dffc5115f94582753e0860538be6232b
> ec2/deploy.amazon64/root/mesos-ec2/zoo
> ec2/deploy.amazon64/root/persistent-hdfs/conf/core-site.xml
> ec2/deploy.amazon64/root/persistent-hdfs/conf/hadoop-env.sh b38ba01817e3b9c9715a476ecb692bac68983f50
> ec2/deploy.amazon64/root/persistent-hdfs/conf/hdfs-site.xml
> ec2/deploy.amazon64/root/persistent-hdfs/conf/mapred-site.xml
> ec2/deploy.amazon64/root/persistent-hdfs/conf/masters
> ec2/deploy.amazon64/root/persistent-hdfs/conf/slaves
> ec2/deploy.amazon64/root/spark/conf/spark-env.sh 6b331ff2257d1b88c8ebe2e8095887a3e2cf0d54
> ec2/deploy.generic/root/mesos-ec2/ec2-variables.sh 1f76e61b2feffa4eadc53745caa5033dde182531
> ec2/deploy.generic/root/mesos-ec2/mesos-master-env.sh PRE-CREATION
> ec2/deploy.generic/root/mesos-ec2/mesos-slave-env.sh PRE-CREATION
> ec2/deploy.generic/root/mesos-ec2/redeploy-mesos PRE-CREATION
> ec2/deploy.generic/root/spark/conf/spark-env.sh PRE-CREATION
> ec2/mesos-ec2 3bc5d6307b2759cec0cea430b44baa2809a6f2d2
> ec2/mesos_ec2.py 94fd7b2181d138c97da609fb0ed2ebdc6206cfd6
>
> Diff: https://reviews.apache.org/r/11914/diff/
>
>
> Testing
> -------
>
> Not enough yet; I've gotten this to a point where it starts a Mesos cluster, but I haven't made sure that the Hadoop and Spark installation work.
>
>
> Thanks,
>
> Charles Reiss
>
>
Re: Review Request: Get EC2 scripts into workable state
Posted by Ben Mahler <be...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11914/#review22192
-----------------------------------------------------------
I'm happy to give a ship it on this so you can make progress, since it seems no one "owns" the EC2 scripts?
- Ben Mahler
On June 17, 2013, 8:25 p.m., Charles Reiss wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/11914/
> -----------------------------------------------------------
>
> (Updated June 17, 2013, 8:25 p.m.)
>
>
> Review request for mesos.
>
>
> Description
> -------
>
> This is a rather big change that updates the EC2 scripts to deal with changes in Mesos over the past 18 months (or so) and to be usable under the assumption that no one will be maintaining an "official" Mesos AMI.
>
> The big changes are:
> - users are expected to supply their own AMI, there is no default;
> - the webui is firewalled away as exposing the libprocess port is dangerous (AFAIK), instead users are instructed to use the SSH-forwarding SOCKS proxy;
> - instructions and a script for building an AMI (which should work on many Debian- or Fedora-like Linux distros). The resulting AMI should be similar to the one that was originally supplied, including installations of Hadoop (setup to work with Mesos) and Spark;
> - use the deploy scripts to launch mesos;
> - drop support for downloading Mesos from git and rebuilding and redeploying it;
> - drop support for Torque/Hypertable/haproxy in the AMI;
>
>
> This addresses bug MESOS-500.
> https://issues.apache.org/jira/browse/MESOS-500
>
>
> Diffs
> -----
>
> ec2/Makefile.am 8c64f485888df1599697eb181fc76aa83206da07
> ec2/deploy.amazon64/root/ephemeral-hdfs/conf/core-site.xml
> ec2/deploy.amazon64/root/ephemeral-hdfs/conf/hadoop-env.sh 4e1e6991591e09f8860ab130948b0e787fce2b42
> ec2/deploy.amazon64/root/ephemeral-hdfs/conf/hdfs-site.xml
> ec2/deploy.amazon64/root/ephemeral-hdfs/conf/mapred-site.xml
> ec2/deploy.amazon64/root/ephemeral-hdfs/conf/masters
> ec2/deploy.amazon64/root/ephemeral-hdfs/conf/slaves
> ec2/deploy.amazon64/root/mesos-ec2/cluster-url
> ec2/deploy.amazon64/root/mesos-ec2/copy-dir
> ec2/deploy.amazon64/root/mesos-ec2/create-swap
> ec2/deploy.amazon64/root/mesos-ec2/hadoop-framework-conf/core-site.xml
> ec2/deploy.amazon64/root/mesos-ec2/hadoop-framework-conf/hadoop-env.sh d8483140546fe00d6d17494c14ef4b09ae368496
> ec2/deploy.amazon64/root/mesos-ec2/hadoop-framework-conf/mapred-site.xml 0ffa92f115d66adc9e370f4030850216545ade38
> ec2/deploy.amazon64/root/mesos-ec2/haproxy+apache/haproxy.config.template 957c3f6a6b2a2f658e076337d88e69447b2a3341
> ec2/deploy.amazon64/root/mesos-ec2/hypertable/Capfile 8b50912745977cb71232ba1dfa77f8bb0d60191e
> ec2/deploy.amazon64/root/mesos-ec2/hypertable/hypertable.cfg b4d5b7475fab3d4a842e4b0f459abc5ca316996a
> ec2/deploy.amazon64/root/mesos-ec2/masters
> ec2/deploy.amazon64/root/mesos-ec2/mesos-daemon bed27657f8718eecafb83cca5d29e0612a87f129
> ec2/deploy.amazon64/root/mesos-ec2/redeploy-mesos 941d783d82f0708a6da0f4677c3364537dfded63
> ec2/deploy.amazon64/root/mesos-ec2/setup b6b736091d4d5be431c8da29cdb98360a1df2d29
> ec2/deploy.amazon64/root/mesos-ec2/setup-slave 436f417bc5a746ad74cc88c27e630a91d55b0b23
> ec2/deploy.amazon64/root/mesos-ec2/setup-torque 2ac8fd3546063d3ba391147383de53b7824c7c8c
> ec2/deploy.amazon64/root/mesos-ec2/slaves
> ec2/deploy.amazon64/root/mesos-ec2/ssh-no-keychecking
> ec2/deploy.amazon64/root/mesos-ec2/start-hypertable af16c2d7bd615cb0c98f6ba65ff5c69859678850
> ec2/deploy.amazon64/root/mesos-ec2/start-mesos 0f551db396fa7ffebef880dca0232c22808ff7cc
> ec2/deploy.amazon64/root/mesos-ec2/stop-hypertable 7280dc11bfc53ae84b7ecaba34c84810461ed7f4
> ec2/deploy.amazon64/root/mesos-ec2/stop-mesos 9fdb8753dffc5115f94582753e0860538be6232b
> ec2/deploy.amazon64/root/mesos-ec2/zoo
> ec2/deploy.amazon64/root/persistent-hdfs/conf/core-site.xml
> ec2/deploy.amazon64/root/persistent-hdfs/conf/hadoop-env.sh b38ba01817e3b9c9715a476ecb692bac68983f50
> ec2/deploy.amazon64/root/persistent-hdfs/conf/hdfs-site.xml
> ec2/deploy.amazon64/root/persistent-hdfs/conf/mapred-site.xml
> ec2/deploy.amazon64/root/persistent-hdfs/conf/masters
> ec2/deploy.amazon64/root/persistent-hdfs/conf/slaves
> ec2/deploy.amazon64/root/spark/conf/spark-env.sh 6b331ff2257d1b88c8ebe2e8095887a3e2cf0d54
> ec2/deploy.generic/root/mesos-ec2/ec2-variables.sh 1f76e61b2feffa4eadc53745caa5033dde182531
> ec2/deploy.generic/root/mesos-ec2/mesos-master-env.sh PRE-CREATION
> ec2/deploy.generic/root/mesos-ec2/mesos-slave-env.sh PRE-CREATION
> ec2/deploy.generic/root/mesos-ec2/redeploy-mesos PRE-CREATION
> ec2/deploy.generic/root/spark/conf/spark-env.sh PRE-CREATION
> ec2/mesos-ec2 3bc5d6307b2759cec0cea430b44baa2809a6f2d2
> ec2/mesos_ec2.py 94fd7b2181d138c97da609fb0ed2ebdc6206cfd6
>
> Diff: https://reviews.apache.org/r/11914/diff/
>
>
> Testing
> -------
>
> Not enough yet; I've gotten this to a point where it starts a Mesos cluster, but I haven't made sure that the Hadoop and Spark installation work.
>
>
> Thanks,
>
> Charles Reiss
>
>
Re: Review Request 11914: Get EC2 scripts into workable state
Posted by Niklas Nielsen <ni...@qni.dk>.
> On June 4, 2015, 11:01 a.m., Niklas Nielsen wrote:
> > Stale review: Charles, still want this in?
Closing for now.
- Niklas
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11914/#review86673
-----------------------------------------------------------
On July 14, 2013, 3:44 p.m., Charles Reiss wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/11914/
> -----------------------------------------------------------
>
> (Updated July 14, 2013, 3:44 p.m.)
>
>
> Review request for mesos and Andy Konwinski.
>
>
> Bugs: MESOS-500
> https://issues.apache.org/jira/browse/MESOS-500
>
>
> Repository: mesos-incubating
>
>
> Description
> -------
>
> This is a rather big change that updates the EC2 scripts to deal with changes in Mesos over the past 18 months (or so) and to be usable under the assumption that no one will be maintaining an "official" Mesos AMI.
>
> The big changes are:
> - users are expected to supply their own AMI, there is no default;
> - the webui is firewalled away as exposing the libprocess port is dangerous (AFAIK), instead users are instructed to use the SSH-forwarding SOCKS proxy;
> - instructions and a script for building an AMI (which should work on many Debian- or Fedora-like Linux distros). The resulting AMI should be similar to the one that was originally supplied, including installations of Hadoop (setup to work with Mesos) and Spark;
> - use the deploy scripts to launch mesos;
> - drop support for downloading Mesos from git and rebuilding and redeploying it;
> - drop support for Torque/Hypertable/haproxy in the AMI;
>
>
> Diffs
> -----
>
> docs/EC2-Scripts.md PRE-CREATION
> docs/EC2-Scripts.textile 7a1d1da834b4b85404793a27ca748905200835ca
> ec2/Makefile.am 8c64f485888df1599697eb181fc76aa83206da07
> ec2/deploy.amazon64/root/ephemeral-hdfs/conf/core-site.xml
> ec2/deploy.amazon64/root/ephemeral-hdfs/conf/hadoop-env.sh 4e1e6991591e09f8860ab130948b0e787fce2b42
> ec2/deploy.amazon64/root/ephemeral-hdfs/conf/hdfs-site.xml 43e68aa3e2ecbb53bd3b0a99d33b11aadda2cca7
> ec2/deploy.amazon64/root/ephemeral-hdfs/conf/mapred-site.xml
> ec2/deploy.amazon64/root/ephemeral-hdfs/conf/masters
> ec2/deploy.amazon64/root/ephemeral-hdfs/conf/slaves
> ec2/deploy.amazon64/root/mesos-ec2/cluster-url
> ec2/deploy.amazon64/root/mesos-ec2/copy-dir
> ec2/deploy.amazon64/root/mesos-ec2/create-swap
> ec2/deploy.amazon64/root/mesos-ec2/hadoop-framework-conf/core-site.xml
> ec2/deploy.amazon64/root/mesos-ec2/hadoop-framework-conf/hadoop-env.sh d8483140546fe00d6d17494c14ef4b09ae368496
> ec2/deploy.amazon64/root/mesos-ec2/hadoop-framework-conf/mapred-site.xml 0ffa92f115d66adc9e370f4030850216545ade38
> ec2/deploy.amazon64/root/mesos-ec2/haproxy+apache/haproxy.config.template 957c3f6a6b2a2f658e076337d88e69447b2a3341
> ec2/deploy.amazon64/root/mesos-ec2/hypertable/Capfile 8b50912745977cb71232ba1dfa77f8bb0d60191e
> ec2/deploy.amazon64/root/mesos-ec2/hypertable/hypertable.cfg b4d5b7475fab3d4a842e4b0f459abc5ca316996a
> ec2/deploy.amazon64/root/mesos-ec2/masters
> ec2/deploy.amazon64/root/mesos-ec2/mesos-daemon bed27657f8718eecafb83cca5d29e0612a87f129
> ec2/deploy.amazon64/root/mesos-ec2/redeploy-mesos 941d783d82f0708a6da0f4677c3364537dfded63
> ec2/deploy.amazon64/root/mesos-ec2/setup b6b736091d4d5be431c8da29cdb98360a1df2d29
> ec2/deploy.amazon64/root/mesos-ec2/setup-slave 436f417bc5a746ad74cc88c27e630a91d55b0b23
> ec2/deploy.amazon64/root/mesos-ec2/setup-torque 2ac8fd3546063d3ba391147383de53b7824c7c8c
> ec2/deploy.amazon64/root/mesos-ec2/slaves
> ec2/deploy.amazon64/root/mesos-ec2/ssh-no-keychecking
> ec2/deploy.amazon64/root/mesos-ec2/start-hypertable af16c2d7bd615cb0c98f6ba65ff5c69859678850
> ec2/deploy.amazon64/root/mesos-ec2/start-mesos 0f551db396fa7ffebef880dca0232c22808ff7cc
> ec2/deploy.amazon64/root/mesos-ec2/stop-hypertable 7280dc11bfc53ae84b7ecaba34c84810461ed7f4
> ec2/deploy.amazon64/root/mesos-ec2/stop-mesos 9fdb8753dffc5115f94582753e0860538be6232b
> ec2/deploy.amazon64/root/mesos-ec2/zoo
> ec2/deploy.amazon64/root/persistent-hdfs/conf/core-site.xml
> ec2/deploy.amazon64/root/persistent-hdfs/conf/hadoop-env.sh b38ba01817e3b9c9715a476ecb692bac68983f50
> ec2/deploy.amazon64/root/persistent-hdfs/conf/hdfs-site.xml ec000cb2f306232a29cf701b10f230685d2662c9
> ec2/deploy.amazon64/root/persistent-hdfs/conf/mapred-site.xml
> ec2/deploy.amazon64/root/persistent-hdfs/conf/masters
> ec2/deploy.amazon64/root/persistent-hdfs/conf/slaves
> ec2/deploy.amazon64/root/spark/conf/spark-env.sh 6b331ff2257d1b88c8ebe2e8095887a3e2cf0d54
> ec2/deploy.generic/root/mesos-ec2/ec2-variables.sh 1f76e61b2feffa4eadc53745caa5033dde182531
> ec2/deploy.generic/root/mesos-ec2/mesos-master-env.sh PRE-CREATION
> ec2/deploy.generic/root/mesos-ec2/mesos-slave-env.sh PRE-CREATION
> ec2/deploy.generic/root/mesos-ec2/redeploy-mesos PRE-CREATION
> ec2/deploy.generic/root/mesos-ec2/spark-env.sh PRE-CREATION
> ec2/mesos-ec2 3bc5d6307b2759cec0cea430b44baa2809a6f2d2
> ec2/mesos_ec2.py 94fd7b2181d138c97da609fb0ed2ebdc6206cfd6
> ec2/setup-mesos-ami.sh PRE-CREATION
>
> Diff: https://reviews.apache.org/r/11914/diff/
>
>
> Testing
> -------
>
> I've done some very minimal testing that Spark runs over Mesos and can write to the ephemeral HDFS instance and that the Hadoop jobtracker can connect to Mesos and the ephemeral HDFS.
>
>
> Thanks,
>
> Charles Reiss
>
>
Re: Review Request 11914: Get EC2 scripts into workable state
Posted by Niklas Nielsen <ni...@qni.dk>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11914/#review86673
-----------------------------------------------------------
Stale review: Charles, still want this in?
- Niklas Nielsen
On July 14, 2013, 3:44 p.m., Charles Reiss wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/11914/
> -----------------------------------------------------------
>
> (Updated July 14, 2013, 3:44 p.m.)
>
>
> Review request for mesos and Andy Konwinski.
>
>
> Bugs: MESOS-500
> https://issues.apache.org/jira/browse/MESOS-500
>
>
> Repository: mesos-incubating
>
>
> Description
> -------
>
> This is a rather big change that updates the EC2 scripts to deal with changes in Mesos over the past 18 months (or so) and to be usable under the assumption that no one will be maintaining an "official" Mesos AMI.
>
> The big changes are:
> - users are expected to supply their own AMI, there is no default;
> - the webui is firewalled away as exposing the libprocess port is dangerous (AFAIK), instead users are instructed to use the SSH-forwarding SOCKS proxy;
> - instructions and a script for building an AMI (which should work on many Debian- or Fedora-like Linux distros). The resulting AMI should be similar to the one that was originally supplied, including installations of Hadoop (setup to work with Mesos) and Spark;
> - use the deploy scripts to launch mesos;
> - drop support for downloading Mesos from git and rebuilding and redeploying it;
> - drop support for Torque/Hypertable/haproxy in the AMI;
>
>
> Diffs
> -----
>
> docs/EC2-Scripts.md PRE-CREATION
> docs/EC2-Scripts.textile 7a1d1da834b4b85404793a27ca748905200835ca
> ec2/Makefile.am 8c64f485888df1599697eb181fc76aa83206da07
> ec2/deploy.amazon64/root/ephemeral-hdfs/conf/core-site.xml
> ec2/deploy.amazon64/root/ephemeral-hdfs/conf/hadoop-env.sh 4e1e6991591e09f8860ab130948b0e787fce2b42
> ec2/deploy.amazon64/root/ephemeral-hdfs/conf/hdfs-site.xml 43e68aa3e2ecbb53bd3b0a99d33b11aadda2cca7
> ec2/deploy.amazon64/root/ephemeral-hdfs/conf/mapred-site.xml
> ec2/deploy.amazon64/root/ephemeral-hdfs/conf/masters
> ec2/deploy.amazon64/root/ephemeral-hdfs/conf/slaves
> ec2/deploy.amazon64/root/mesos-ec2/cluster-url
> ec2/deploy.amazon64/root/mesos-ec2/copy-dir
> ec2/deploy.amazon64/root/mesos-ec2/create-swap
> ec2/deploy.amazon64/root/mesos-ec2/hadoop-framework-conf/core-site.xml
> ec2/deploy.amazon64/root/mesos-ec2/hadoop-framework-conf/hadoop-env.sh d8483140546fe00d6d17494c14ef4b09ae368496
> ec2/deploy.amazon64/root/mesos-ec2/hadoop-framework-conf/mapred-site.xml 0ffa92f115d66adc9e370f4030850216545ade38
> ec2/deploy.amazon64/root/mesos-ec2/haproxy+apache/haproxy.config.template 957c3f6a6b2a2f658e076337d88e69447b2a3341
> ec2/deploy.amazon64/root/mesos-ec2/hypertable/Capfile 8b50912745977cb71232ba1dfa77f8bb0d60191e
> ec2/deploy.amazon64/root/mesos-ec2/hypertable/hypertable.cfg b4d5b7475fab3d4a842e4b0f459abc5ca316996a
> ec2/deploy.amazon64/root/mesos-ec2/masters
> ec2/deploy.amazon64/root/mesos-ec2/mesos-daemon bed27657f8718eecafb83cca5d29e0612a87f129
> ec2/deploy.amazon64/root/mesos-ec2/redeploy-mesos 941d783d82f0708a6da0f4677c3364537dfded63
> ec2/deploy.amazon64/root/mesos-ec2/setup b6b736091d4d5be431c8da29cdb98360a1df2d29
> ec2/deploy.amazon64/root/mesos-ec2/setup-slave 436f417bc5a746ad74cc88c27e630a91d55b0b23
> ec2/deploy.amazon64/root/mesos-ec2/setup-torque 2ac8fd3546063d3ba391147383de53b7824c7c8c
> ec2/deploy.amazon64/root/mesos-ec2/slaves
> ec2/deploy.amazon64/root/mesos-ec2/ssh-no-keychecking
> ec2/deploy.amazon64/root/mesos-ec2/start-hypertable af16c2d7bd615cb0c98f6ba65ff5c69859678850
> ec2/deploy.amazon64/root/mesos-ec2/start-mesos 0f551db396fa7ffebef880dca0232c22808ff7cc
> ec2/deploy.amazon64/root/mesos-ec2/stop-hypertable 7280dc11bfc53ae84b7ecaba34c84810461ed7f4
> ec2/deploy.amazon64/root/mesos-ec2/stop-mesos 9fdb8753dffc5115f94582753e0860538be6232b
> ec2/deploy.amazon64/root/mesos-ec2/zoo
> ec2/deploy.amazon64/root/persistent-hdfs/conf/core-site.xml
> ec2/deploy.amazon64/root/persistent-hdfs/conf/hadoop-env.sh b38ba01817e3b9c9715a476ecb692bac68983f50
> ec2/deploy.amazon64/root/persistent-hdfs/conf/hdfs-site.xml ec000cb2f306232a29cf701b10f230685d2662c9
> ec2/deploy.amazon64/root/persistent-hdfs/conf/mapred-site.xml
> ec2/deploy.amazon64/root/persistent-hdfs/conf/masters
> ec2/deploy.amazon64/root/persistent-hdfs/conf/slaves
> ec2/deploy.amazon64/root/spark/conf/spark-env.sh 6b331ff2257d1b88c8ebe2e8095887a3e2cf0d54
> ec2/deploy.generic/root/mesos-ec2/ec2-variables.sh 1f76e61b2feffa4eadc53745caa5033dde182531
> ec2/deploy.generic/root/mesos-ec2/mesos-master-env.sh PRE-CREATION
> ec2/deploy.generic/root/mesos-ec2/mesos-slave-env.sh PRE-CREATION
> ec2/deploy.generic/root/mesos-ec2/redeploy-mesos PRE-CREATION
> ec2/deploy.generic/root/mesos-ec2/spark-env.sh PRE-CREATION
> ec2/mesos-ec2 3bc5d6307b2759cec0cea430b44baa2809a6f2d2
> ec2/mesos_ec2.py 94fd7b2181d138c97da609fb0ed2ebdc6206cfd6
> ec2/setup-mesos-ami.sh PRE-CREATION
>
> Diff: https://reviews.apache.org/r/11914/diff/
>
>
> Testing
> -------
>
> I've done some very minimal testing that Spark runs over Mesos and can write to the ephemeral HDFS instance and that the Hadoop jobtracker can connect to Mesos and the ephemeral HDFS.
>
>
> Thanks,
>
> Charles Reiss
>
>
Re: Review Request 11914: Get EC2 scripts into workable state
Posted by Mesos ReviewBot <re...@mesos.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11914/#review86707
-----------------------------------------------------------
Bad patch!
Reviews applied: [11914]
Failed command: ./support/apply-review.sh -n -r 11914
Error:
2015-06-04 19:45:24 URL:https://reviews.apache.org/r/11914/diff/raw/ [86959/86959] -> "11914.patch" [1]
11914.patch:97: trailing whitespace.
11914.patch:188: trailing whitespace.
11914.patch:501: trailing whitespace.
error: docs/EC2-Scripts.textile: does not exist in index
error: patch failed: ec2/deploy.amazon64/root/mesos-ec2/hypertable/Capfile:1
error: ec2/deploy.amazon64/root/mesos-ec2/hypertable/Capfile: patch does not apply
11914.patch:1832: new blank line at EOF.
+
error: patch failed: ec2/mesos-ec2:17
error: ec2/mesos-ec2: patch does not apply
error: patch failed: ec2/mesos_ec2.py:307
error: ec2/mesos_ec2.py: patch does not apply
Failed to apply patch
- Mesos ReviewBot
On July 14, 2013, 10:44 p.m., Charles Reiss wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/11914/
> -----------------------------------------------------------
>
> (Updated July 14, 2013, 10:44 p.m.)
>
>
> Review request for mesos and Andy Konwinski.
>
>
> Bugs: MESOS-500
> https://issues.apache.org/jira/browse/MESOS-500
>
>
> Repository: mesos-incubating
>
>
> Description
> -------
>
> This is a rather big change that updates the EC2 scripts to deal with changes in Mesos over the past 18 months (or so) and to be usable under the assumption that no one will be maintaining an "official" Mesos AMI.
>
> The big changes are:
> - users are expected to supply their own AMI, there is no default;
> - the webui is firewalled away as exposing the libprocess port is dangerous (AFAIK), instead users are instructed to use the SSH-forwarding SOCKS proxy;
> - instructions and a script for building an AMI (which should work on many Debian- or Fedora-like Linux distros). The resulting AMI should be similar to the one that was originally supplied, including installations of Hadoop (setup to work with Mesos) and Spark;
> - use the deploy scripts to launch mesos;
> - drop support for downloading Mesos from git and rebuilding and redeploying it;
> - drop support for Torque/Hypertable/haproxy in the AMI;
>
>
> Diffs
> -----
>
> docs/EC2-Scripts.md PRE-CREATION
> docs/EC2-Scripts.textile 7a1d1da834b4b85404793a27ca748905200835ca
> ec2/Makefile.am 8c64f485888df1599697eb181fc76aa83206da07
> ec2/deploy.amazon64/root/ephemeral-hdfs/conf/core-site.xml
> ec2/deploy.amazon64/root/ephemeral-hdfs/conf/hadoop-env.sh 4e1e6991591e09f8860ab130948b0e787fce2b42
> ec2/deploy.amazon64/root/ephemeral-hdfs/conf/hdfs-site.xml 43e68aa3e2ecbb53bd3b0a99d33b11aadda2cca7
> ec2/deploy.amazon64/root/ephemeral-hdfs/conf/mapred-site.xml
> ec2/deploy.amazon64/root/ephemeral-hdfs/conf/masters
> ec2/deploy.amazon64/root/ephemeral-hdfs/conf/slaves
> ec2/deploy.amazon64/root/mesos-ec2/cluster-url
> ec2/deploy.amazon64/root/mesos-ec2/copy-dir
> ec2/deploy.amazon64/root/mesos-ec2/create-swap
> ec2/deploy.amazon64/root/mesos-ec2/hadoop-framework-conf/core-site.xml
> ec2/deploy.amazon64/root/mesos-ec2/hadoop-framework-conf/hadoop-env.sh d8483140546fe00d6d17494c14ef4b09ae368496
> ec2/deploy.amazon64/root/mesos-ec2/hadoop-framework-conf/mapred-site.xml 0ffa92f115d66adc9e370f4030850216545ade38
> ec2/deploy.amazon64/root/mesos-ec2/haproxy+apache/haproxy.config.template 957c3f6a6b2a2f658e076337d88e69447b2a3341
> ec2/deploy.amazon64/root/mesos-ec2/hypertable/Capfile 8b50912745977cb71232ba1dfa77f8bb0d60191e
> ec2/deploy.amazon64/root/mesos-ec2/hypertable/hypertable.cfg b4d5b7475fab3d4a842e4b0f459abc5ca316996a
> ec2/deploy.amazon64/root/mesos-ec2/masters
> ec2/deploy.amazon64/root/mesos-ec2/mesos-daemon bed27657f8718eecafb83cca5d29e0612a87f129
> ec2/deploy.amazon64/root/mesos-ec2/redeploy-mesos 941d783d82f0708a6da0f4677c3364537dfded63
> ec2/deploy.amazon64/root/mesos-ec2/setup b6b736091d4d5be431c8da29cdb98360a1df2d29
> ec2/deploy.amazon64/root/mesos-ec2/setup-slave 436f417bc5a746ad74cc88c27e630a91d55b0b23
> ec2/deploy.amazon64/root/mesos-ec2/setup-torque 2ac8fd3546063d3ba391147383de53b7824c7c8c
> ec2/deploy.amazon64/root/mesos-ec2/slaves
> ec2/deploy.amazon64/root/mesos-ec2/ssh-no-keychecking
> ec2/deploy.amazon64/root/mesos-ec2/start-hypertable af16c2d7bd615cb0c98f6ba65ff5c69859678850
> ec2/deploy.amazon64/root/mesos-ec2/start-mesos 0f551db396fa7ffebef880dca0232c22808ff7cc
> ec2/deploy.amazon64/root/mesos-ec2/stop-hypertable 7280dc11bfc53ae84b7ecaba34c84810461ed7f4
> ec2/deploy.amazon64/root/mesos-ec2/stop-mesos 9fdb8753dffc5115f94582753e0860538be6232b
> ec2/deploy.amazon64/root/mesos-ec2/zoo
> ec2/deploy.amazon64/root/persistent-hdfs/conf/core-site.xml
> ec2/deploy.amazon64/root/persistent-hdfs/conf/hadoop-env.sh b38ba01817e3b9c9715a476ecb692bac68983f50
> ec2/deploy.amazon64/root/persistent-hdfs/conf/hdfs-site.xml ec000cb2f306232a29cf701b10f230685d2662c9
> ec2/deploy.amazon64/root/persistent-hdfs/conf/mapred-site.xml
> ec2/deploy.amazon64/root/persistent-hdfs/conf/masters
> ec2/deploy.amazon64/root/persistent-hdfs/conf/slaves
> ec2/deploy.amazon64/root/spark/conf/spark-env.sh 6b331ff2257d1b88c8ebe2e8095887a3e2cf0d54
> ec2/deploy.generic/root/mesos-ec2/ec2-variables.sh 1f76e61b2feffa4eadc53745caa5033dde182531
> ec2/deploy.generic/root/mesos-ec2/mesos-master-env.sh PRE-CREATION
> ec2/deploy.generic/root/mesos-ec2/mesos-slave-env.sh PRE-CREATION
> ec2/deploy.generic/root/mesos-ec2/redeploy-mesos PRE-CREATION
> ec2/deploy.generic/root/mesos-ec2/spark-env.sh PRE-CREATION
> ec2/mesos-ec2 3bc5d6307b2759cec0cea430b44baa2809a6f2d2
> ec2/mesos_ec2.py 94fd7b2181d138c97da609fb0ed2ebdc6206cfd6
> ec2/setup-mesos-ami.sh PRE-CREATION
>
> Diff: https://reviews.apache.org/r/11914/diff/
>
>
> Testing
> -------
>
> I've done some very minimal testing that Spark runs over Mesos and can write to the ephemeral HDFS instance and that the Hadoop jobtracker can connect to Mesos and the ephemeral HDFS.
>
>
> Thanks,
>
> Charles Reiss
>
>
Re: Review Request 11914: Get EC2 scripts into workable state
Posted by Charles Reiss <wo...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11914/
-----------------------------------------------------------
(Updated July 14, 2013, 10:44 p.m.)
Review request for mesos and Andy Konwinski.
Changes
-------
Move docs from ec2/README.md to docs/EC2-Scripts.md (merging in old docs/EC2-Scripts.textile).
Bugs: MESOS-500
https://issues.apache.org/jira/browse/MESOS-500
Repository: mesos
Description
-------
This is a rather big change that updates the EC2 scripts to deal with changes in Mesos over the past 18 months (or so) and to be usable under the assumption that no one will be maintaining an "official" Mesos AMI.
The big changes are:
- users are expected to supply their own AMI, there is no default;
- the webui is firewalled away as exposing the libprocess port is dangerous (AFAIK), instead users are instructed to use the SSH-forwarding SOCKS proxy;
- instructions and a script for building an AMI (which should work on many Debian- or Fedora-like Linux distros). The resulting AMI should be similar to the one that was originally supplied, including installations of Hadoop (setup to work with Mesos) and Spark;
- use the deploy scripts to launch mesos;
- drop support for downloading Mesos from git and rebuilding and redeploying it;
- drop support for Torque/Hypertable/haproxy in the AMI;
Diffs (updated)
-----
docs/EC2-Scripts.md PRE-CREATION
docs/EC2-Scripts.textile 7a1d1da834b4b85404793a27ca748905200835ca
ec2/Makefile.am 8c64f485888df1599697eb181fc76aa83206da07
ec2/deploy.amazon64/root/ephemeral-hdfs/conf/core-site.xml
ec2/deploy.amazon64/root/ephemeral-hdfs/conf/hadoop-env.sh 4e1e6991591e09f8860ab130948b0e787fce2b42
ec2/deploy.amazon64/root/ephemeral-hdfs/conf/hdfs-site.xml 43e68aa3e2ecbb53bd3b0a99d33b11aadda2cca7
ec2/deploy.amazon64/root/ephemeral-hdfs/conf/mapred-site.xml
ec2/deploy.amazon64/root/ephemeral-hdfs/conf/masters
ec2/deploy.amazon64/root/ephemeral-hdfs/conf/slaves
ec2/deploy.amazon64/root/mesos-ec2/cluster-url
ec2/deploy.amazon64/root/mesos-ec2/copy-dir
ec2/deploy.amazon64/root/mesos-ec2/create-swap
ec2/deploy.amazon64/root/mesos-ec2/hadoop-framework-conf/core-site.xml
ec2/deploy.amazon64/root/mesos-ec2/hadoop-framework-conf/hadoop-env.sh d8483140546fe00d6d17494c14ef4b09ae368496
ec2/deploy.amazon64/root/mesos-ec2/hadoop-framework-conf/mapred-site.xml 0ffa92f115d66adc9e370f4030850216545ade38
ec2/deploy.amazon64/root/mesos-ec2/haproxy+apache/haproxy.config.template 957c3f6a6b2a2f658e076337d88e69447b2a3341
ec2/deploy.amazon64/root/mesos-ec2/hypertable/Capfile 8b50912745977cb71232ba1dfa77f8bb0d60191e
ec2/deploy.amazon64/root/mesos-ec2/hypertable/hypertable.cfg b4d5b7475fab3d4a842e4b0f459abc5ca316996a
ec2/deploy.amazon64/root/mesos-ec2/masters
ec2/deploy.amazon64/root/mesos-ec2/mesos-daemon bed27657f8718eecafb83cca5d29e0612a87f129
ec2/deploy.amazon64/root/mesos-ec2/redeploy-mesos 941d783d82f0708a6da0f4677c3364537dfded63
ec2/deploy.amazon64/root/mesos-ec2/setup b6b736091d4d5be431c8da29cdb98360a1df2d29
ec2/deploy.amazon64/root/mesos-ec2/setup-slave 436f417bc5a746ad74cc88c27e630a91d55b0b23
ec2/deploy.amazon64/root/mesos-ec2/setup-torque 2ac8fd3546063d3ba391147383de53b7824c7c8c
ec2/deploy.amazon64/root/mesos-ec2/slaves
ec2/deploy.amazon64/root/mesos-ec2/ssh-no-keychecking
ec2/deploy.amazon64/root/mesos-ec2/start-hypertable af16c2d7bd615cb0c98f6ba65ff5c69859678850
ec2/deploy.amazon64/root/mesos-ec2/start-mesos 0f551db396fa7ffebef880dca0232c22808ff7cc
ec2/deploy.amazon64/root/mesos-ec2/stop-hypertable 7280dc11bfc53ae84b7ecaba34c84810461ed7f4
ec2/deploy.amazon64/root/mesos-ec2/stop-mesos 9fdb8753dffc5115f94582753e0860538be6232b
ec2/deploy.amazon64/root/mesos-ec2/zoo
ec2/deploy.amazon64/root/persistent-hdfs/conf/core-site.xml
ec2/deploy.amazon64/root/persistent-hdfs/conf/hadoop-env.sh b38ba01817e3b9c9715a476ecb692bac68983f50
ec2/deploy.amazon64/root/persistent-hdfs/conf/hdfs-site.xml ec000cb2f306232a29cf701b10f230685d2662c9
ec2/deploy.amazon64/root/persistent-hdfs/conf/mapred-site.xml
ec2/deploy.amazon64/root/persistent-hdfs/conf/masters
ec2/deploy.amazon64/root/persistent-hdfs/conf/slaves
ec2/deploy.amazon64/root/spark/conf/spark-env.sh 6b331ff2257d1b88c8ebe2e8095887a3e2cf0d54
ec2/deploy.generic/root/mesos-ec2/ec2-variables.sh 1f76e61b2feffa4eadc53745caa5033dde182531
ec2/deploy.generic/root/mesos-ec2/mesos-master-env.sh PRE-CREATION
ec2/deploy.generic/root/mesos-ec2/mesos-slave-env.sh PRE-CREATION
ec2/deploy.generic/root/mesos-ec2/redeploy-mesos PRE-CREATION
ec2/deploy.generic/root/mesos-ec2/spark-env.sh PRE-CREATION
ec2/mesos-ec2 3bc5d6307b2759cec0cea430b44baa2809a6f2d2
ec2/mesos_ec2.py 94fd7b2181d138c97da609fb0ed2ebdc6206cfd6
ec2/setup-mesos-ami.sh PRE-CREATION
Diff: https://reviews.apache.org/r/11914/diff/
Testing
-------
I've done some very minimal testing that Spark runs over Mesos and can write to the ephemeral HDFS instance and that the Hadoop jobtracker can connect to Mesos and the ephemeral HDFS.
Thanks,
Charles Reiss
Re: Review Request 11914: Get EC2 scripts into workable state
Posted by Charles Reiss <wo...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11914/
-----------------------------------------------------------
(Updated July 8, 2013, 6:54 p.m.)
Review request for mesos and Andy Konwinski.
Bugs: MESOS-500
https://issues.apache.org/jira/browse/MESOS-500
Repository: mesos
Description
-------
This is a rather big change that updates the EC2 scripts to deal with changes in Mesos over the past 18 months (or so) and to be usable under the assumption that no one will be maintaining an "official" Mesos AMI.
The big changes are:
- users are expected to supply their own AMI, there is no default;
- the webui is firewalled away as exposing the libprocess port is dangerous (AFAIK), instead users are instructed to use the SSH-forwarding SOCKS proxy;
- instructions and a script for building an AMI (which should work on many Debian- or Fedora-like Linux distros). The resulting AMI should be similar to the one that was originally supplied, including installations of Hadoop (setup to work with Mesos) and Spark;
- use the deploy scripts to launch mesos;
- drop support for downloading Mesos from git and rebuilding and redeploying it;
- drop support for Torque/Hypertable/haproxy in the AMI;
Diffs
-----
ec2/Makefile.am 8c64f485888df1599697eb181fc76aa83206da07
ec2/README.md PRE-CREATION
ec2/deploy.amazon64/root/ephemeral-hdfs/conf/core-site.xml
ec2/deploy.amazon64/root/ephemeral-hdfs/conf/hadoop-env.sh 4e1e6991591e09f8860ab130948b0e787fce2b42
ec2/deploy.amazon64/root/ephemeral-hdfs/conf/hdfs-site.xml 43e68aa3e2ecbb53bd3b0a99d33b11aadda2cca7
ec2/deploy.amazon64/root/ephemeral-hdfs/conf/mapred-site.xml
ec2/deploy.amazon64/root/ephemeral-hdfs/conf/masters
ec2/deploy.amazon64/root/ephemeral-hdfs/conf/slaves
ec2/deploy.amazon64/root/mesos-ec2/cluster-url
ec2/deploy.amazon64/root/mesos-ec2/copy-dir
ec2/deploy.amazon64/root/mesos-ec2/create-swap
ec2/deploy.amazon64/root/mesos-ec2/hadoop-framework-conf/core-site.xml
ec2/deploy.amazon64/root/mesos-ec2/hadoop-framework-conf/hadoop-env.sh d8483140546fe00d6d17494c14ef4b09ae368496
ec2/deploy.amazon64/root/mesos-ec2/hadoop-framework-conf/mapred-site.xml 0ffa92f115d66adc9e370f4030850216545ade38
ec2/deploy.amazon64/root/mesos-ec2/haproxy+apache/haproxy.config.template 957c3f6a6b2a2f658e076337d88e69447b2a3341
ec2/deploy.amazon64/root/mesos-ec2/hypertable/Capfile 8b50912745977cb71232ba1dfa77f8bb0d60191e
ec2/deploy.amazon64/root/mesos-ec2/hypertable/hypertable.cfg b4d5b7475fab3d4a842e4b0f459abc5ca316996a
ec2/deploy.amazon64/root/mesos-ec2/masters
ec2/deploy.amazon64/root/mesos-ec2/mesos-daemon bed27657f8718eecafb83cca5d29e0612a87f129
ec2/deploy.amazon64/root/mesos-ec2/redeploy-mesos 941d783d82f0708a6da0f4677c3364537dfded63
ec2/deploy.amazon64/root/mesos-ec2/setup b6b736091d4d5be431c8da29cdb98360a1df2d29
ec2/deploy.amazon64/root/mesos-ec2/setup-slave 436f417bc5a746ad74cc88c27e630a91d55b0b23
ec2/deploy.amazon64/root/mesos-ec2/setup-torque 2ac8fd3546063d3ba391147383de53b7824c7c8c
ec2/deploy.amazon64/root/mesos-ec2/slaves
ec2/deploy.amazon64/root/mesos-ec2/ssh-no-keychecking
ec2/deploy.amazon64/root/mesos-ec2/start-hypertable af16c2d7bd615cb0c98f6ba65ff5c69859678850
ec2/deploy.amazon64/root/mesos-ec2/start-mesos 0f551db396fa7ffebef880dca0232c22808ff7cc
ec2/deploy.amazon64/root/mesos-ec2/stop-hypertable 7280dc11bfc53ae84b7ecaba34c84810461ed7f4
ec2/deploy.amazon64/root/mesos-ec2/stop-mesos 9fdb8753dffc5115f94582753e0860538be6232b
ec2/deploy.amazon64/root/mesos-ec2/zoo
ec2/deploy.amazon64/root/persistent-hdfs/conf/core-site.xml
ec2/deploy.amazon64/root/persistent-hdfs/conf/hadoop-env.sh b38ba01817e3b9c9715a476ecb692bac68983f50
ec2/deploy.amazon64/root/persistent-hdfs/conf/hdfs-site.xml ec000cb2f306232a29cf701b10f230685d2662c9
ec2/deploy.amazon64/root/persistent-hdfs/conf/mapred-site.xml
ec2/deploy.amazon64/root/persistent-hdfs/conf/masters
ec2/deploy.amazon64/root/persistent-hdfs/conf/slaves
ec2/deploy.amazon64/root/spark/conf/spark-env.sh 6b331ff2257d1b88c8ebe2e8095887a3e2cf0d54
ec2/deploy.generic/root/mesos-ec2/ec2-variables.sh 1f76e61b2feffa4eadc53745caa5033dde182531
ec2/deploy.generic/root/mesos-ec2/mesos-master-env.sh PRE-CREATION
ec2/deploy.generic/root/mesos-ec2/mesos-slave-env.sh PRE-CREATION
ec2/deploy.generic/root/mesos-ec2/redeploy-mesos PRE-CREATION
ec2/deploy.generic/root/mesos-ec2/spark-env.sh PRE-CREATION
ec2/mesos-ec2 3bc5d6307b2759cec0cea430b44baa2809a6f2d2
ec2/mesos_ec2.py 94fd7b2181d138c97da609fb0ed2ebdc6206cfd6
ec2/setup-mesos-ami.sh PRE-CREATION
Diff: https://reviews.apache.org/r/11914/diff/
Testing
-------
I've done some very minimal testing that Spark runs over Mesos and can write to the ephemeral HDFS instance and that the Hadoop jobtracker can connect to Mesos and the ephemeral HDFS.
Thanks,
Charles Reiss
Re: Review Request: Get EC2 scripts into workable state
Posted by Dharmesh Kakadia <dh...@gmail.com>.
Thanks !! I had tried downloading patch from review-board, but the patched
version was failing. I have stopped that instance, but it was during
hadoop compilation as far as I remember. I will try this.
Thanks,
Dharmesh
On Mon, Jul 8, 2013 at 11:19 PM, Charles Reiss <wo...@gmail.com> wrote:
> On 7/8/13 24:55 , Dharmesh Kakadia wrote:
> > Hey Charles,
> >
> > How can I try this new scripts as its not yet "shipped" ?
>
> You should be able to download the diff from reviewboard and apply with
> something like 'patch -p1 <rbXXXXX.patch'. (I also have it at
> https://github.com/woggle/mesos/tree/fix-ec2 ). Then the patch has some
> instructions in ec2/README.md. (I see now that I neglected to fixup
> docs/EC2-Scripts.textile, which I probably didn't remember was there.)
>
> - Charles
>
>
Re: Review Request: Get EC2 scripts into workable state
Posted by Charles Reiss <wo...@gmail.com>.
On 7/8/13 24:55 , Dharmesh Kakadia wrote:
> Hey Charles,
>
> How can I try this new scripts as its not yet "shipped" ?
You should be able to download the diff from reviewboard and apply with
something like 'patch -p1 <rbXXXXX.patch'. (I also have it at
https://github.com/woggle/mesos/tree/fix-ec2 ). Then the patch has some
instructions in ec2/README.md. (I see now that I neglected to fixup
docs/EC2-Scripts.textile, which I probably didn't remember was there.)
- Charles
Re: Review Request: Get EC2 scripts into workable state
Posted by Dharmesh Kakadia <dh...@gmail.com>.
Hey Charles,
How can I try this new scripts as its not yet "shipped" ?
Thanks,
Dharmesh
On Fri, Jun 21, 2013 at 6:18 AM, Charles Reiss <wo...@gmail.com> wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/11914/
> -----------------------------------------------------------
>
> (Updated June 21, 2013, 12:48 a.m.)
>
>
> Review request for mesos.
>
>
> Changes
> -------
>
> Updated diffs; some fixes that came up during testing, and remember to git
> add the setup AMI script + README file for it.
>
> Unfortunately, it seems that this version of reviewboard has some problems
> with some renames in the diff.
>
>
> Description
> -------
>
> This is a rather big change that updates the EC2 scripts to deal with
> changes in Mesos over the past 18 months (or so) and to be usable under the
> assumption that no one will be maintaining an "official" Mesos AMI.
>
> The big changes are:
> - users are expected to supply their own AMI, there is no default;
> - the webui is firewalled away as exposing the libprocess port is
> dangerous (AFAIK), instead users are instructed to use the SSH-forwarding
> SOCKS proxy;
> - instructions and a script for building an AMI (which should work on many
> Debian- or Fedora-like Linux distros). The resulting AMI should be similar
> to the one that was originally supplied, including installations of Hadoop
> (setup to work with Mesos) and Spark;
> - use the deploy scripts to launch mesos;
> - drop support for downloading Mesos from git and rebuilding and
> redeploying it;
> - drop support for Torque/Hypertable/haproxy in the AMI;
>
>
> This addresses bug MESOS-500.
> https://issues.apache.org/jira/browse/MESOS-500
>
>
> Diffs (updated)
> -----
>
> ec2/Makefile.am 8c64f485888df1599697eb181fc76aa83206da07
> ec2/README.md PRE-CREATION
> ec2/deploy.amazon64/root/ephemeral-hdfs/conf/core-site.xml
> ec2/deploy.amazon64/root/ephemeral-hdfs/conf/hadoop-env.sh
> 4e1e6991591e09f8860ab130948b0e787fce2b42
> ec2/deploy.amazon64/root/ephemeral-hdfs/conf/hdfs-site.xml
> 43e68aa3e2ecbb53bd3b0a99d33b11aadda2cca7
> ec2/deploy.amazon64/root/ephemeral-hdfs/conf/mapred-site.xml
> ec2/deploy.amazon64/root/ephemeral-hdfs/conf/masters
> ec2/deploy.amazon64/root/ephemeral-hdfs/conf/slaves
> ec2/deploy.amazon64/root/mesos-ec2/cluster-url
> ec2/deploy.amazon64/root/mesos-ec2/copy-dir
> ec2/deploy.amazon64/root/mesos-ec2/create-swap
> ec2/deploy.amazon64/root/mesos-ec2/hadoop-framework-conf/core-site.xml
> ec2/deploy.amazon64/root/mesos-ec2/hadoop-framework-conf/hadoop-env.sh
> d8483140546fe00d6d17494c14ef4b09ae368496
> ec2/deploy.amazon64/root/mesos-ec2/hadoop-framework-conf/mapred-site.xml
> 0ffa92f115d66adc9e370f4030850216545ade38
>
> ec2/deploy.amazon64/root/mesos-ec2/haproxy+apache/haproxy.config.template
> 957c3f6a6b2a2f658e076337d88e69447b2a3341
> ec2/deploy.amazon64/root/mesos-ec2/hypertable/Capfile
> 8b50912745977cb71232ba1dfa77f8bb0d60191e
> ec2/deploy.amazon64/root/mesos-ec2/hypertable/hypertable.cfg
> b4d5b7475fab3d4a842e4b0f459abc5ca316996a
> ec2/deploy.amazon64/root/mesos-ec2/masters
> ec2/deploy.amazon64/root/mesos-ec2/mesos-daemon
> bed27657f8718eecafb83cca5d29e0612a87f129
> ec2/deploy.amazon64/root/mesos-ec2/redeploy-mesos
> 941d783d82f0708a6da0f4677c3364537dfded63
> ec2/deploy.amazon64/root/mesos-ec2/setup
> b6b736091d4d5be431c8da29cdb98360a1df2d29
> ec2/deploy.amazon64/root/mesos-ec2/setup-slave
> 436f417bc5a746ad74cc88c27e630a91d55b0b23
> ec2/deploy.amazon64/root/mesos-ec2/setup-torque
> 2ac8fd3546063d3ba391147383de53b7824c7c8c
> ec2/deploy.amazon64/root/mesos-ec2/slaves
> ec2/deploy.amazon64/root/mesos-ec2/ssh-no-keychecking
> ec2/deploy.amazon64/root/mesos-ec2/start-hypertable
> af16c2d7bd615cb0c98f6ba65ff5c69859678850
> ec2/deploy.amazon64/root/mesos-ec2/start-mesos
> 0f551db396fa7ffebef880dca0232c22808ff7cc
> ec2/deploy.amazon64/root/mesos-ec2/stop-hypertable
> 7280dc11bfc53ae84b7ecaba34c84810461ed7f4
> ec2/deploy.amazon64/root/mesos-ec2/stop-mesos
> 9fdb8753dffc5115f94582753e0860538be6232b
> ec2/deploy.amazon64/root/mesos-ec2/zoo
> ec2/deploy.amazon64/root/persistent-hdfs/conf/core-site.xml
> ec2/deploy.amazon64/root/persistent-hdfs/conf/hadoop-env.sh
> b38ba01817e3b9c9715a476ecb692bac68983f50
> ec2/deploy.amazon64/root/persistent-hdfs/conf/hdfs-site.xml
> ec000cb2f306232a29cf701b10f230685d2662c9
> ec2/deploy.amazon64/root/persistent-hdfs/conf/mapred-site.xml
> ec2/deploy.amazon64/root/persistent-hdfs/conf/masters
> ec2/deploy.amazon64/root/persistent-hdfs/conf/slaves
> ec2/deploy.amazon64/root/spark/conf/spark-env.sh
> 6b331ff2257d1b88c8ebe2e8095887a3e2cf0d54
> ec2/deploy.generic/root/mesos-ec2/ec2-variables.sh
> 1f76e61b2feffa4eadc53745caa5033dde182531
> ec2/deploy.generic/root/mesos-ec2/mesos-master-env.sh PRE-CREATION
> ec2/deploy.generic/root/mesos-ec2/mesos-slave-env.sh PRE-CREATION
> ec2/deploy.generic/root/mesos-ec2/redeploy-mesos PRE-CREATION
> ec2/deploy.generic/root/mesos-ec2/spark-env.sh PRE-CREATION
> ec2/mesos-ec2 3bc5d6307b2759cec0cea430b44baa2809a6f2d2
> ec2/mesos_ec2.py 94fd7b2181d138c97da609fb0ed2ebdc6206cfd6
> ec2/setup-mesos-ami.sh PRE-CREATION
>
> Diff: https://reviews.apache.org/r/11914/diff/
>
>
> Testing (updated)
> -------
>
> I've done some very minimal testing that Spark runs over Mesos and can
> write to the ephemeral HDFS instance and that the Hadoop jobtracker can
> connect to Mesos and the ephemeral HDFS.
>
>
> Thanks,
>
> Charles Reiss
>
>
Re: Review Request: Get EC2 scripts into workable state
Posted by Charles Reiss <wo...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11914/
-----------------------------------------------------------
(Updated June 21, 2013, 12:48 a.m.)
Review request for mesos.
Changes
-------
Updated diffs; some fixes that came up during testing, and remember to git add the setup AMI script + README file for it.
Unfortunately, it seems that this version of reviewboard has some problems with some renames in the diff.
Description
-------
This is a rather big change that updates the EC2 scripts to deal with changes in Mesos over the past 18 months (or so) and to be usable under the assumption that no one will be maintaining an "official" Mesos AMI.
The big changes are:
- users are expected to supply their own AMI, there is no default;
- the webui is firewalled away as exposing the libprocess port is dangerous (AFAIK), instead users are instructed to use the SSH-forwarding SOCKS proxy;
- instructions and a script for building an AMI (which should work on many Debian- or Fedora-like Linux distros). The resulting AMI should be similar to the one that was originally supplied, including installations of Hadoop (setup to work with Mesos) and Spark;
- use the deploy scripts to launch mesos;
- drop support for downloading Mesos from git and rebuilding and redeploying it;
- drop support for Torque/Hypertable/haproxy in the AMI;
This addresses bug MESOS-500.
https://issues.apache.org/jira/browse/MESOS-500
Diffs (updated)
-----
ec2/Makefile.am 8c64f485888df1599697eb181fc76aa83206da07
ec2/README.md PRE-CREATION
ec2/deploy.amazon64/root/ephemeral-hdfs/conf/core-site.xml
ec2/deploy.amazon64/root/ephemeral-hdfs/conf/hadoop-env.sh 4e1e6991591e09f8860ab130948b0e787fce2b42
ec2/deploy.amazon64/root/ephemeral-hdfs/conf/hdfs-site.xml 43e68aa3e2ecbb53bd3b0a99d33b11aadda2cca7
ec2/deploy.amazon64/root/ephemeral-hdfs/conf/mapred-site.xml
ec2/deploy.amazon64/root/ephemeral-hdfs/conf/masters
ec2/deploy.amazon64/root/ephemeral-hdfs/conf/slaves
ec2/deploy.amazon64/root/mesos-ec2/cluster-url
ec2/deploy.amazon64/root/mesos-ec2/copy-dir
ec2/deploy.amazon64/root/mesos-ec2/create-swap
ec2/deploy.amazon64/root/mesos-ec2/hadoop-framework-conf/core-site.xml
ec2/deploy.amazon64/root/mesos-ec2/hadoop-framework-conf/hadoop-env.sh d8483140546fe00d6d17494c14ef4b09ae368496
ec2/deploy.amazon64/root/mesos-ec2/hadoop-framework-conf/mapred-site.xml 0ffa92f115d66adc9e370f4030850216545ade38
ec2/deploy.amazon64/root/mesos-ec2/haproxy+apache/haproxy.config.template 957c3f6a6b2a2f658e076337d88e69447b2a3341
ec2/deploy.amazon64/root/mesos-ec2/hypertable/Capfile 8b50912745977cb71232ba1dfa77f8bb0d60191e
ec2/deploy.amazon64/root/mesos-ec2/hypertable/hypertable.cfg b4d5b7475fab3d4a842e4b0f459abc5ca316996a
ec2/deploy.amazon64/root/mesos-ec2/masters
ec2/deploy.amazon64/root/mesos-ec2/mesos-daemon bed27657f8718eecafb83cca5d29e0612a87f129
ec2/deploy.amazon64/root/mesos-ec2/redeploy-mesos 941d783d82f0708a6da0f4677c3364537dfded63
ec2/deploy.amazon64/root/mesos-ec2/setup b6b736091d4d5be431c8da29cdb98360a1df2d29
ec2/deploy.amazon64/root/mesos-ec2/setup-slave 436f417bc5a746ad74cc88c27e630a91d55b0b23
ec2/deploy.amazon64/root/mesos-ec2/setup-torque 2ac8fd3546063d3ba391147383de53b7824c7c8c
ec2/deploy.amazon64/root/mesos-ec2/slaves
ec2/deploy.amazon64/root/mesos-ec2/ssh-no-keychecking
ec2/deploy.amazon64/root/mesos-ec2/start-hypertable af16c2d7bd615cb0c98f6ba65ff5c69859678850
ec2/deploy.amazon64/root/mesos-ec2/start-mesos 0f551db396fa7ffebef880dca0232c22808ff7cc
ec2/deploy.amazon64/root/mesos-ec2/stop-hypertable 7280dc11bfc53ae84b7ecaba34c84810461ed7f4
ec2/deploy.amazon64/root/mesos-ec2/stop-mesos 9fdb8753dffc5115f94582753e0860538be6232b
ec2/deploy.amazon64/root/mesos-ec2/zoo
ec2/deploy.amazon64/root/persistent-hdfs/conf/core-site.xml
ec2/deploy.amazon64/root/persistent-hdfs/conf/hadoop-env.sh b38ba01817e3b9c9715a476ecb692bac68983f50
ec2/deploy.amazon64/root/persistent-hdfs/conf/hdfs-site.xml ec000cb2f306232a29cf701b10f230685d2662c9
ec2/deploy.amazon64/root/persistent-hdfs/conf/mapred-site.xml
ec2/deploy.amazon64/root/persistent-hdfs/conf/masters
ec2/deploy.amazon64/root/persistent-hdfs/conf/slaves
ec2/deploy.amazon64/root/spark/conf/spark-env.sh 6b331ff2257d1b88c8ebe2e8095887a3e2cf0d54
ec2/deploy.generic/root/mesos-ec2/ec2-variables.sh 1f76e61b2feffa4eadc53745caa5033dde182531
ec2/deploy.generic/root/mesos-ec2/mesos-master-env.sh PRE-CREATION
ec2/deploy.generic/root/mesos-ec2/mesos-slave-env.sh PRE-CREATION
ec2/deploy.generic/root/mesos-ec2/redeploy-mesos PRE-CREATION
ec2/deploy.generic/root/mesos-ec2/spark-env.sh PRE-CREATION
ec2/mesos-ec2 3bc5d6307b2759cec0cea430b44baa2809a6f2d2
ec2/mesos_ec2.py 94fd7b2181d138c97da609fb0ed2ebdc6206cfd6
ec2/setup-mesos-ami.sh PRE-CREATION
Diff: https://reviews.apache.org/r/11914/diff/
Testing (updated)
-------
I've done some very minimal testing that Spark runs over Mesos and can write to the ephemeral HDFS instance and that the Hadoop jobtracker can connect to Mesos and the ephemeral HDFS.
Thanks,
Charles Reiss