You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@mesos.apache.org by Niklas Nielsen <ni...@qni.dk> on 2015/06/04 20:01:38 UTC
Re: Review Request 11914: Get EC2 scripts into workable state
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11914/#review86673
-----------------------------------------------------------
Stale review: Charles, still want this in?
- Niklas Nielsen
On July 14, 2013, 3:44 p.m., Charles Reiss wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/11914/
> -----------------------------------------------------------
>
> (Updated July 14, 2013, 3:44 p.m.)
>
>
> Review request for mesos and Andy Konwinski.
>
>
> Bugs: MESOS-500
> https://issues.apache.org/jira/browse/MESOS-500
>
>
> Repository: mesos-incubating
>
>
> Description
> -------
>
> This is a rather big change that updates the EC2 scripts to deal with changes in Mesos over the past 18 months (or so) and to be usable under the assumption that no one will be maintaining an "official" Mesos AMI.
>
> The big changes are:
> - users are expected to supply their own AMI, there is no default;
> - the webui is firewalled away as exposing the libprocess port is dangerous (AFAIK), instead users are instructed to use the SSH-forwarding SOCKS proxy;
> - instructions and a script for building an AMI (which should work on many Debian- or Fedora-like Linux distros). The resulting AMI should be similar to the one that was originally supplied, including installations of Hadoop (setup to work with Mesos) and Spark;
> - use the deploy scripts to launch mesos;
> - drop support for downloading Mesos from git and rebuilding and redeploying it;
> - drop support for Torque/Hypertable/haproxy in the AMI;
>
>
> Diffs
> -----
>
> docs/EC2-Scripts.md PRE-CREATION
> docs/EC2-Scripts.textile 7a1d1da834b4b85404793a27ca748905200835ca
> ec2/Makefile.am 8c64f485888df1599697eb181fc76aa83206da07
> ec2/deploy.amazon64/root/ephemeral-hdfs/conf/core-site.xml
> ec2/deploy.amazon64/root/ephemeral-hdfs/conf/hadoop-env.sh 4e1e6991591e09f8860ab130948b0e787fce2b42
> ec2/deploy.amazon64/root/ephemeral-hdfs/conf/hdfs-site.xml 43e68aa3e2ecbb53bd3b0a99d33b11aadda2cca7
> ec2/deploy.amazon64/root/ephemeral-hdfs/conf/mapred-site.xml
> ec2/deploy.amazon64/root/ephemeral-hdfs/conf/masters
> ec2/deploy.amazon64/root/ephemeral-hdfs/conf/slaves
> ec2/deploy.amazon64/root/mesos-ec2/cluster-url
> ec2/deploy.amazon64/root/mesos-ec2/copy-dir
> ec2/deploy.amazon64/root/mesos-ec2/create-swap
> ec2/deploy.amazon64/root/mesos-ec2/hadoop-framework-conf/core-site.xml
> ec2/deploy.amazon64/root/mesos-ec2/hadoop-framework-conf/hadoop-env.sh d8483140546fe00d6d17494c14ef4b09ae368496
> ec2/deploy.amazon64/root/mesos-ec2/hadoop-framework-conf/mapred-site.xml 0ffa92f115d66adc9e370f4030850216545ade38
> ec2/deploy.amazon64/root/mesos-ec2/haproxy+apache/haproxy.config.template 957c3f6a6b2a2f658e076337d88e69447b2a3341
> ec2/deploy.amazon64/root/mesos-ec2/hypertable/Capfile 8b50912745977cb71232ba1dfa77f8bb0d60191e
> ec2/deploy.amazon64/root/mesos-ec2/hypertable/hypertable.cfg b4d5b7475fab3d4a842e4b0f459abc5ca316996a
> ec2/deploy.amazon64/root/mesos-ec2/masters
> ec2/deploy.amazon64/root/mesos-ec2/mesos-daemon bed27657f8718eecafb83cca5d29e0612a87f129
> ec2/deploy.amazon64/root/mesos-ec2/redeploy-mesos 941d783d82f0708a6da0f4677c3364537dfded63
> ec2/deploy.amazon64/root/mesos-ec2/setup b6b736091d4d5be431c8da29cdb98360a1df2d29
> ec2/deploy.amazon64/root/mesos-ec2/setup-slave 436f417bc5a746ad74cc88c27e630a91d55b0b23
> ec2/deploy.amazon64/root/mesos-ec2/setup-torque 2ac8fd3546063d3ba391147383de53b7824c7c8c
> ec2/deploy.amazon64/root/mesos-ec2/slaves
> ec2/deploy.amazon64/root/mesos-ec2/ssh-no-keychecking
> ec2/deploy.amazon64/root/mesos-ec2/start-hypertable af16c2d7bd615cb0c98f6ba65ff5c69859678850
> ec2/deploy.amazon64/root/mesos-ec2/start-mesos 0f551db396fa7ffebef880dca0232c22808ff7cc
> ec2/deploy.amazon64/root/mesos-ec2/stop-hypertable 7280dc11bfc53ae84b7ecaba34c84810461ed7f4
> ec2/deploy.amazon64/root/mesos-ec2/stop-mesos 9fdb8753dffc5115f94582753e0860538be6232b
> ec2/deploy.amazon64/root/mesos-ec2/zoo
> ec2/deploy.amazon64/root/persistent-hdfs/conf/core-site.xml
> ec2/deploy.amazon64/root/persistent-hdfs/conf/hadoop-env.sh b38ba01817e3b9c9715a476ecb692bac68983f50
> ec2/deploy.amazon64/root/persistent-hdfs/conf/hdfs-site.xml ec000cb2f306232a29cf701b10f230685d2662c9
> ec2/deploy.amazon64/root/persistent-hdfs/conf/mapred-site.xml
> ec2/deploy.amazon64/root/persistent-hdfs/conf/masters
> ec2/deploy.amazon64/root/persistent-hdfs/conf/slaves
> ec2/deploy.amazon64/root/spark/conf/spark-env.sh 6b331ff2257d1b88c8ebe2e8095887a3e2cf0d54
> ec2/deploy.generic/root/mesos-ec2/ec2-variables.sh 1f76e61b2feffa4eadc53745caa5033dde182531
> ec2/deploy.generic/root/mesos-ec2/mesos-master-env.sh PRE-CREATION
> ec2/deploy.generic/root/mesos-ec2/mesos-slave-env.sh PRE-CREATION
> ec2/deploy.generic/root/mesos-ec2/redeploy-mesos PRE-CREATION
> ec2/deploy.generic/root/mesos-ec2/spark-env.sh PRE-CREATION
> ec2/mesos-ec2 3bc5d6307b2759cec0cea430b44baa2809a6f2d2
> ec2/mesos_ec2.py 94fd7b2181d138c97da609fb0ed2ebdc6206cfd6
> ec2/setup-mesos-ami.sh PRE-CREATION
>
> Diff: https://reviews.apache.org/r/11914/diff/
>
>
> Testing
> -------
>
> I've done some very minimal testing that Spark runs over Mesos and can write to the ephemeral HDFS instance and that the Hadoop jobtracker can connect to Mesos and the ephemeral HDFS.
>
>
> Thanks,
>
> Charles Reiss
>
>
Re: Review Request 11914: Get EC2 scripts into workable state
Posted by Niklas Nielsen <ni...@qni.dk>.
> On June 4, 2015, 11:01 a.m., Niklas Nielsen wrote:
> > Stale review: Charles, still want this in?
Closing for now.
- Niklas
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11914/#review86673
-----------------------------------------------------------
On July 14, 2013, 3:44 p.m., Charles Reiss wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/11914/
> -----------------------------------------------------------
>
> (Updated July 14, 2013, 3:44 p.m.)
>
>
> Review request for mesos and Andy Konwinski.
>
>
> Bugs: MESOS-500
> https://issues.apache.org/jira/browse/MESOS-500
>
>
> Repository: mesos-incubating
>
>
> Description
> -------
>
> This is a rather big change that updates the EC2 scripts to deal with changes in Mesos over the past 18 months (or so) and to be usable under the assumption that no one will be maintaining an "official" Mesos AMI.
>
> The big changes are:
> - users are expected to supply their own AMI, there is no default;
> - the webui is firewalled away as exposing the libprocess port is dangerous (AFAIK), instead users are instructed to use the SSH-forwarding SOCKS proxy;
> - instructions and a script for building an AMI (which should work on many Debian- or Fedora-like Linux distros). The resulting AMI should be similar to the one that was originally supplied, including installations of Hadoop (setup to work with Mesos) and Spark;
> - use the deploy scripts to launch mesos;
> - drop support for downloading Mesos from git and rebuilding and redeploying it;
> - drop support for Torque/Hypertable/haproxy in the AMI;
>
>
> Diffs
> -----
>
> docs/EC2-Scripts.md PRE-CREATION
> docs/EC2-Scripts.textile 7a1d1da834b4b85404793a27ca748905200835ca
> ec2/Makefile.am 8c64f485888df1599697eb181fc76aa83206da07
> ec2/deploy.amazon64/root/ephemeral-hdfs/conf/core-site.xml
> ec2/deploy.amazon64/root/ephemeral-hdfs/conf/hadoop-env.sh 4e1e6991591e09f8860ab130948b0e787fce2b42
> ec2/deploy.amazon64/root/ephemeral-hdfs/conf/hdfs-site.xml 43e68aa3e2ecbb53bd3b0a99d33b11aadda2cca7
> ec2/deploy.amazon64/root/ephemeral-hdfs/conf/mapred-site.xml
> ec2/deploy.amazon64/root/ephemeral-hdfs/conf/masters
> ec2/deploy.amazon64/root/ephemeral-hdfs/conf/slaves
> ec2/deploy.amazon64/root/mesos-ec2/cluster-url
> ec2/deploy.amazon64/root/mesos-ec2/copy-dir
> ec2/deploy.amazon64/root/mesos-ec2/create-swap
> ec2/deploy.amazon64/root/mesos-ec2/hadoop-framework-conf/core-site.xml
> ec2/deploy.amazon64/root/mesos-ec2/hadoop-framework-conf/hadoop-env.sh d8483140546fe00d6d17494c14ef4b09ae368496
> ec2/deploy.amazon64/root/mesos-ec2/hadoop-framework-conf/mapred-site.xml 0ffa92f115d66adc9e370f4030850216545ade38
> ec2/deploy.amazon64/root/mesos-ec2/haproxy+apache/haproxy.config.template 957c3f6a6b2a2f658e076337d88e69447b2a3341
> ec2/deploy.amazon64/root/mesos-ec2/hypertable/Capfile 8b50912745977cb71232ba1dfa77f8bb0d60191e
> ec2/deploy.amazon64/root/mesos-ec2/hypertable/hypertable.cfg b4d5b7475fab3d4a842e4b0f459abc5ca316996a
> ec2/deploy.amazon64/root/mesos-ec2/masters
> ec2/deploy.amazon64/root/mesos-ec2/mesos-daemon bed27657f8718eecafb83cca5d29e0612a87f129
> ec2/deploy.amazon64/root/mesos-ec2/redeploy-mesos 941d783d82f0708a6da0f4677c3364537dfded63
> ec2/deploy.amazon64/root/mesos-ec2/setup b6b736091d4d5be431c8da29cdb98360a1df2d29
> ec2/deploy.amazon64/root/mesos-ec2/setup-slave 436f417bc5a746ad74cc88c27e630a91d55b0b23
> ec2/deploy.amazon64/root/mesos-ec2/setup-torque 2ac8fd3546063d3ba391147383de53b7824c7c8c
> ec2/deploy.amazon64/root/mesos-ec2/slaves
> ec2/deploy.amazon64/root/mesos-ec2/ssh-no-keychecking
> ec2/deploy.amazon64/root/mesos-ec2/start-hypertable af16c2d7bd615cb0c98f6ba65ff5c69859678850
> ec2/deploy.amazon64/root/mesos-ec2/start-mesos 0f551db396fa7ffebef880dca0232c22808ff7cc
> ec2/deploy.amazon64/root/mesos-ec2/stop-hypertable 7280dc11bfc53ae84b7ecaba34c84810461ed7f4
> ec2/deploy.amazon64/root/mesos-ec2/stop-mesos 9fdb8753dffc5115f94582753e0860538be6232b
> ec2/deploy.amazon64/root/mesos-ec2/zoo
> ec2/deploy.amazon64/root/persistent-hdfs/conf/core-site.xml
> ec2/deploy.amazon64/root/persistent-hdfs/conf/hadoop-env.sh b38ba01817e3b9c9715a476ecb692bac68983f50
> ec2/deploy.amazon64/root/persistent-hdfs/conf/hdfs-site.xml ec000cb2f306232a29cf701b10f230685d2662c9
> ec2/deploy.amazon64/root/persistent-hdfs/conf/mapred-site.xml
> ec2/deploy.amazon64/root/persistent-hdfs/conf/masters
> ec2/deploy.amazon64/root/persistent-hdfs/conf/slaves
> ec2/deploy.amazon64/root/spark/conf/spark-env.sh 6b331ff2257d1b88c8ebe2e8095887a3e2cf0d54
> ec2/deploy.generic/root/mesos-ec2/ec2-variables.sh 1f76e61b2feffa4eadc53745caa5033dde182531
> ec2/deploy.generic/root/mesos-ec2/mesos-master-env.sh PRE-CREATION
> ec2/deploy.generic/root/mesos-ec2/mesos-slave-env.sh PRE-CREATION
> ec2/deploy.generic/root/mesos-ec2/redeploy-mesos PRE-CREATION
> ec2/deploy.generic/root/mesos-ec2/spark-env.sh PRE-CREATION
> ec2/mesos-ec2 3bc5d6307b2759cec0cea430b44baa2809a6f2d2
> ec2/mesos_ec2.py 94fd7b2181d138c97da609fb0ed2ebdc6206cfd6
> ec2/setup-mesos-ami.sh PRE-CREATION
>
> Diff: https://reviews.apache.org/r/11914/diff/
>
>
> Testing
> -------
>
> I've done some very minimal testing that Spark runs over Mesos and can write to the ephemeral HDFS instance and that the Hadoop jobtracker can connect to Mesos and the ephemeral HDFS.
>
>
> Thanks,
>
> Charles Reiss
>
>