You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Nicholas Chammas (JIRA)" <ji...@apache.org> on 2015/12/18 22:08:46 UTC
[jira] [Comment Edited] (SPARK-3821) Develop an automated way of
creating Spark images (AMI, Docker, and others)
[ https://issues.apache.org/jira/browse/SPARK-3821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14203280#comment-14203280 ]
Nicholas Chammas edited comment on SPARK-3821 at 12/18/15 9:08 PM:
-------------------------------------------------------------------
After much dilly-dallying, I am happy to present:
* A brief proposal / design doc ([fixed JIRA attachment | https://issues.apache.org/jira/secure/attachment/12680371/packer-proposal.html], [md file on GitHub | https://github.com/nchammas/spark-ec2/blob/packer/image-build/proposal.md])
* [Initial implementation | https://github.com/nchammas/spark-ec2/tree/packer/image-build] and [README | https://github.com/nchammas/spark-ec2/blob/packer/image-build/README.md]
* New AMIs generated by this implementation: [Base AMIs | https://github.com/nchammas/spark-ec2/tree/packer/ami-list/base], [Spark 1.1.0 Pre-Installed | https://github.com/nchammas/spark-ec2/tree/packer/ami-list/1.1.0]
To try out the new AMIs with {{spark-ec2}}, you'll need to update [these | https://github.com/apache/spark/blob/7e9d975676d56ace0e84c2200137e4cd4eba074a/ec2/spark_ec2.py#L47] [two | https://github.com/apache/spark/blob/7e9d975676d56ace0e84c2200137e4cd4eba074a/ec2/spark_ec2.py#L593] lines (well, really, just the first one) to point to [my {{spark-ec2}} repo on the {{packer}} branch | https://github.com/nchammas/spark-ec2/tree/packer/image-build].
Your candid feedback and/or improvements are most welcome!
was (Author: nchammas):
After much dilly-dallying, I am happy to present:
* A brief proposal / design doc ([fixed JIRA attachment | https://issues.apache.org/jira/secure/attachment/12680371/packer-proposal.html], [md file on GitHub | https://github.com/nchammas/spark-ec2/blob/packer/packer/proposal.md])
* [Initial implementation | https://github.com/nchammas/spark-ec2/tree/packer/packer] and [README | https://github.com/nchammas/spark-ec2/blob/packer/packer/README.md]
* New AMIs generated by this implementation: [Base AMIs | https://github.com/nchammas/spark-ec2/tree/packer/ami-list/base], [Spark 1.1.0 Pre-Installed | https://github.com/nchammas/spark-ec2/tree/packer/ami-list/1.1.0]
To try out the new AMIs with {{spark-ec2}}, you'll need to update [these | https://github.com/apache/spark/blob/7e9d975676d56ace0e84c2200137e4cd4eba074a/ec2/spark_ec2.py#L47] [two | https://github.com/apache/spark/blob/7e9d975676d56ace0e84c2200137e4cd4eba074a/ec2/spark_ec2.py#L593] lines (well, really, just the first one) to point to [my {{spark-ec2}} repo on the {{packer}} branch | https://github.com/nchammas/spark-ec2/tree/packer/packer].
Your candid feedback and/or improvements are most welcome!
> Develop an automated way of creating Spark images (AMI, Docker, and others)
> ---------------------------------------------------------------------------
>
> Key: SPARK-3821
> URL: https://issues.apache.org/jira/browse/SPARK-3821
> Project: Spark
> Issue Type: Improvement
> Components: Build, EC2
> Reporter: Nicholas Chammas
> Assignee: Nicholas Chammas
> Attachments: packer-proposal.html
>
>
> Right now the creation of Spark AMIs or Docker containers is done manually. With tools like [Packer|http://www.packer.io/], we should be able to automate this work, and do so in such a way that multiple types of machine images can be created from a single template.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org