[jira] [Commented] (SPARK-3821) Develop an automated way of creating Spark images (AMI, Docker, and others)

["Nicholas Chammas (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-3821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14256683#comment-14256683 ] 

Nicholas Chammas commented on SPARK-3821:
-----------------------------------------

Per the discussion earlier, I've [updated|https://github.com/nchammas/spark-ec2/tree/packer/packer] the Packer build configuration to drop the release-specific builds. I've also added GNU parallel to the list of installed tools and will use it in place of the {{while ... rsync ... & wait}} pattern used throughout the various setup scripts.

I'll test out these changes on small (< 5 nodes) and large (>= 100 nodes) cluster launches and post updated benchmarks as well as an updated README and proposal.

> Develop an automated way of creating Spark images (AMI, Docker, and others)
> ---------------------------------------------------------------------------
>
>                 Key: SPARK-3821
>                 URL: https://issues.apache.org/jira/browse/SPARK-3821
>             Project: Spark
>          Issue Type: Improvement
>          Components: Build, EC2
>            Reporter: Nicholas Chammas
>            Assignee: Nicholas Chammas
>         Attachments: packer-proposal.html
>
>
> Right now the creation of Spark AMIs or Docker containers is done manually. With tools like [Packer|http://www.packer.io/], we should be able to automate this work, and do so in such a way that multiple types of machine images can be created from a single template.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org