[jira] [Created] (SPARK-5879) spary_ec2.py should expose/return master and slave lists (e.g. write to file)

["Florian Verhein (JIRA)" <ji...@apache.org>]

Florian Verhein created SPARK-5879:
--------------------------------------

             Summary: spary_ec2.py should expose/return master and slave lists (e.g. write to file)
                 Key: SPARK-5879
                 URL: https://issues.apache.org/jira/browse/SPARK-5879
             Project: Spark
          Issue Type: Improvement
          Components: EC2
            Reporter: Florian Verhein



After running spark_ec2.py, it is often useful/necessary to know the master's ip / dn. Particularly if running spark_ec2.py is part of a larger pipeline.

For example, consider a wrapper that launches a cluster, then waits for completion of some application running on it (e.g. polling via ssh), before destroying the cluster.

Some options: 
- write `launch-variables.sh` with MASTERS and SLAVES exports (i.e. basically a subset of the ec2_variables.sh that is temporarily created as part of deploy_files variable substitution)
- launch-variables.json (same info but as json) 

Both would be useful depending on the wrapper language. 

I think we should incorporate the cluster name for the case that multiple clusters are launched. E.g. <cluster_name>_variables.sh/.json

Thoughts?




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org